Voice recommendation method and device, model training method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of voice processing, and provides a voice recommendation method and device, a model training method and device, electronic equipment and a storage medium, and the voice recommendation method comprises the steps: obtaining candidate voices; inputting the candidate voices into an importance characterization model to obtain importance features of the candidate voices output by the importance characterization model based on text content features of the candidate voices and / or speaker features; the importance characterization model is obtained by training a plurality of tasks, and the plurality of tasks comprise a speech recognition task based on text content features and / or a speaker recognition task based on speaker features, and an importance classification task based on importance features; and performing voice recommendation based on the importance features of the candidate voices. According to the method, the device, the electronic equipment and the storage medium provided by the invention, the importance degree of the candidate voice is reflected from the text content and / or the speaker information and whether the importance degree is important, so that the importance degree measurement is more comprehensive, and the voice recommendation accuracy can be ensured.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a speech recommendation, model training method, device, electronic device and storage medium. Background Art

[0002] With the development of information technology, massive amounts of information are presented in the form of voice, and automated voice recommendation methods have emerged.

[0003] Most of the current voice recommendation methods first transcribe the voice into text, then perform text analysis such as intent recognition on the text, and select important voice recommendations based on the results of the text analysis to recommend to relevant personnel. The recommendation effect of the above methods is affected by the effects of various links such as voice transcription and intent recognition. Once the effect of any link is not good, it is very likely that missed recommendations or wrong recommendations will occur, affecting the personnel experience. Summary of the invention

[0004] The present invention provides a voice recommendation, model training method, device, electronic device and storage medium, which are used to solve the defect of poor voice recommendation effect in related technologies.

[0005] The present invention provides a voice recommendation method, comprising: Obtain candidate voices; Inputting the candidate speech into an importance characterization model, obtaining the importance feature of the candidate speech output by the importance characterization model based on the text content feature and / or speaker feature of the candidate speech; the importance characterization model is obtained through training of multiple tasks, the multiple tasks including a speech recognition task based on the text content feature and / or a speaker recognition task based on the speaker feature, and an importance classification task based on the importance feature; Based on the importance features of the candidate voices, voice recommendations are performed.

[0006] According to a speech recommendation method provided by the present invention, the step of inputting the candidate speech into an importance characterization model to obtain the importance feature of the candidate speech output by the importance characterization model based on the text content feature and / or speaker feature of the candidate speech comprises: Extracting text content features of the candidate speech based on the text content extraction network in the importance representation model and the speech features of the candidate speech, and / or extracting speaker features of the candidate speech based on the speaker extraction network in the importance representation model and the speech features of the candidate speech; Importance features of the candidate speech are extracted based on the importance extraction network in the importance representation model, the speech features, and the text content features and / or the speaker features.

[0007] According to a voice recommendation method provided by the present invention, the voice recommendation based on the importance feature of the candidate voice includes: Clustering the importance features of each candidate speech to obtain at least one speech cluster; A candidate voice is determined from each voice cluster as a deduplicated voice, and voice recommendation is performed based on the importance features of the deduplicated voice.

[0008] According to a voice recommendation method provided by the present invention, the voice recommendation is performed based on the importance feature of the deduplicated voice, comprising: Performing voice recommendation based on the similarity between the importance features of the deduplicated voice and the importance features of each registered voice cluster; The registered speech clusters are obtained by clustering the importance features of the registered speech.

[0009] According to a voice recommendation method provided by the present invention, the voice recommendation is performed based on the similarity between the importance features of the deduplicated voice and the importance features of each registered voice cluster, including: When the maximum value of the similarities between the importance features of the deduplicated speech and the importance features of each registered speech cluster is greater than a preset threshold, and the registered speech cluster corresponding to the maximum value is a recommended speech cluster, speech recommendation is performed on the deduplicated speech.

[0010] According to a speech recommendation method provided by the present invention, the step of inputting the candidate speech into an importance characterization model to obtain the importance feature of the candidate speech output by the importance characterization model based on the text content feature and / or speaker feature of the candidate speech comprises: Inputting each speech segment of the candidate speech into an importance characterization model, and obtaining importance features of each speech segment output by the importance characterization model based on speech features and / or text content features of each speech segment; Based on the importance features of the speech segments, the importance features of the candidate speech are determined.

[0011] The present invention also provides a model training method, comprising: Determining training samples for a plurality of tasks, wherein the plurality of tasks include a speech recognition task based on text content features and / or a speaker recognition task based on speaker features, and an importance classification task based on importance features; Model training is performed based on training samples of multiple tasks to obtain an importance characterization model, wherein the importance characterization model is used to output importance features of the candidate speech based on text content features and / or speaker features of the candidate speech.

[0012] According to a model training method provided by the present invention, the model training is performed based on training samples of multiple tasks to obtain an importance representation model, including: Extracting sample text content features from speech features of training samples of the speech recognition task based on a text content extraction network in the initial model, and / or extracting sample speaker features from speech features of training samples of the speaker recognition task based on a speaker extraction network in the initial model; Extracting sample importance features from speech features of the training samples of the importance classification task, and sample text content features and / or sample speaker features of the training samples of the importance classification task based on the importance extraction network in the initial model; Based on the speech recognition loss determined by the sample text content features and / or the speaker recognition loss determined by the sample speaker features, and the importance loss determined by the sample importance features, the initial model is iterated on parameters to obtain the importance representation model.

[0013] The present invention also provides a voice recommendation device, comprising: A speech acquisition unit, used for acquiring candidate speech; A feature extraction unit, used for inputting the candidate speech into an importance characterization model, and obtaining the importance feature of the candidate speech output by the importance characterization model based on the text content feature and / or speaker feature of the candidate speech; the importance characterization model is obtained through training of multiple tasks, the multiple tasks including a speech recognition task based on the text content feature and / or a speaker recognition task based on the speaker feature, and an importance classification task based on the importance feature; The voice recommendation unit is used to make voice recommendations based on the importance features of the candidate voices.

[0014] The present invention also provides a model training device, comprising: A sample acquisition unit, used to determine training samples for a plurality of tasks, wherein the plurality of tasks include a speech recognition task based on text content features and / or a speaker recognition task based on speaker features, and an importance classification task based on importance features; The model training unit is used to perform model training based on training samples of multiple tasks to obtain an importance representation model, wherein the importance representation model is used to output the importance features of the candidate speech based on the text content features and / or speaker features of the candidate speech.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the voice recommendation method or model training method described above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described speech recommendation methods or model training methods.

[0017] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned voice recommendation methods or model training methods.

[0018] The speech recommendation, model training method, device, electronic device and storage medium provided by the present invention extract the importance features of candidate speech through the importance representation model obtained by multiple task training, and perform speech recommendation based on this. On the one hand, end-to-end speech recommendation is realized, and there is no situation in the prior art where multiple links are cascaded and the accuracy of each link affects the speech recommendation effect, which can improve the reliability of speech recommendation; on the other hand, the importance features can reflect the importance of candidate speech from aspects such as text content and / or speaker information, and whether it is important, so that the measurement of importance is more comprehensive and reliable, which can further ensure the accuracy of speech recommendation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flowchart of voice recommendation in related technology.

[0021] Figure 2 It is a flow chart of the voice recommendation method provided by the present invention.

[0022] Figure 3 It is a structural schematic diagram of the importance characterization model provided by the present invention.

[0023] Figure 4 It is a flow chart of the model training method provided by the present invention.

[0024] Figure 5 It is a structural schematic diagram of the voice recommendation device provided by the present invention.

[0025] Figure 6 It is a structural schematic diagram of the model training device provided by the present invention.

[0026] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] With the development of information technology, a large amount of information is presented in the form of voice. In order to mine important information, relevant personnel usually need to use various means to collect a large amount of voice data and then screen it. However, due to the uncertainty of the collection target and the diversity of collection methods, the voice data obtained is often large in volume and complex in content, which brings challenges to relevant personnel in information mining.

[0029] In order to facilitate relevant personnel to conduct information mining based on voice data, automated voice recommendation methods came into being.

[0030] Figure 1 is a flowchart of voice recommendation in related technologies, such as Figure 1 As shown, most of the current voice recommendation methods are to first transcribe the voice into text, then perform text analysis such as intent recognition on the text, and select important voice recommendations to relevant personnel based on the results of the text analysis.

[0031] However, since the amount of voice data to be recommended is huge and there may be a lot of repeated voice content, directly transcribing the voice will result in a large waste of computing power and reduce the efficiency of voice recommendation. Moreover, due to the limitations of the accuracy of each link such as voice transcription and intent recognition, each link in the voice recommendation will have an impact on the recommendation effect, which will lead to the omission of important voices or the mistaken recommendation of unimportant voices. Only when the accuracy of each link such as voice transcription and intent recognition is very high can better voice recommendation be achieved.

[0032] In view of the above problems, an embodiment of the present invention provides a voice recommendation method. Figure 2 FIG. 1 is a flow chart of the voice recommendation method provided by the present invention. Figure 2 As shown, the method includes: Step 210: Acquire candidate speech.

[0033] Here, the candidate voices are pre-collected voices that can be used for voice recommendation.

[0034] The candidate voice can be obtained through a sound pickup device, which can be a smart phone, a tablet computer, or a smart appliance such as a stereo, a television, and an air conditioner. After the sound pickup device obtains the candidate voice through a microphone array, it can also amplify and reduce noise for the candidate voice. In addition, the candidate voice can be a voice segment formed after the sound pickup is completed, or it can be a voice stream in the real-time sound pickup process, which is not specifically limited in the embodiment of the present invention.

[0035] It is understandable that the number of candidate voices may be one or more, and this is not specifically limited in the embodiment of the present invention.

[0036] Step 220, input the candidate speech into the importance representation model, and obtain the importance features of the candidate speech output by the importance representation model based on the text content features and / or speaker features of the candidate speech; the importance representation model is obtained through training of multiple tasks, and the multiple tasks include a speech recognition task based on the text content features and / or a speaker recognition task based on the speaker features, and an importance classification task based on the importance features.

[0037] Specifically, after obtaining the candidate speech, the candidate speech can be input into the importance representation model, and the importance of the input candidate speech at the speech level is characterized by the importance representation model, thereby obtaining the importance feature of the candidate speech output by the importance representation model. Here, the importance feature of the candidate speech is used to characterize the importance of the candidate speech in the speech recommendation scenario.

[0038] Here, the importance representation model is a pre-trained neural network model for extracting importance features of input speech. When training the importance representation model, a training method combining multiple tasks can be adopted, and the multiple tasks referred to here include importance classification tasks, speech recognition tasks and / or speaker recognition tasks.

[0039] Among them, the importance classification task is to connect the classification layer after the importance features extracted from the speech to realize the classification of whether the speech is important. The speech recognition task is to connect the fully connected layer of speech recognition after the text content features extracted from the speech to realize speech recognition. The speaker recognition task is to connect the classification layer after the speaker features extracted from the speech to realize the classification of the speaker to which the speech belongs.

[0040] When training the importance representation model, the training can be combined with the importance classification task and the speech recognition task. The importance representation model trained in this way can extract the text content features of the input candidate speech, and can extract the importance features of the candidate speech based on the text content features of the candidate speech. The importance features obtained in this way integrate the text content in the candidate speech, and can reflect the importance of the candidate speech from two aspects: text content and whether it is important.

[0041] Alternatively, when training the importance representation model, the training can be combined with the importance classification task and the speaker recognition task, so that the importance representation model trained in this way can extract the speaker features of the input candidate speech, and can extract the importance features of the candidate speech based on the speaker features of the candidate speech. The importance features thus obtained integrate the speaker information in the candidate speech, and can reflect the importance of the candidate speech from two aspects: speaker information and whether it is important.

[0042] Alternatively, when training the importance representation model, the training can be combined with the importance classification task, the speech recognition task and the speaker recognition task. The importance representation model thus trained can extract the text content features of the candidate speech and the speaker features of the candidate speech for the input candidate speech, and can extract the importance features of the candidate speech based on the text content features and speaker features of the candidate speech. The importance features thus obtained integrate the text content and speaker information in the candidate speech, and can reflect the importance of the candidate speech from three aspects: text content, speaker information and whether it is important.

[0043] That is, the importance representation model obtained through multiple task training can realize the extraction of importance features of candidate speech, and the importance features extracted in this way can reflect the importance of the candidate speech from the aspects of text content and / or speaker information, and whether it is important. Compared with the related technology that only considers whether the candidate speech is important from the intention, the measurement of the importance of the candidate speech is more comprehensive and reliable.

[0044] Moreover, in the embodiment of the present invention, the importance representation model is an end-to-end model, and the importance characteristics of the candidate speech can be obtained by inputting it. There is no situation in the related technology where multiple links are cascaded and the accuracy of each link will affect the effect of speech recommendation. This can greatly enhance the reliability of speech recommendation.

[0045] Step 230: perform voice recommendation based on the importance features of the candidate voices.

[0046] Specifically, after obtaining the importance features of the candidate voices, voice recommendations can be made based on the importance features. For example, based on the importance features of the candidate voices, the candidate voices can be classified as important, thereby recommending the important candidate voices as voices that need to be recommended; for another example, the distance between the importance features of the candidate voices and the pre-set recommendation importance features can be calculated. When the distance is less than a preset threshold, the candidate voice is recommended as a voice that needs to be recommended. When the distance is greater than or equal to the preset threshold, the candidate voice is not recommended. The embodiment of the present invention does not specifically limit this.

[0047] The method provided by the embodiment of the present invention extracts the importance features of candidate voices through an importance representation model obtained through multiple task training, and performs voice recommendation based on this. On the one hand, end-to-end voice recommendation is realized, and there is no situation in the prior art where multiple links are cascaded and the accuracy of each link will affect the voice recommendation effect, which can improve the reliability of voice recommendation; on the other hand, the importance features can reflect the importance of candidate voices from aspects such as text content and / or speaker information, and whether they are important, so that the measurement of importance is more comprehensive and reliable, which can further ensure the accuracy of voice recommendation.

[0048] Based on the above embodiment, in step 220, the step of inputting the candidate speech into the importance characterization model to obtain the importance feature of the candidate speech output by the importance characterization model based on the text content feature and / or speaker feature of the candidate speech includes: Extracting text content features of the candidate speech based on the text content extraction network in the importance representation model and the speech features of the candidate speech, and / or extracting speaker features of the candidate speech based on the speaker extraction network in the importance representation model and the speech features of the candidate speech; Importance features of the candidate speech are extracted based on the importance extraction network in the importance representation model, the speech features, and the text content features and / or the speaker features.

[0049] Specifically, the importance representation model may include a text content extraction network and / or a speaker extraction network. The text content extraction network is a network used to extract text content features, and the text content extraction network can be obtained by training based on speech recognition tasks in the multi-task training of the importance representation model; the speaker extraction network is a network used to extract speaker features, and the speaker extraction network can be obtained by training based on speaker recognition tasks in the multi-task training of the importance representation model.

[0050] When the candidate speech is input into the importance representation model, the importance representation model can extract the speech features of the candidate speech. For example, the importance representation model can also include a pre-trained speech model, and the speech features of the candidate speech are extracted through the pre-trained speech model. The pre-trained speech model here can be a CPC (Contrastive Predictive Coding) neural network, a wav2vec2.0 neural network, etc., and the embodiments of the present invention do not specifically limit this.

[0051] After obtaining the speech features of the candidate speech, the importance representation model can extract the text content features of the candidate speech from the speech features of the candidate speech based on the text content extraction network contained in itself; and / or, the importance representation model can extract the speaker features of the candidate speech from the speech features of the candidate speech based on the speaker extraction network contained in itself.

[0052] Here, both the text content extraction network and the speaker extraction network can be network structures including convolutional neural networks, and both can further extract high-level features of the corresponding tasks from the speech features of the candidate speech, corresponding to the high-level features extracted by the text content extraction network, namely the text content features, and corresponding to the high-level features extracted by the speaker extraction network, namely the speaker features.

[0053] In addition, the importance representation model also includes an importance extraction network, and the input of the importance extraction network is connected to the output of the text content extraction network and / or the speaker extraction network. The importance extraction network can be obtained by combining speech recognition tasks and / or speaker recognition tasks and importance classification tasks in multi-task training of the importance representation model.

[0054] After obtaining the speech features of the candidate speech, as well as the text content features and / or speaker features of the candidate speech, the importance representation model can extract the importance features of the candidate speech based on the importance extraction network contained in itself, combined with the speech features of the candidate speech, as well as the text content features and / or speaker features of the candidate speech.

[0055] Here, the importance extraction network can also be a network structure including a convolutional neural network, and the importance extraction network can further extract high-level features from the voice features of the candidate speech, as well as the text content features and / or speaker features, that is, obtain importance features that can integrate text content and / or speaker information.

[0056] Based on any of the above embodiments, Figure 3 Schematic diagram of the structure of the importance representation model provided by the present invention. Figure 3 As shown, the importance representation model can include Figure 3The part framed by the dotted line, i.e., the importance representation model, includes a pre-training model, a text content extraction network, an importance extraction network, and a speaker extraction network.

[0057] The pre-trained model takes the candidate speech as input and the speech features of the candidate speech as output. Moreover, the output of the pre-trained model can be used as the input of the text content extraction network, the importance extraction network and the speaker extraction network respectively. The pre-trained model here can be a CPC neural network, a wav2vec2.0 neural network, etc. Taking the CPC neural network as an example, the CPC neural network can first use a nonlinear coding layer (for example, a 5-layer convolutional neural network CNN+rectified linear unit ReLU) to downsample and encode the input candidate speech to obtain hidden variables, and then use an autoregressive model (for example, a layer of gated recurrent unit recurrent neural network GRU RNN) to encode the hidden variables at historical moments to obtain context features as the speech features of the candidate speech. When pre-training the CPC neural network, the contrastive learning loss function of InfoNCE (Information Noise-Contrastive Estimation) is used to optimize the parameters of the CPC neural network. The initial learning rate of the pre-trained model can be set to 0.001, and the above loss function is used to iterate repeatedly until the contrastive learning loss function of the pre-trained model is stable or the maximum number of iterations is reached, thus completing the pre-trained model training.

[0058] The text content extraction network takes the speech features of the candidate speech as input and the text content features of the candidate speech as output. When performing multi-task training on the importance representation model, a speech recognition fully connected layer can be connected after the text content extraction network. The speech recognition fully connected layer takes the text content features as input and the speech recognition text as output, and the loss of the speech recognition task is constructed based on the speech recognition text and the text label of the corresponding speech as part of the loss of the importance representation model.

[0059] The speaker extraction network takes the speech features of the candidate speech as input and the speaker features of the candidate speech as output. When performing multi-task training on the importance representation model, a speaker classification layer can be connected after the speaker extraction network. The speaker classification layer takes the speaker features as input and the speaker classification result as output. The loss of the speaker recognition task is constructed based on the speaker classification result and the speaker label of the corresponding speech as part of the loss of the importance representation model.

[0060] The importance extraction network takes the speech features, text content features and speaker features of the candidate speech as input, and takes the importance features of the candidate speech as output. That is, the input of the importance extraction network is connected to the output of the pre-trained model, the output of the text content extraction network, and the output of the speaker extraction network respectively. When performing multi-task training on the importance representation model, an important / unimportant classification layer can be connected after the importance extraction network. The important / unimportant classification layer takes the importance features as input and the importance classification result as output. The importance classification result here reflects whether the speech itself is important or not. Based on the importance classification result and the importance label of the corresponding speech, the loss of the importance classification task can be constructed as part of the loss of the importance representation model.

[0061] Accordingly, when performing joint training of multiple tasks for the importance representation model, the loss function applied in the joint training can be expressed as follows: Loss = CTCLoss + CELoss spk + CELoss important-uninmportant Where Loss is the loss function of joint training, and α, β, and γ are all pre-set parameters, such as α can be set to 0.30, β to 0.30, and γ to 0.40. CTCLoss is the loss of the speech recognition task, and CELoss spk The loss for speaker identification task, CELoss important-uninmportant is the loss for the importance classification task.

[0062] Based on the loss of joint training, the entire importance representation model can be iterated repeatedly until the loss of joint training is stable or the maximum number of iterations is reached, thus completing the training of the importance representation model. It can be understood that Figure 3 The speech recognition fully connected layer, speaker classification layer and important / unimportant classification layer in the image processing method all participate in the training of the importance representation model, and after the importance representation model training is completed, the speech recognition fully connected layer, speaker classification layer and important / unimportant classification layer no longer participate in the importance feature extraction for the candidate speech.

[0063] Based on any of the above embodiments, in step 230, the voice recommendation based on the importance feature of the candidate voice includes: Clustering the importance features of each candidate speech to obtain at least one speech cluster; A candidate voice is determined from each voice cluster as a deduplicated voice, and voice recommendation is performed based on the importance features of the deduplicated voice.

[0064] Specifically, due to the non-fixed nature of the collection targets and the diversity of the collection methods, there are often a large number of voices with repeated content in the massive number of candidate voices. If they are not screened, the voices with repeated content will be recommended repeatedly, affecting the recommendation experience.

[0065] To address this problem, in an embodiment of the present invention, feature clustering can be performed for the importance features of each candidate speech extracted. Feature clustering is performed here for the importance features, aiming to divide each candidate speech into several speech clusters, and the importance features of the candidate speech in the same speech cluster are similar to each other, while the importance features of the candidate speech in different speech clusters are different. The algorithm for feature clustering here can be K-means, hierarchical clustering, DBSCAN (Density-Based Spatial Clustering of Applications with Noise, density-based noise application spatial clustering), etc. For example, the importance features of each candidate speech can be clustered based on AP clustering (Affinity Propagation Clustering), and the clustering threshold can be set to 0.9999 or other values ​​for AP clustering. It can be understood that the higher the value of the clustering threshold here, the greater the repetition of the candidate speech in the speech cluster obtained by clustering.

[0066] After clustering is completed, at least one speech cluster can be obtained, and each speech cluster includes at least one candidate speech.

[0067] For each speech cluster, a candidate speech can be selected from the speech cluster as the representative of the speech cluster, that is, as the deduplicated speech of the speech cluster, and according to the importance feature of the deduplicated speech, it is determined whether to recommend the deduplicated speech as the recommended speech.

[0068] It is understandable that there can be a deduplicated voice in each speech cluster, and the reason for selecting only one deduplicated voice in a speech cluster is that the candidate voices in the same speech cluster are all candidate voices with highly similar importance features, that is, the candidate voices in the same speech cluster are almost all repeated voices. In this case, if each candidate voice in the same speech cluster is judged as a recommended voice, it is very likely to cause the problem of repeated recommendation. If only one candidate voice is selected from a speech cluster as a deduplicated voice, the deduplication operation for the repeated voice can be realized, thereby avoiding the problem of repeated recommendation and greatly reducing the workload required for voice recommendation based on importance features.

[0069] Furthermore, a candidate speech is selected from each speech cluster. Specifically, a candidate speech can be randomly selected from each speech cluster as the deduplicated speech, or the candidate speech with the longest duration can be selected from each speech cluster as the deduplicated speech. The text content contained in the deduplicated speech thus selected is the richest in the entire speech cluster. The embodiment of the present invention does not make any specific limitation on this.

[0070] In the method provided in the embodiment of the present invention, by clustering the importance features of each candidate voice, a deduplication operation is implemented for the candidate voices, thereby avoiding repeated recommendations and improving the efficiency of voice recommendation.

[0071] Based on any of the above embodiments, in step 230, performing voice recommendation based on the importance feature of the deduplicated voice includes: Performing voice recommendation based on the similarity between the importance features of the deduplicated voice and the importance features of each registered voice cluster; The registered speech clusters are obtained by clustering the importance features of the registered speech.

[0072] Specifically, in order to implement voice recommendation based on importance features, registered voices may be collected in advance. Here, the registered voices are reference voices for voice recommendation, and the registered voices may be defaulted as important voices that need to be recommended, or may be defaulted as unimportant voices that are not recommended.

[0073] After the registration voice is acquired, the importance feature of the registration voice can be extracted, and feature clustering can be performed on the importance feature of the registration voice, thereby obtaining at least one voice cluster, and the voice cluster composed of the registration voice is recorded as the registration voice cluster. It can be understood that the method of extracting the importance feature of the registration voice and the method of performing feature clustering on the importance feature of the registration voice are consistent with the method of extracting the importance feature of the candidate voice and the method of performing feature clustering on the importance feature of the candidate voice in the above embodiment, and will not be described in detail here.

[0074] After obtaining the registered speech cluster, the importance feature of the registered speech cluster can be determined. Here, for any registered speech cluster, the importance feature of the registered speech cluster can be the average of the importance features of all the registered speech in the registered speech cluster. It can be understood that the importance feature of the registered speech cluster can reflect the common importance feature of a class of registered speech. For example, for a registered speech cluster, the importance feature of the registered speech cluster can be identified as: In the formula, is the importance feature of the kth registered speech cluster. The kth registered speech cluster contains M registered speech in total, among which That is, the importance feature of the mth registered speech among the M registered speech.

[0075] After completing deduplication and obtaining deduplicated speech, the importance features of the deduplicated speech can be compared with the importance features of each registered speech cluster, thereby determining whether to recommend the deduplicated speech as a recommended speech. Here, the similarity between the importance features of the deduplicated speech and the importance features of each registered speech cluster can be calculated. It can be understood that the higher the similarity, the more similar the deduplicated speech is to the registered speech in the registered speech cluster, and the higher the probability of determining whether the deduplicated speech is recommended according to the recommended or not recommended label corresponding to the registered speech in the registered speech cluster. The lower the similarity, the more different the deduplicated speech is from the registered speech in the registered speech cluster, and the lower the probability of determining whether the deduplicated speech is recommended according to the recommended or not recommended label corresponding to the registered speech in the registered speech cluster.

[0076] Based on any of the above embodiments, in step 230, performing voice recommendation based on the similarity between the importance features of the deduplicated voice and the importance features of each registered voice cluster includes: When the maximum value of the similarities between the importance features of the deduplicated speech and the importance features of each registered speech cluster is greater than a preset threshold, and the registered speech cluster corresponding to the maximum value is a recommended speech cluster, speech recommendation is performed on the deduplicated speech.

[0077] Specifically, when there are multiple registered speech clusters, the deduplicated speech needs to calculate the similarity of the importance features with each registered speech cluster respectively, thereby obtaining multiple similarities. The similarity calculation here can be to calculate the cosine similarity between the importance features, or can be implemented by other similarity calculation methods, which is not specifically limited in the embodiment of the present invention.

[0078] When multiple similarities are obtained, the maximum value among the multiple similarities can be obtained, and the maximum value reflects the importance feature similarity between the deduplicated speech and the registered speech cluster most similar to the deduplicated speech. After the maximum value is obtained, the size between the maximum value and the preset threshold can be compared.

[0079] Here, the preset threshold is a preset similarity threshold used to reflect whether the deduplicated speech belongs to the registered speech cluster. If the maximum value is greater than the preset threshold, it can be determined that the deduplicated speech and the registered speech in the registered speech cluster corresponding to the maximum value belong to the same type of speech. In this case, if the registered speech in the registered speech cluster corresponding to the maximum value is an important speech recommended by default, the deduplicated speech can also be determined as a recommended speech and recommended; If the maximum value is greater than the preset threshold, and the registered speech in the registered speech cluster corresponding to the maximum value is a non-important speech that is not recommended, it can also be determined that the deduplicated non-important speech is not recommended.

[0080] In addition, if the maximum value is less than or equal to the preset threshold, it is determined that the deduplicated voice does not belong to any type of registered voice. In this case, it can be assumed that the deduplicated voice is not an important voice and is not recommended.

[0081] Based on any of the above embodiments, in step 220, the step of inputting the candidate speech into the importance characterization model to obtain the importance feature of the candidate speech output by the importance characterization model based on the text content feature and / or speaker feature of the candidate speech includes: Inputting each speech segment of the candidate speech into an importance characterization model, and obtaining importance features of each speech segment output by the importance characterization model based on speech features and / or text content features of each speech segment; Based on the importance features of the speech segments, the importance features of the candidate speech are determined.

[0082] Specifically, considering that some candidate speech has a long duration, if the entire candidate speech is directly input into the importance representation model for importance feature extraction, it will take a long time. In an embodiment of the present invention, the candidate speech can be divided into segments, that is, the candidate speech is divided into multiple speech segments, each of which is a part of the candidate speech. For example, the duration of each speech segment can be between 0.5 seconds and 20 seconds.

[0083] Subsequently, the candidate speech can be input into the importance characterization model in units of speech segments, that is, each speech segment of the candidate speech can be input into the importance characterization model. Thus, the importance characterization model can extract importance features in units of speech segments, thereby outputting importance features of each speech segment.

[0084] In this case, the importance feature of the candidate speech as a whole can be determined by combining the importance feature of each speech segment in the candidate speech. For example, the importance feature of the candidate speech can be determined by the following formula: In the formula, is the importance feature of the i-th candidate speech. The i-th candidate speech is divided into L speech segments, where the importance feature of the i-th speech segment is That is, the importance feature of the i-th candidate speech is the mean of the importance features of all speech segments.

[0085] In the method provided in the embodiment of the present invention, the candidate speech is divided into speech segments to extract importance features, which can improve the extraction efficiency of importance features and further improve the speech recommendation efficiency.

[0086] Based on any of the above embodiments, Figure 4 Schematic diagram of the model training method provided by the present invention. Figure 4 As shown, the method includes: Step 410 , determining training samples for multiple tasks, wherein the multiple tasks include a speech recognition task based on text content features and / or a speaker recognition task based on speaker features, and an importance classification task based on importance features.

[0087] Step 420, performing model training based on training samples of multiple tasks to obtain an importance characterization model, wherein the importance characterization model is used to output importance features of the candidate speech based on text content features and / or speaker features of the candidate speech.

[0088] The model training method provided in the embodiment of the present invention is applied to the model training of the importance representation model. Here, the importance representation model is used to extract the importance features of speech, and the importance features of speech can be used as an indicator to judge whether the speech is important or whether to recommend the speech, which is not specifically limited in the embodiment of the present invention.

[0089] In the process of training the importance representation model, firstly, training samples of multiple tasks may be determined. Here, the multiple tasks include importance classification tasks, speech recognition tasks and / or speaker recognition tasks.

[0090] Among them, the importance classification task is to connect the classification layer after the importance features extracted from the speech to realize the classification of whether the speech is important. The speech recognition task is to connect the fully connected layer of speech recognition after the text content features extracted from the speech to realize speech recognition. The speaker recognition task is to connect the classification layer after the speaker features extracted from the speech to realize the classification of the speaker to which the speech belongs.

[0091] For the importance classification task, speech recognition task and speaker recognition task, respective training samples can be determined respectively. For the importance classification task, its training samples can include speech labeled with whether it is important; for the speech recognition task, its training samples can include speech labeled with corresponding text; for the speaker recognition task, its training samples can include speech labeled with speaker labels.

[0092] When training the importance representation model, the training can be combined with the importance classification task and the speech recognition task. The importance representation model trained in this way can extract the text content features of the input candidate speech, and can extract the importance features of the candidate speech based on the text content features of the candidate speech. The importance features obtained in this way integrate the text content in the candidate speech, and can reflect the importance of the candidate speech from two aspects: text content and whether it is important.

[0093] Alternatively, when training the importance representation model, the training can be combined with the importance classification task and the speaker recognition task, so that the importance representation model trained in this way can extract the speaker features of the input candidate speech, and can extract the importance features of the candidate speech based on the speaker features of the candidate speech. The importance features thus obtained integrate the speaker information in the candidate speech, and can reflect the importance of the candidate speech from two aspects: speaker information and whether it is important.

[0094] Alternatively, when training the importance representation model, the training can be combined with the importance classification task, the speech recognition task and the speaker recognition task. The importance representation model thus trained can extract the text content features of the candidate speech and the speaker features of the candidate speech for the input candidate speech, and can extract the importance features of the candidate speech based on the text content features and speaker features of the candidate speech. The importance features thus obtained integrate the text content and speaker information in the candidate speech, and can reflect the importance of the candidate speech from three aspects: text content, speaker information and whether it is important.

[0095] That is, the importance representation model obtained through multiple task training can realize the extraction of importance features of candidate speech, and the importance features extracted in this way can reflect the importance of the candidate speech from the aspects of text content and / or speaker information, and whether it is important. Compared with the related technology that only considers whether the candidate speech is important from the intention, the measurement of the importance of the candidate speech is more comprehensive and reliable.

[0096] The method provided by the embodiment of the present invention provides a basis for extracting the importance features of candidate voices and performing voice recommendations based on a multi-task importance representation model, thereby achieving end-to-end voice recommendation. There is no cascading of multiple links in the prior art, and the accuracy of each link will affect the voice recommendation effect, which can improve the reliability of voice recommendation. Secondly, the importance representation model obtained through multi-task training can reflect the importance of candidate voices from the aspects of text content and / or speaker information, as well as whether it is important, when extracting importance features, so that the measurement of importance is more comprehensive and reliable, which can further ensure the accuracy of voice recommendation.

[0097] Based on any of the above embodiments, in step 420, the model training is performed based on the training samples of multiple tasks to obtain the importance representation model, including: Extracting sample text content features from speech features of training samples of the speech recognition task based on a text content extraction network in the initial model, and / or extracting sample speaker features from speech features of training samples of the speaker recognition task based on a speaker extraction network in the initial model; Extracting sample importance features from speech features of the training samples of the importance classification task, and sample text content features and / or sample speaker features of the training samples of the importance classification task based on the importance extraction network in the initial model; Based on the speech recognition loss determined by the sample text content features and / or the speaker recognition loss determined by the sample speaker features, and the importance loss determined by the sample importance features, the initial model is iterated on parameters to obtain the importance representation model.

[0098] Specifically, the initial model is the importance representation model before training. The initial model can be understood as a model after parameter initialization. The initial model can include a text content extraction network and / or a speaker extraction network. The text content extraction network is a network used to extract text content features. In the multi-task training of the importance representation model, the text content extraction network can be used to support the training of speech recognition tasks; the speaker extraction network is a network used to extract speaker features. In the multi-task training of the importance representation model, the speaker extraction network can be used to support the training of speaker recognition tasks.

[0099] In specific implementation, for the text content extraction network, the speech features of the speech in the training samples of the speech recognition task can be used as the training input of the text content extraction network. The text content extraction network can extract sample text content features from the speech features.

[0100] For the speaker extraction network, the speech features of the speech in the training sample of the speaker recognition task can be used as the training input of the speaker extraction network. The speaker extraction network can extract sample speaker features from the speech features.

[0101] In addition, the initial model may also include an importance extraction network, the input of which is connected to the output of the text content extraction network and / or the speaker extraction network. The importance extraction network may support the training of importance classification tasks in the multi-task training of the importance representation model.

[0102] In a specific implementation, for the importance extraction network, the speech features of the speech in the training samples of the importance classification task, and the sample text content features and / or sample speaker features of the speech in the training samples of the importance classification task can be used as training inputs of the importance extraction network. The importance extraction network can extract sample importance features from the speech features, and the sample text content features and / or sample speaker features.

[0103] After obtaining the sample text content features for the speech recognition task and / or the sample speaker features for the speaker recognition task, as well as the sample importance features for the importance classification task, the initial model parameters can be repeatedly iteratively trained based on the above features to obtain the importance representation model.

[0104] Specifically, for the sample text content features of the speech recognition task, the speech recognition text can be obtained through the speech recognition fully connected layer, and the speech recognition text can be compared with the text labels in the training samples to obtain the loss of the speech recognition task; for the sample speaker features of the speaker recognition task, the speaker classification result can be obtained through the speaker recognition classification layer, and the speaker classification result can be compared with the speaker label in the training sample to obtain the loss of the speaker recognition task; for the sample importance features of the importance classification task, the importance classification layer can be used to obtain the important or not classification result, and the important or not classification result can be compared with the important or not label in the training sample to obtain the loss of the importance classification task.

[0105] Subsequently, the loss of the speech recognition task, the loss of the speaker recognition task, and the loss of the importance classification task can be combined to construct a loss function for the initial model, and based on this, the parameters of the initial model are iterated to obtain an importance representation model.

[0106] Based on any of the above embodiments, a voice recommendation method may include two stages: model training and model application.

[0107] The model training phase may include the following steps: First, collect training samples: The training samples here include unlabeled business data and labeled speech data.

[0108] Among them, the unlabeled business data is in the form of voice and has a validity period of no less than 1,000 hours; the labeled voice data can include open source voice recognition data, speaker recognition data, and business data labeled as important / unimportant. For example, voice recognition data can include data in major languages ​​such as Chinese and English, and speaker data is required to be no less than 10,000 characters.

[0109] Secondly, preprocess the collected training samples: Specifically, invalid data such as silent noise may be filtered out to leave valid voice segments, and the valid voice segments may be randomly divided into single segments and kept between 0.5 seconds and 20 seconds.

[0110] Next, perform pre-training: The unlabeled training samples after preprocessing can be randomly input into the pre-trained model in batches at a time to train the pre-trained model. The pre-trained model obtained by the training can realize speech feature extraction.

[0111] Subsequently, multiple tasks are trained: You can connect pre-trained models with networks for multiple tasks, such as Figure 3 In the example, the output of the pre-trained network is connected to the input of the text content extraction network, the importance extraction network, and the speaker extraction network. In this case, three batches of labeled training samples, namely, speech recognition data, speaker recognition data, and business data labeled as important / unimportant, can be selected and sent to the pre-trained model, and then the sample text content features, sample speaker features, and sample importance features are extracted through the networks corresponding to the speech recognition task, speaker recognition task, and importance classification task.

[0112] The loss value of joint training is calculated based on the sample text content features, sample speaker features and sample importance features, and based on the loss value, the parameters of the models including the pre-trained model, text content extraction network, importance extraction network and speaker extraction network are iterated to obtain the importance representation model.

[0113] The model application phase may include a registration phase and a recommendation phase, wherein the registration phase includes the following steps: First, collect the registration voice: Here, the registered voices may include voices that are deemed important and need to be recommended by default, and voices that are deemed unimportant and not recommended by default. The registered voices may overlap with the business data labeled as important / unimportant in the training samples collected during the model training phase.

[0114] Secondly, preprocess the registered voice: Specifically, invalid data such as silent noise may be filtered out to leave valid voice segments, and the valid voice segments may be randomly divided into single segments and kept between 0.5 seconds and 20 seconds.

[0115] Next, the importance representation model obtained through training is loaded, and importance features are extracted for the valid speech segments in the registered speech respectively. The importance features of the valid speech segments in the same registered speech are averaged as the importance features of the registered speech.

[0116] Subsequently, clustering is performed on the importance features of the registered speech, thereby obtaining a plurality of registered speech clusters, wherein the importance feature of each registered speech cluster is the mean value of the importance features of all the registered speech clusters.

[0117] The recommendation phase includes the following steps: First, candidate voices to be recommended are obtained, and preprocessed to obtain valid voice segments of the candidate voices.

[0118] Subsequently, the trained importance representation model is loaded, and importance features are extracted for the valid speech segments in the candidate speech respectively, and the importance features of the valid speech segments in the same candidate speech are averaged as the importance features of the candidate speech.

[0119] Next, clustering is performed on the importance features of each candidate speech to obtain at least one speech cluster, and the candidate speech with the longest valid speech segment is selected from each speech cluster as the deduplicated speech.

[0120] Finally, a preset threshold is set to calculate the similarity between the importance feature of each deduplicated speech and the importance feature of each registered speech cluster obtained in the registration phase. For each deduplicated speech, the maximum value of the similarity is compared with the preset threshold. If the maximum value is greater than or equal to the preset threshold, and the registered speech in the registered speech cluster corresponding to the maximum value is an important speech that needs to be recommended, then the deduplicated speech is recommended as a recommended speech; otherwise, the deduplicated speech is regarded as an unimportant speech and is not recommended.

[0121] The voice recommendation method provided by the embodiment of the present invention proposes an importance representation model obtained by training based on multiple tasks. When representing the importance feature, the model integrates information such as text content, speaker information, and whether it is important, which can effectively improve the robustness of the importance feature. And because the importance representation model realizes the mapping from voice to importance features, there is no need to perform the process of voice transcription and then intention recognition in this process, and it also avoids the situation where multiple links are cascaded and the accuracy of each link will affect the effect of voice recommendation.

[0122] In addition, in the speech recommendation method provided in the embodiment of the present invention, clustering is performed on the importance features of each candidate speech, thereby obtaining at least one speech cluster, and the candidate speech with the longest valid speech segment is selected from each speech cluster as the deduplicated speech, thereby realizing the deduplication operation for the candidate speech and avoiding the waste of computing power on repeated speech in speech recommendation.

[0123] The voice recommendation device provided by the present invention is described below. The voice recommendation device described below and the voice recommendation method described above can be referred to each other.

[0124] Figure 5 is a structural diagram of the voice recommendation device provided by the present invention, such as Figure 5 As shown, the device comprises: A speech acquisition unit 510, used to acquire candidate speech; The feature extraction unit 520 is used to input the candidate speech into an importance characterization model to obtain the importance feature of the candidate speech output by the importance characterization model based on the text content feature and / or speaker feature of the candidate speech; the importance characterization model is obtained through training of multiple tasks, the multiple tasks including a speech recognition task based on the text content feature and / or a speaker recognition task based on the speaker feature, and an importance classification task based on the importance feature; The voice recommendation unit 530 is used to make voice recommendations based on the importance features of the candidate voices.

[0125] The speech recommendation device provided by the embodiment of the present invention extracts the importance features of candidate speech through the importance representation model obtained by multiple task training, and performs speech recommendation based on the importance features. Firstly, end-to-end speech recommendation is realized, and there is no situation in the prior art where multiple links are cascaded and the accuracy of each link affects the speech recommendation effect, which can improve the reliability of speech recommendation; secondly, the importance features can reflect the importance of candidate speech from aspects such as text content and / or speaker information, and whether it is important, so that the measurement of importance is more comprehensive and reliable, which can further ensure the accuracy of speech recommendation.

[0126] Based on any of the above embodiments, the feature extraction unit is specifically used for: Extracting text content features of the candidate speech based on the text content extraction network in the importance representation model and the speech features of the candidate speech, and / or extracting speaker features of the candidate speech based on the speaker extraction network in the importance representation model and the speech features of the candidate speech; Importance features of the candidate speech are extracted based on the importance extraction network in the importance representation model, the speech features, and the text content features and / or the speaker features.

[0127] Based on any of the above embodiments, the voice recommendation unit is specifically used for: Clustering the importance features of each candidate speech to obtain at least one speech cluster; A candidate voice is determined from each voice cluster as a deduplicated voice, and voice recommendation is performed based on the importance features of the deduplicated voice.

[0128] Based on any of the above embodiments, the voice recommendation unit is specifically used for: Performing voice recommendation based on the similarity between the importance features of the deduplicated voice and the importance features of each registered voice cluster; The registered speech clusters are obtained by clustering the importance features of the registered speech.

[0129] Based on any of the above embodiments, the voice recommendation unit is specifically used for: When the maximum value of the similarities between the importance features of the deduplicated speech and the importance features of each registered speech cluster is greater than a preset threshold, and the registered speech cluster corresponding to the maximum value is a recommended speech cluster, speech recommendation is performed on the deduplicated speech.

[0130] Based on any of the above embodiments, the feature extraction unit is specifically used for: Inputting each speech segment of the candidate speech into an importance characterization model, and obtaining importance features of each speech segment output by the importance characterization model based on speech features and / or text content features of each speech segment; Based on the importance features of the speech segments, the importance features of the candidate speech are determined.

[0131] The model training device provided by the present invention is described below. The model training device described below and the model training method described above can be referenced to each other.

[0132] Figure 6 It is a structural schematic diagram of the model training device provided by the present invention, such as Figure 6 As shown, the device comprises: A sample acquisition unit 610 is used to determine training samples for multiple tasks, wherein the multiple tasks include a speech recognition task based on text content features and / or a speaker recognition task based on speaker features, and an importance classification task based on importance features; The model training unit 620 is used to perform model training based on training samples of multiple tasks to obtain an importance representation model, wherein the importance representation model is used to output the importance features of the candidate speech based on the text content features and / or speaker features of the candidate speech.

[0133] The model training device provided by the embodiment of the present invention trains the importance representation model through multiple tasks, thereby providing a basis for extracting the importance features of candidate voices and performing voice recommendations based on this, thereby achieving end-to-end voice recommendation. There is no situation in the prior art where multiple links are cascaded and the accuracy of each link affects the voice recommendation effect, which can improve the reliability of voice recommendation; secondly, the importance representation model obtained through multiple task training can reflect the importance of candidate voices from aspects such as text content and / or speaker information, and whether it is important when extracting importance features, making the measurement of importance more comprehensive and reliable, and can further ensure the accuracy of voice recommendation.

[0134] Based on any of the above embodiments, the model training unit is specifically used for: Extracting sample text content features from speech features of training samples of the speech recognition task based on a text content extraction network in the initial model, and / or extracting sample speaker features from speech features of training samples of the speaker recognition task based on a speaker extraction network in the initial model; Extracting sample importance features from speech features of the training samples of the importance classification task, and sample text content features and / or sample speaker features of the training samples of the importance classification task based on the importance extraction network in the initial model; Based on the speech recognition loss determined by the sample text content features and / or the speaker recognition loss determined by the sample speaker features, and the importance loss determined by the sample importance features, the initial model is iterated on parameters to obtain the importance representation model.

[0135] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the voice recommendation method, which includes: Obtain candidate voices; Inputting the candidate speech into an importance characterization model, obtaining the importance feature of the candidate speech output by the importance characterization model based on the text content feature and / or speaker feature of the candidate speech; the importance characterization model is obtained through training of multiple tasks, the multiple tasks including a speech recognition task based on the text content feature and / or a speaker recognition task based on the speaker feature, and an importance classification task based on the importance feature; Based on the importance features of the candidate voices, voice recommendations are performed.

[0136] Alternatively, the processor 710 may call the logic instructions in the memory 730 to execute the model training method, which includes: Determining training samples for a plurality of tasks, wherein the plurality of tasks include a speech recognition task based on text content features and / or a speaker recognition task based on speaker features, and an importance classification task based on importance features; Model training is performed based on training samples of multiple tasks to obtain an importance characterization model, wherein the importance characterization model is used to output importance features of the candidate speech based on text content features and / or speaker features of the candidate speech.

[0137] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0138] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can perform the voice recommendation method provided by the above methods, the method includes: Obtain candidate voices; Inputting the candidate speech into an importance characterization model, obtaining the importance feature of the candidate speech output by the importance characterization model based on the text content feature and / or speaker feature of the candidate speech; the importance characterization model is obtained through training of multiple tasks, the multiple tasks including a speech recognition task based on the text content feature and / or a speaker recognition task based on the speaker feature, and an importance classification task based on the importance feature; Based on the importance features of the candidate voices, voice recommendations are performed.

[0139] Alternatively, the computer can execute the model training method provided by the above methods, which includes: Determining training samples for a plurality of tasks, wherein the plurality of tasks include a speech recognition task based on text content features and / or a speaker recognition task based on speaker features, and an importance classification task based on importance features; Model training is performed based on training samples of multiple tasks to obtain an importance characterization model, wherein the importance characterization model is used to output importance features of the candidate speech based on text content features and / or speaker features of the candidate speech.

[0140] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for voice recommendation provided by the above methods is implemented, and the method includes: Obtain candidate voices; Inputting the candidate speech into an importance characterization model, obtaining the importance feature of the candidate speech output by the importance characterization model based on the text content feature and / or speaker feature of the candidate speech; the importance characterization model is obtained through training of multiple tasks, the multiple tasks including a speech recognition task based on the text content feature and / or a speaker recognition task based on the speaker feature, and an importance classification task based on the importance feature; Based on the importance features of the candidate voices, voice recommendations are performed.

[0141] Alternatively, when the computer program is executed by a processor, the model training method provided by the above methods is implemented, and the method includes: Determining training samples for a plurality of tasks, wherein the plurality of tasks include a speech recognition task based on text content features and / or a speaker recognition task based on speaker features, and an importance classification task based on importance features; Model training is performed based on training samples of multiple tasks to obtain an importance characterization model, wherein the importance characterization model is used to output importance features of the candidate speech based on text content features and / or speaker features of the candidate speech.

[0142] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voice recommendation method, characterized in that: include: Obtain candidate voices; Inputting the candidate speech into an importance characterization model to obtain an importance feature of the candidate speech output by the importance characterization model based on text content features and / or speaker features of the candidate speech; The importance representation model is obtained through training of multiple tasks, wherein the multiple tasks include a speech recognition task based on the text content feature and / or a speaker recognition task based on the speaker feature, and an importance classification task based on the importance feature; Based on the importance features of the candidate voices, voice recommendations are performed.

2. The voice recommendation method according to claim 1, characterized in that: The step of inputting the candidate speech into an importance characterization model to obtain an importance feature of the candidate speech output by the importance characterization model based on a text content feature and / or a speaker feature of the candidate speech comprises: Extracting text content features of the candidate speech based on the text content extraction network in the importance representation model and the speech features of the candidate speech, and / or extracting speaker features of the candidate speech based on the speaker extraction network in the importance representation model and the speech features of the candidate speech; Importance features of the candidate speech are extracted based on the importance extraction network in the importance representation model, the speech features, and the text content features and / or the speaker features.

3. The voice recommendation method according to claim 1, characterized in that: The voice recommendation based on the importance feature of the candidate voices includes: Clustering the importance features of each candidate speech to obtain at least one speech cluster; A candidate voice is determined from each voice cluster as a deduplicated voice, and voice recommendation is performed based on the importance features of the deduplicated voice.

4. The voice recommendation method according to claim 3, characterized in that: The performing voice recommendation based on the importance feature of the deduplicated voice includes: Performing voice recommendation based on the similarity between the importance features of the deduplicated voice and the importance features of each registered voice cluster; The registered speech clusters are obtained by clustering the importance features of the registered speech.

5. The voice recommendation method according to claim 4, characterized in that: The voice recommendation based on the similarity between the importance features of the deduplicated voice and the importance features of each registered voice cluster includes: When the maximum value of the similarities between the importance features of the deduplicated speech and the importance features of each registered speech cluster is greater than a preset threshold, and the registered speech cluster corresponding to the maximum value is a recommended speech cluster, speech recommendation is performed on the deduplicated speech.

6. The voice recommendation method according to any one of claims 1 to 5, characterized in that: The step of inputting the candidate speech into an importance characterization model to obtain an importance feature of the candidate speech output by the importance characterization model based on a text content feature and / or a speaker feature of the candidate speech comprises: Inputting each speech segment of the candidate speech into an importance characterization model, and obtaining importance features of each speech segment output by the importance characterization model based on speech features and / or text content features of each speech segment; Based on the importance features of the speech segments, the importance features of the candidate speech are determined.

7. A model training method, characterized in that: include: Determining training samples for a plurality of tasks, wherein the plurality of tasks include a speech recognition task based on text content features and / or a speaker recognition task based on speaker features, and an importance classification task based on importance features; Model training is performed based on training samples of multiple tasks to obtain an importance characterization model, wherein the importance characterization model is used to output importance features of the candidate speech based on text content features and / or speaker features of the candidate speech.

8. The model training method according to claim 7, characterized in that: The training samples based on multiple tasks are used to perform model training to obtain an importance representation model, including: Extracting sample text content features from speech features of training samples of the speech recognition task based on a text content extraction network in the initial model, and / or extracting sample speaker features from speech features of training samples of the speaker recognition task based on a speaker extraction network in the initial model; Extracting sample importance features from speech features of the training samples of the importance classification task, and sample text content features and / or sample speaker features of the training samples of the importance classification task based on the importance extraction network in the initial model; Based on the speech recognition loss determined by the sample text content features and / or the speaker recognition loss determined by the sample speaker features, and the importance loss determined by the sample importance features, the initial model is iterated on parameters to obtain the importance representation model.

9. A voice recommendation device, characterized in that: include: A speech acquisition unit, used for acquiring candidate speech; A feature extraction unit, configured to input the candidate speech into an importance characterization model, and obtain an importance feature of the candidate speech output by the importance characterization model based on a text content feature and / or a speaker feature of the candidate speech; The importance representation model is obtained through training of multiple tasks, wherein the multiple tasks include a speech recognition task based on the text content feature and / or a speaker recognition task based on the speaker feature, and an importance classification task based on the importance feature; The voice recommendation unit is used to make voice recommendations based on the importance features of the candidate voices.

10. A model training device, characterized in that: include: A sample acquisition unit, used to determine training samples for a plurality of tasks, wherein the plurality of tasks include a speech recognition task based on text content features and / or a speaker recognition task based on speaker features, and an importance classification task based on importance features; The model training unit is used to perform model training based on training samples of multiple tasks to obtain an importance representation model, wherein the importance representation model is used to output the importance features of the candidate speech based on the text content features and / or speaker features of the candidate speech.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the voice recommendation method as described in any one of claims 1 to 6, or the model training method as described in claim 7 or 8 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the voice recommendation method according to any one of claims 1 to 6, or the model training method according to claim 7 or 8 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the voice recommendation method according to any one of claims 1 to 6, or the model training method according to claim 7 or 8 is implemented.

Images