Voice content recognition method and device, voice synthesis method and electronic equipment

Abstract

The embodiment of the invention provides a voice content recognition method and device, a voice synthesis method and electronic equipment, and belongs to the field of artificial intelligence and financial science and technology. The voice content recognition method comprises the following steps: acquiring source voice data; encoding the source voice data through the teacher model to obtain voice encoding reference data; encoding the source voice data through the original student model to obtain voice encoding content data; performing hidden content prediction on the voice coding content data to obtain voice coding prediction data; carrying out loss calculation on the voice coding reference data and the voice coding prediction data to obtain coding loss data; performing parameter adjustment on the original student model according to the coding loss data to obtain a voice content recognition model; and performing coding processing on pre-acquired target voice data through the voice content recognition model to obtain target content data. According to the embodiment of the invention, the complexity of a voice content recognition model can be reduced, and memory occupation is reduced.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a method and device for voice content recognition, a voice synthesis method and an electronic device. Background Art

[0002] At present, the application of intelligent voice customer service in the financial field is becoming more and more widespread. Taking the insurance business as an example, the intelligent voice customer service uses a preset voice content recognition model to identify the voice content of insurance customers' inquiries, and searches for the corresponding answer from the preset voice library based on the voice content of the insurance customers and generates a voice answer in combination with the target timbre. In the related technology, the traditional voice content recognition model is relatively complex and occupies a large amount of device memory, which is not convenient for lightweight promotion and application. Therefore, how to reduce the complexity of the voice content recognition model and reduce memory usage has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a speech content recognition method and device, a speech synthesis method and an electronic device, which predicts hidden content and compares it with the output data of a teacher model, adjusts the model parameters according to the comparison results, and thus learns the hidden representation of the teacher model. While ensuring the content extraction capability of the model, the complexity of the model structure is significantly reduced and the model memory usage is reduced.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides a method for voice content recognition, the method comprising:

[0005] Obtain source voice data;

[0006] The source speech data is encoded by a preset teacher model to obtain speech encoding reference data;

[0007] The source speech data is encoded by a preset original student model to obtain speech encoding content data; wherein the original student model is constructed based on a part of the encoding layer of the teacher model;

[0008] Performing hidden content prediction on the speech coding content data to obtain speech coding prediction data;

[0009] Performing loss calculation on the speech coding reference data and the speech coding prediction data to obtain coding loss data;

[0010] Adjusting parameters of the original student model according to the coding loss data to obtain a speech content recognition model;

[0011] The target voice data acquired in advance is encoded by the voice content recognition model to obtain the target content data.

[0012] In some embodiments, the teacher model includes a teacher encoding layer and a teacher prediction layer, and any two layers of the teacher prediction layers are not connected; the step of encoding the source speech data through the preset teacher model to obtain speech coding reference data includes:

[0013] Masking the source speech data through the teacher coding layer to obtain training mask data;

[0014] The training mask data is encoded through the teacher prediction layer to obtain the speech encoding reference data.

[0015] In some embodiments, before the step of performing hidden content prediction on the speech coding content data to obtain speech coding prediction data, the speech content recognition method further includes:

[0016] Obtain the number of layers of the teacher prediction layer to obtain the reference prediction layer number;

[0017] Constructing a student prediction layer in the original student model according to the reference prediction layer number; wherein the number of layers of the student prediction layer is equal to the number of layers of the teacher prediction layer.

[0018] In some embodiments, the step of performing hidden content prediction on the speech coding content data to obtain speech coding prediction data includes:

[0019] Performing mask processing on the speech coding content data to obtain speech mask data;

[0020] The speech mask data is subjected to content prediction by the student prediction layer to obtain the speech coding prediction data.

[0021] In some embodiments, before the step of encoding the source voice data using a preset original student model to obtain voice encoding content data, the voice content recognition method further includes:

[0022] Constructing the original student model specifically includes:

[0023] Obtaining parameters of the teacher coding layer to obtain reference parameters;

[0024] The reference parameters are transferred to the student coding layer of the preset speech training model to obtain the original student model.

[0025] In some embodiments, after the step of adjusting the parameters of the original student model according to the coding loss data to obtain the speech content recognition model, the speech content recognition method further includes:

[0026] Updating the speech content recognition model specifically includes:

[0027] Selecting a reference coding layer from the student coding layers; wherein the reference coding layer is the bottom layer of the student coding layers;

[0028] Get training speech data;

[0029] Performing encoding processing on the training speech data through the reference encoding layer to obtain reference encoding data;

[0030] The training speech data is encoded by the student encoding layer to obtain training encoding data; wherein the training encoding data is encoding data of other layers in the student encoding layer except the reference encoding layer;

[0031] Perform loss calculation based on the reference encoding data and the training encoding data to obtain updated loss data;

[0032] Parameters of the speech content recognition model are updated according to the updated loss data.

[0033] To achieve the above object, a second aspect of an embodiment of the present application provides a speech synthesis method, the speech synthesis method comprising:

[0034] Acquire target content data; wherein the target content data is obtained by the voice content recognition method described in the first aspect above;

[0035] Extracting timbre features from a preset timbre feature library according to a preset target timbre type to obtain a target timbre feature;

[0036] Speech synthesis is performed according to the target content data and the target timbre characteristics to obtain target speech data.

[0037] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application provides a speech content recognition device, the device comprising:

[0038] A data acquisition module, used to acquire source voice data;

[0039] A first encoding module, used for encoding the source speech data through a preset teacher model to obtain speech encoding reference data;

[0040] A second encoding module, used for encoding the source speech data through a preset original student model to obtain speech encoding content data; wherein the original student model is constructed based on a part of the encoding layer of the teacher model;

[0041] A content prediction module, used for performing hidden content prediction on the speech coding content data to obtain speech coding prediction data;

[0042] A loss calculation module, used for performing loss calculation on the speech coding reference data and the speech coding prediction data to obtain coding loss data;

[0043] A parameter adjustment module, used for adjusting the parameters of the original student model according to the coding loss data to obtain a speech content recognition model;

[0044] The content recognition module is used to encode the pre-acquired target voice data through the voice content recognition model to obtain target content data.

[0045] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the speech content recognition method described in the first aspect or the speech synthesis method described in the second aspect.

[0046] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the speech content recognition method described in the first aspect or the speech synthesis method described in the second aspect.

[0047] The speech content recognition method and device, speech synthesis method and electronic device proposed in the present application predict hidden content and compare it with the output data of the teacher model, and adjust the model parameters according to the comparison results, thereby learning the hidden representation of the teacher model. While ensuring the content extraction capability of the model, it significantly reduces the complexity of the model structure and reduces the model memory usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of a method for voice content recognition provided by an embodiment of the present application;

[0049] Figure 2 It is a structural schematic diagram of an embodiment of the teacher model and the original student model of the present application;

[0050] Figure 3 yes Figure 1 A flowchart of an embodiment of step S102;

[0051] Figure 4 is a flow chart of a voice content recognition method provided by another embodiment of the present application;

[0052] Figure 5 yes Figure 1 A flowchart of an embodiment of step S104;

[0053] Figure 6 is a flow chart of a method for voice content recognition provided by another embodiment of the present application;

[0054] Figure 7 is a flow chart of a voice content recognition method provided by another embodiment of the present application;

[0055] Figure 8 is a flow chart of a speech synthesis method provided in an embodiment of the present application;

[0056] Fig. 9 is a structural diagram of a speech content recognition device provided in an embodiment of the present application;

[0057] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] First, some nouns involved in this application are analyzed:

[0062] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0063] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0064] Speech Recognition (Automatic Speech Recognition, ASR): The goal of speech recognition is to convert the vocabulary content in human speech into computer-readable input, such as keystrokes, binary codes or character sequences. Speech recognition is an interdisciplinary subject involving acoustics, linguistics, information theory, signal processing, computer science, pattern recognition and other disciplines. It has broad application prospects, such as voice retrieval, command control, automatic customer service, machine translation, etc. Speech recognition technology is an important branch of artificial intelligence and a key link in human-computer natural interaction technology.

[0065] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0066] Cluster analysis: Cluster analysis refers to the analytical process of grouping a collection of physical or abstract objects into multiple classes consisting of similar objects. The goal of cluster analysis is to collect data for classification based on similarity. Clustering originates from many fields, including mathematics, computer science, statistics, biology, and economics. Cluster analysis techniques have been developed in different application fields and are used to describe data, measure the similarity between different data sources, and classify data sources into different clusters. From a statistical point of view, cluster analysis is a method of simplifying data through data modeling. Traditional statistical cluster analysis methods include systematic clustering, decomposition method, joining method, dynamic clustering method, ordered sample clustering, overlapping clustering and fuzzy clustering. Cluster analysis tools using algorithms such as k-means and k-centers have been widely used in many statistical analysis software, such as SPSS, SAS, etc.

[0067] At present, the application of intelligent voice customer service in the financial field is becoming more and more widespread. Taking the insurance business as an example, the intelligent voice customer service uses a preset voice content recognition model to identify the voice content of insurance customers' inquiries, and searches for the corresponding answer from the preset voice library based on the voice content of the insurance customers and generates a voice answer in combination with the target timbre. In the related technology, the traditional voice content recognition model is relatively complex and occupies a large amount of device memory, which is not convenient for lightweight promotion and application. Therefore, how to reduce the complexity of the voice content recognition model and reduce memory usage has become a technical problem that needs to be solved urgently.

[0068] Based on this, the embodiments of the present application provide a speech content recognition method and device, a speech synthesis method and an electronic device, and further propose a computer-readable storage medium, aiming to reduce the complexity of the speech content recognition model and reduce memory usage.

[0069] The speech content recognition method and device, speech synthesis method, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the speech content recognition method in the embodiments of the present application is described.

[0070] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0071] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0072] The speech content recognition method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The speech content recognition method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech content recognition method, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0075] See also Figure 1 and Figure 2 , Figure 1 is an optional flowchart of a method for voice content recognition according to an embodiment of the present application. Figure 2 It is a structural diagram of an embodiment of the teacher model and the original student model of the present application. The speech content recognition method of the embodiment of the present application includes but is not limited to steps S101 to S107:

[0076] Step S101, obtaining source voice data;

[0077] Step S102, encoding the source speech data using a preset teacher model to obtain speech encoding reference data;

[0078] Step S103, encoding the source speech data through a preset original student model to obtain speech coding content data; wherein the original student model is constructed based on a part of the coding layer of the teacher model;

[0079] Step S104, performing hidden content prediction on the speech coding content data to obtain speech coding prediction data;

[0080] Step S105, performing loss calculation on the speech coding reference data and the speech coding prediction data to obtain coding loss data;

[0081] Step S106, adjusting the parameters of the original student model according to the coding loss data to obtain a speech content recognition model;

[0082] Step S107: encoding the pre-acquired target voice data through a voice content recognition model to obtain target content data.

[0083] In steps S101 to S107 shown in the embodiment of the present application, after obtaining the source speech data, it is encoded by a preset teacher model and an original student model respectively, and then the hidden content is predicted according to the encoded output data of the original student model, and the prediction result of the hidden content is compared with the encoded output data of the teacher model, and the model parameters are adjusted according to the comparison result, so as to learn the hidden representation of the teacher model (that is, learn the content extraction capability of the teacher model), which significantly reduces the complexity of the model structure and reduces the model memory usage while ensuring the content extraction capability of the model.

[0084] The speech content recognition method of the embodiment of the present application can learn and extract speech content representation from a piece of speech, and the extracted speech content representation can be used for downstream speech synthesis tasks.

[0085] In step S101 of some embodiments, the source voice data is voice data recorded according to set content, and is used to assist in building a voice content recognition model.

[0086] In step S102 of some embodiments, the teacher model is provided with a teacher sampling layer, which is used to sample the source speech data, and after sampling, other layers of the teacher model perform encoding processing to obtain speech coding reference data. In the embodiment of the present application, the teacher model is a model that has been verified to have excellent performance in speech content recognition. The model structure of the teacher model is usually complex and occupies a large amount of memory, so it is not suitable for lightweight promotion and application.

[0087] In an illustrative embodiment, the teacher model of the embodiment of the present application can be a model based on the HuBert network. The HuBert model is a model based on self-supervised speech learning. It uses unlabeled speech data to learn speech representation, uses k-means clustering analysis to provide labels for the loss function used by the BERT model in the NLP field, and learns meaningful speech content representation through a mask prediction mechanism. The main body of the classic HuBert model includes a CNN sampling layer and a Transformer layer. The CNN sampling layer is responsible for sampling and dimensionality increase. Before entering the Transformer layer, the sampled data is randomly masked, and the labels of some data units will be hidden; through the processing of the Transformer layer, the label of each data unit is predicted to obtain a predicted label sequence, and the learning ability of the network is trained by calculating the label loss.

[0088] In step S103 of some embodiments, the original student model is provided with a student sampling layer, and the student sampling layer is used to sample the source speech data, and after sampling, the original student model is encoded by other layers to obtain speech coding content data. It should be noted that the original student model is constructed based on a part of the coding layer of the teacher model, thereby inheriting the speech content recognition performance of the teacher model to a certain extent, and the model structure is simpler.

[0089] In step S104 of some embodiments, hidden content prediction is performed on the speech coding content data, thereby obtaining speech coding prediction data that can characterize the prediction performance of the original student model.

[0090] In step S105 of some embodiments, loss calculation is performed on the speech coding reference data and the speech coding prediction data to obtain coding loss data that measures the difference between the speech content extraction performance of the teacher model and the speech content extraction performance of the original student model.

[0091] In step S106 of some embodiments, the parameters of the original student model are adjusted according to the coding loss data, so as to minimize the gap between the speech content extraction performance of the original student model and the language content extraction performance of the teacher model. It should be noted that each time the parameters of the original student model are adjusted according to the coding loss data, the source speech data is encoded again by the adjusted original student model to obtain new speech coding content data, and then the hidden content prediction is performed on the new speech coding content data to obtain new speech coding prediction data, and the loss calculation is performed again according to the new speech coding prediction data and the speech coding reference data to update the coding loss data, and then the original student model is updated again according to the updated coding loss data until the coding loss data reaches the minimum value, and finally the speech content recognition model is obtained.

[0092] In step S107 of some embodiments, the pre-acquired target voice data includes online voice data (such as customer service phone recordings, customer service software chat voices, etc.) and offline real-time voice data (such as offline customer service terminal real-time recordings). Online voice data is usually stored in a relay device or server, and the voice data must be called from the relay device or server before voice content extraction can be performed. Offline real-time voice data can be directly used for voice content extraction after being collected and recorded by a customer service terminal equipped with a voice content recognition model.

[0093] Taking fund business as an example, customers inquire about various matters of fund products through telephone or voice chat of customer service software. For example, the content of the customer's inquiry is: "What are the methods of raising, operating and trading of Fund A, and what are its investment philosophy and legal form?" The intelligent voice customer service system, with the customer's authorization, retrieves the telephone recording or chat voice of the customer's consultation from the server or related relay equipment, and extracts the target voice content therein, so that the subsequent intelligent customer service system can reply to the customer's consultation content according to the extracted target voice content and the preset database. For example, the intelligent customer service system extracts keywords from the target voice content and searches for the corresponding answers in the preset database, and generates a reply based on the format of the customer's target voice content: "Fund A's fundraising method is public offering, its operation method is closed-end, its trading method is on-site trading, its investment philosophy is active investment, and its legal form is contractual."

[0094] Taking insurance business as an example, insured customers can view and consult various detailed information of insurance products through intelligent voice customer service terminals in offline customer service halls. For example, the content of the insured customer's consultation is: "What type of B insurance is, what is the reimbursement scope, reimbursement ratio and reimbursement upper limit", and the intelligent voice customer service terminal directly records the insured customer's consultation voice through a preset microphone, and extracts the target voice content, so as to facilitate the subsequent voice reply to the insured customer through the preset speaker according to the extracted target voice content. For example, the intelligent voice customer service terminal extracts keywords from the target voice content and searches for the corresponding answers through the preset database, and generates a reply based on the format of the customer's target voice content: "B insurance is medical insurance, the reimbursement scope is self-paid, the reimbursement ratio is 50%, and the reimbursement upper limit is 300,000".

[0095] In some embodiments, see Figure 2 and Figure 3 , the teacher model includes a teacher encoding layer and a teacher prediction layer, and any two teacher prediction layers are not connected. Figure 2As shown, the teacher sampling layer of the teacher model is used to sample the source speech data, and after the sampling is completed, the teacher coding layer and the teacher prediction layer of the teacher model are used to encode and process to obtain speech coding reference data. It should be noted that the teacher coding layer is used to assist in the construction of the original student model, and the teacher coding layer and the teacher prediction layer are both network layers with encoders, and their number of layers can be adjusted as needed. In an embodiment of the present application, the teacher model can be a model based on the HuBert network, the teacher sampling layer corresponds to the CNN sampling layer of the HuBert model, and the teacher coding layer and the teacher prediction layer correspond to the Transformer layer of the HuBert model. The performance of adjacent Transformer layers is similar, therefore, when setting the teacher prediction layer, any two layers are set to be spaced apart from each other to improve the performance diversity of the teacher model represented by the teacher prediction layer, so as to facilitate the subsequent original student model to learn the teacher model performance more comprehensively.

[0096] Step S102 of the voice content recognition method of the present application embodiment includes but is not limited to step S201 and step S202:

[0097] Step S201, masking the source speech data through the teacher coding layer to obtain training mask data;

[0098] Step S202, encoding the training mask data through the teacher prediction layer to obtain speech coding reference data.

[0099] Steps S201 and S202 shown in the embodiment of the present application mask the source speech data, and encode and predict the training mask data obtained after masking through the teacher prediction layer, so as to obtain speech coding reference data representing the hidden representation of the teacher model, so that the subsequent original student model can improve the model performance by learning the hidden representation of the teacher model. It should be noted that the hidden representation is the relevant content mapping of the teacher model to generate the corresponding prediction data based on the masked speech data. The speech coding prediction data output by different teacher prediction layers are different.

[0100] In step S201 of some embodiments, the teacher coding layer is provided with a masker, and the collected source voice data is masked by a preset masker to selectively hide the voice content features of part of the data.

[0101] In step S202 of some embodiments, the teacher prediction layer predicts speech content features of the hidden part of the data based on the unhidden data in the training mask data, and obtains speech coding reference data representing the hidden representation of the teacher prediction layer in the teacher model.

[0102] For an illustrative embodiment, see Figure 2, the teacher model of the embodiment of the present application is provided with a total of 14 Transformer layers (some numbered Transformer layers are not configured as teacher coding layers or teacher prediction layers, and therefore are not shown in the figure), the teacher coding layers are configured as: Transformer layer 1, Transformer layer 2, Transformer layer 3 and Transformer layer 4, and the teacher prediction layer of the teacher model is configured as: Transformer layer 6, Transformer layer 10 and Transformer layer 14, wherein Transformer layer 14 is the last layer of the Transformer layer. In other embodiments, Transformer layer 4 can also be set as a common layer of the teacher coding layer and the teacher prediction layer, so as to reduce the difference between the speech coding reference data output by the teacher prediction layer and the speech coding content data output by the original student model constructed according to the teacher coding layer, speed up the subsequent loss calculation of the speech coding reference data and the speech coding prediction data, and adjust the model parameters according to the loss calculation to obtain the speech content recognition model.

[0103] In some embodiments, see Figure 2 and Figure 4 Before step S104, the voice content recognition method of the embodiment of the present application further includes but is not limited to step S301 and step S302:

[0104] Step S301, obtaining the number of teacher prediction layers to obtain the reference prediction layer number;

[0105] Step S302, constructing a student prediction layer in the original student model according to the reference prediction layer number; wherein the number of student prediction layers is equal to the number of teacher prediction layers.

[0106] In steps S301 to S302 shown in the embodiment of the present application, by obtaining the number of teacher prediction layers and constructing student prediction layers in the original student model according to the number of teacher prediction layers, the student prediction layers correspond one to one with the teacher prediction layers, so that the original student model can learn the hidden representation of the teacher model more comprehensively through the student prediction layers, thereby improving the speech content recognition performance of the original student model.

[0107] In step S301 of some embodiments, see Figure 2 ,The teacher prediction layer configuration of the teacher model is: Transformer layer 6, Transformer layer 10 and Transformer layer 14, a total of 3 layers.

[0108] In step S302 of some embodiments, see Figure 2, three student prediction layers are constructed according to the number of teacher prediction layers, where student prediction layer 1 corresponds to Transformer layer 6, student prediction layer 2 corresponds to Transformer layer 10, and student prediction layer 3 corresponds to Transformer layer 14.

[0109] In some embodiments, see Figure 5 Step S104 of the voice content recognition method of the present application embodiment includes but is not limited to step S401 and step S402:

[0110] Step S401, masking the speech coding content data to obtain speech mask data;

[0111] Step S402, performing content prediction on the speech mask data through the student prediction layer to obtain speech coding prediction data.

[0112] Steps S401 and S402 shown in the embodiment of the present application perform masking on the speech coding content data, and perform coding prediction on the masked speech mask data through the student prediction layer, thereby obtaining speech coding reference data that characterizes the hidden content prediction performance of the original student model, so as to facilitate subsequent comparison and adjustment with the speech coding reference data output by the teacher model, thereby learning the hidden representation of the teacher model to improve the performance of the original student model.

[0113] In step S401 of some embodiments, the original student model is further provided with a mask module, which masks the speech coding content data through a preset mask module to selectively hide the speech content features of part of the data (such as content labels, indexes, etc.).

[0114] In step S402 of some embodiments, the student prediction layer predicts speech content features of the hidden part of the data based on the data not hidden in the speech mask data, and obtains speech coding reference data that characterizes the hidden content prediction performance of the original student model.

[0115] For an illustrative embodiment, see Figure 2, the implementation process of step S105 of the speech content recognition method of the embodiment of the present application is as follows: calculate the loss value L1 of the speech coding prediction data output by the student prediction layer 1 and the language coding reference data output by the Transformer layer 6; calculate the loss value L2 of the speech coding prediction data output by the student prediction layer 2 and the language coding reference data output by the Transformer layer 10; calculate the loss value L3 of the speech coding prediction data output by the student prediction layer 3 and the language coding reference data output by the Transformer layer 14; calculate the coding loss data Lcon between the original student model and the teacher model, where Lcon = L1 + L2 + L3. The implementation process of step S106 is as follows: adjust the model parameters of the original student model according to the coding loss data Lcon, repeat the previous steps to adjust the parameters of the original student model multiple times to converge the value of the coding loss data, and thus obtain a speech content recognition model.

[0116] It should be noted that after the value of the encoding loss data converges, the student prediction layer is removed to further simplify the structure of the speech content recognition model and reduce memory usage.

[0117] In some embodiments, see Figure 2 and Figure 6 Before step S103, the voice content recognition method of the embodiment of the present application also includes but is not limited to:

[0118] Constructing the original student model specifically includes but is not limited to step S501 and step S502:

[0119] Step S501, obtaining the parameters of the teacher coding layer to obtain reference parameters;

[0120] Step S502, migrating the reference parameters to the student coding layer of the preset speech training model to obtain the original student model.

[0121] Step S501 and step S502 shown in the embodiment of the present application migrate the parameters of the teacher coding layer of the teacher model to the student coding layer of the preset speech training model to construct the original student model, so that the original student model inherits the speech content recognition performance of the teacher model to a certain extent, simplifies the pre-training process required to construct the original student model, and speeds up the training speed.

[0122] Please note that Figure 2 , the speech coding content data is output to the student prediction layer through the bottom layer of the student coding layer (student coding layer 4). When adjusting the parameters of the original student model according to the coding loss data, the parameters of the student coding layer 4 are mainly adjusted, and the parameters of other student encoder layers are automatically updated through reverse transmission.

[0123] In step S501 of some embodiments, see Figure 2 , select Transformer layer 1, Transformer layer 2, Transformer layer 3 and Transformer layer 4 of the teacher model as the teacher encoding layer, and obtain the network configuration parameters of Transformer layer 1, Transformer layer 2, Transformer layer 3 and Transformer layer 4 to obtain reference parameters.

[0124] In step S502 of some embodiments, the teacher coding layer corresponds to the student coding layer one by one, and the reference parameters obtained from the teacher coding layer are transferred to the corresponding student coding layer. Figure 2 , the parameters obtained from Transformer layer 1 are migrated to student encoding layer 1; the parameters obtained from Transformer layer 2 are migrated to student encoding layer 2; the parameters obtained from Transformer layer 3 are migrated to student encoding layer 3; the parameters obtained from Transformer layer 4 are migrated to student encoding layer 4, thereby completing the construction of the original student model.

[0125] In some embodiments, see Figure 7 After step S106, the voice content recognition method of the embodiment of the present application further includes but is not limited to:

[0126] Updating the speech content recognition model specifically includes but is not limited to steps S601 to S606:

[0127] Step S601, selecting a reference coding layer from the student coding layer; wherein the reference coding layer is the bottom layer of the student coding layer;

[0128] Step S602, obtaining training speech data;

[0129] Step S603, encoding the training speech data through the reference encoding layer to obtain reference encoding data;

[0130] Step S604, encoding the training speech data through the student coding layer to obtain training coding data; wherein the training coding data is the coding data of other layers in the student coding layer except the reference coding layer;

[0131] Step S605, performing loss calculation based on the reference coding data and the training coding data to obtain updated loss data;

[0132] Step S606: update the parameters of the speech content recognition model according to the updated loss data.

[0133] In the embodiment of the present application, the teacher prediction layer has worse performance in extracting speech content than the teacher encoding layer. Although the performance of the original student model can be improved more comprehensively by constructing a student prediction layer in the original student model to learn the hidden representation of the teacher model, it will slightly affect the content extraction performance of the speech content recognition model.

[0134] In steps S601 to S607 shown in the embodiment of the present application, a reference coding layer is selected in the student coding layer, and self-supervised parameter adjustment is performed according to the output loss between the reference coding layer and other coding layers to update the speech content recognition model's ability to extract language content. It should be noted that the self-supervised parameter adjustment of this implementation is only performed through the intermediate output of the student coding layer, and there is no need to decode the output of the coding layer to generate a complete mel spectrum and compare it with the mel spectrum of the training speech data for parameter adjustment, which simplifies the model update process.

[0135] In step S601 of some embodiments, the bottom layer of the student coding layer is selected to facilitate subsequent rapid parameter adjustment through reverse transfer.

[0136] In step S602 of some embodiments, the training voice data is voice data recorded according to set content.

[0137] In step S603 and step S604 of some embodiments, encoding outputs of the reference encoding layer and other encoding layers in the student encoding layer on the training speech data are obtained for comparison.

[0138] In step S605 and step S606 of some embodiments, the loss value between the coding output results of the reference coding layer and other coding layers is calculated, and the parameters of the speech content recognition model are updated to converge the values ​​of the updated loss data.

[0139] In an exemplary embodiment, assuming that there are K student coding layers in the speech content recognition model, the calculation formula for updating the loss data is:

[0140]

[0141] In formula (1), L INTER represents the value of updating the loss data, l is an integer between 1 and K (the K value is not taken), M represents the set of speech content features hidden by the mask module, t represents time, z t represents the output of the reference coding layer at time t, Represents the output of the lth student encoding layer at time t.

[0142] See also Figure 8 The present application also provides a speech synthesis method, which includes but is not limited to steps S701 to S703:

[0143] Step S701, obtaining target content data; wherein the target content data is obtained by the voice content recognition method of any of the above embodiments;

[0144] Step S702, extracting timbre features from a preset timbre feature library according to a preset target timbre type to obtain a target timbre feature;

[0145] Step S703, performing speech synthesis according to the target content data and the target timbre characteristics to obtain target speech data.

[0146] In steps S701 to S703 shown in the embodiment of the present application, the target speech data is generated by acquiring the content information extracted by the speech content recognition model and combining the extracted timbre features, thereby completing the timbre conversion of the speech.

[0147] When the speech synthesis method of the embodiment of the present application is applied to intelligent customer service, the customer's consultation content is extracted according to the speech content recognition model, the answer content is searched from the preset database, and speech synthesis is performed according to the timbre characteristics corresponding to the answer content and the target timbre type to obtain the target voice reply and report it to the customer. The target timbre type can be the default timbre type of the customer service system or terminal, or it can be the timbre type selected by the customer through the interactive interface. In addition, if the customer consults by phone, the timbre type currently used for the intelligent reply to the customer (or the timbre type of the seat customer service currently replying to the customer) can be recorded. When the customer calls again through the same number for consultation, the customer is replied according to the recorded timbre type to enhance the customer experience.

[0148] See also Fig. 9 The embodiment of the present application further provides a voice content recognition device, which can implement the above-mentioned voice content recognition method, and the device includes:

[0149] A data acquisition module, used to acquire source voice data;

[0150] A first encoding module is used to encode the source speech data through a preset teacher model to obtain speech encoding reference data;

[0151] A second encoding module is used to encode the source speech data through a preset original student model to obtain speech encoding content data; wherein the original student model is constructed based on a part of the encoding layer of the teacher model;

[0152] A content prediction module, used for performing hidden content prediction on speech coding content data to obtain speech coding prediction data;

[0153] A loss calculation module, used to perform loss calculation on speech coding reference data and speech coding prediction data to obtain coding loss data;

[0154] A parameter adjustment module, used for adjusting the parameters of the original student model according to the coding loss data to obtain a speech content recognition model;

[0155] The content recognition module is used to encode the pre-acquired target voice data through a voice content recognition model to obtain target content data.

[0156] The specific implementation of the speech content recognition device is substantially the same as the specific implementation of the speech content recognition method described above, and will not be described in detail herein.

[0157] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned voice content recognition method or voice synthesis method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0158] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0159] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0160] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the speech content recognition method or speech synthesis method of the embodiments of this application;

[0161] Input / output interface 1003, used to implement information input and output;

[0162] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0163] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0164] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0165] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned speech content recognition method or speech synthesis method is implemented.

[0166] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] The speech content recognition method and device, speech synthesis method, electronic device and storage medium proposed in this application predict hidden content through the original student model, and compare it with the output data of the teacher model, and adjust the model parameters according to the comparison results, so as to learn the hidden representation of the teacher model (that is, learn the content extraction ability of the teacher model). While ensuring the content extraction ability of the model, the complexity of the model structure is significantly reduced and the model memory usage is reduced.

[0168] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0169] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0170] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, and may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0172] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0173] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0174] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0175] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0178] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A method for speech content recognition, characterized in that: include: Obtain source voice data; The source speech data is encoded by a preset teacher model to obtain speech encoding reference data; The source speech data is encoded by a preset original student model to obtain speech encoding content data; wherein the original student model is constructed based on a part of the encoding layer of the teacher model; Performing hidden content prediction on the speech coding content data to obtain speech coding prediction data; Performing loss calculation on the speech coding reference data and the speech coding prediction data to obtain coding loss data; Adjusting parameters of the original student model according to the coding loss data to obtain a speech content recognition model; The target voice data acquired in advance is encoded by the voice content recognition model to obtain the target content data.

2. The method for speech content recognition according to claim 1, characterized in that: The teacher model includes a teacher encoding layer and a teacher prediction layer, and any two of the teacher prediction layers are not connected; the step of encoding the source speech data through the preset teacher model to obtain speech coding reference data includes: Masking the source speech data through the teacher coding layer to obtain training mask data; The training mask data is encoded through the teacher prediction layer to obtain the speech encoding reference data.

3. The method for speech content recognition according to claim 2, characterized in that: Before the step of performing hidden content prediction on the speech coding content data to obtain speech coding prediction data, the speech content recognition method further includes: Obtain the number of layers of the teacher prediction layer to obtain the reference prediction layer number; Constructing a student prediction layer in the original student model according to the reference prediction layer number; wherein the number of layers of the student prediction layer is equal to the number of layers of the teacher prediction layer.

4. The method for speech content recognition according to claim 3, characterized in that: The step of performing hidden content prediction on the speech coding content data to obtain speech coding prediction data comprises: Performing mask processing on the speech coding content data to obtain speech mask data; The speech mask data is subjected to content prediction by the student prediction layer to obtain the speech coding prediction data.

5. The method for speech content recognition according to any one of claims 2 to 4, characterized in that: Before the step of encoding the source voice data by using a preset original student model to obtain voice encoding content data, the voice content recognition method further includes: Constructing the original student model specifically includes: Obtaining parameters of the teacher coding layer to obtain reference parameters; The reference parameters are transferred to the student coding layer of the preset speech training model to obtain the original student model.

6. The method for speech content recognition according to claim 5, characterized in that: After the step of adjusting the parameters of the original student model according to the coding loss data to obtain the speech content recognition model, the speech content recognition method further comprises: Updating the speech content recognition model specifically includes: Selecting a reference coding layer from the student coding layers; wherein the reference coding layer is the bottom layer of the student coding layers; Get training speech data; Performing encoding processing on the training speech data through the reference encoding layer to obtain reference encoding data; The training speech data is encoded by the student encoding layer to obtain training encoding data; wherein the training encoding data is encoding data of other layers in the student encoding layer except the reference encoding layer; Perform loss calculation based on the reference encoding data and the training encoding data to obtain updated loss data; Parameters of the speech content recognition model are updated according to the updated loss data.

7. A speech synthesis method, characterized in that: include: Acquire target content data; wherein the target content data is obtained by the voice content recognition method according to any one of claims 1 to 6; Extracting timbre features from a preset timbre feature library according to a preset target timbre type to obtain a target timbre feature; Speech synthesis is performed according to the target content data and the target timbre characteristics to obtain target speech data.

8. A speech content recognition device, characterized in that: The device comprises: A data acquisition module, used to acquire source voice data; A first encoding module, used for encoding the source speech data through a preset teacher model to obtain speech encoding reference data; A second encoding module, used for encoding the source speech data through a preset original student model to obtain speech encoding content data; wherein the original student model is constructed based on a part of the encoding layer of the teacher model; A content prediction module, used for performing hidden content prediction on the speech coding content data to obtain speech coding prediction data; A loss calculation module, used for performing loss calculation on the speech coding reference data and the speech coding prediction data to obtain coding loss data; A parameter adjustment module, used for adjusting the parameters of the original student model according to the coding loss data to obtain a speech content recognition model; The content recognition module is used to encode the pre-acquired target voice data through the voice content recognition model to obtain target content data.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the speech content recognition method according to any one of claims 1 to 6 or the speech synthesis method according to claim 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech content recognition method according to any one of claims 1 to 6 or the speech synthesis method according to claim 7 is implemented.

Images