Pronunciation feature processing method, device, server and medium

Abstract

The embodiments of the present application disclose a pronunciation feature processing method, device, computer equipment and medium, which belong to the field of speech processing technology. The method includes: obtaining text data and speech data corresponding to the text data, wherein the speech data includes multiple speech segments; recognizing the text data and the speech data to obtain the first pronunciation feature of each speech segment; fusing the first pronunciation feature of each speech segment with the first pronunciation feature of other speech segments to obtain the fusion feature of each speech segment; adjusting the first pronunciation feature of each speech segment according to the fusion feature of each speech segment to obtain the second pronunciation feature of each speech segment. The method realizes noise compensation for the first pronunciation feature, so that the obtained second pronunciation feature can more accurately represent the pronunciation feature of the speech segment, thereby improving the accuracy of the pronunciation feature of the speech data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of speech processing technology, and in particular to a pronunciation feature processing method, device, server and medium. Background Art

[0002] With the development of computer technology and speech processing technology, using speech processing technology to assist users in practicing the pronunciation of a certain language has become a popular research direction. For example, the speech data corresponding to a certain word read by the user is collected, and the speech data is recognized to obtain the pronunciation characteristics of the speech data, so as to facilitate subsequent processing of the speech data according to the pronunciation characteristics.

[0003] However, due to the influence of the user's environment, collection equipment or other factors, the collected voice data may contain noise, and the pronunciation features may also be affected by the noise, so the accuracy of the pronunciation features is low. Summary of the invention

[0004] The embodiment of the present application provides a method, device, server and medium for processing pronunciation features, which improves the accuracy of pronunciation features. The technical solution is as follows:

[0005] On the one hand, a method for processing pronunciation features is provided, the method comprising:

[0006] Acquire text data and voice data corresponding to the text data, wherein the text data includes a plurality of character segments, the voice data includes a plurality of voice segments, and the sound in each voice segment is emitted according to a character segment;

[0007] Recognize the text data and the voice data to obtain a first pronunciation feature of each voice segment, wherein the first pronunciation feature indicates a matching degree between a template pronunciation of a character segment corresponding to the voice segment and the voice segment;

[0008] Fusing the first pronunciation feature of each speech segment with the first pronunciation features of other speech segments to obtain a fusion feature of each speech segment;

[0009] The first pronunciation feature of each speech segment is adjusted according to the fusion feature of each speech segment to obtain the second pronunciation feature of each speech segment.

[0010] In a possible implementation, the recognizing the text data and the voice data to obtain a first pronunciation feature of each voice segment includes:

[0011] The speech recognition model is called to recognize the text data and the speech data to obtain the first pronunciation feature of each speech segment.

[0012] On the other hand, a pronunciation feature processing device is provided, the device comprising:

[0013] A data acquisition module, used to acquire text data and voice data corresponding to the text data, wherein the text data includes a plurality of character segments, and the voice data includes a plurality of voice segments, and the sound in each voice segment is emitted according to a character segment;

[0014] A data recognition module, used to recognize the text data and the voice data to obtain a first pronunciation feature of each voice segment, wherein the first pronunciation feature indicates a matching degree between a template pronunciation of a character segment corresponding to the voice segment and the voice segment;

[0015] A feature fusion module, used for fusing the first pronunciation feature of each speech segment with the first pronunciation features of other speech segments to obtain a fusion feature of each speech segment;

[0016] The feature adjustment module is used to adjust the first pronunciation feature of each speech segment according to the fusion feature of each speech segment to obtain the second pronunciation feature of each speech segment.

[0017] In a possible implementation, the feature fusion module is used to call the noise compensation model to fuse the first pronunciation feature of each speech segment with the first pronunciation features of the other speech segments to obtain the fusion feature of each speech segment;

[0018] The feature adjustment module is used to call the noise compensation model to adjust the first pronunciation feature of each speech segment according to the fusion feature of each speech segment to obtain the second pronunciation feature of each speech segment.

[0019] In another possible implementation, the device further includes:

[0020] A sample data acquisition module, used to acquire sample data, wherein the sample data includes sample text data and sample voice data corresponding to the sample text data, wherein the sample text data includes a plurality of sample character segments, and the sample voice data includes a plurality of sample voice segments, and the sound in each sample voice segment is emitted according to a sample character segment;

[0021] A sample feature acquisition module, used to acquire a first sample feature and a second sample feature of each sample speech segment, wherein the first sample feature indicates a matching degree between a template pronunciation of a sample character segment corresponding to the sample speech segment and the sample speech segment, and the second sample feature is a feature obtained after denoising the first sample feature;

[0022] A training module is used to call the noise compensation model, adjust the first sample feature of each sample speech segment respectively, obtain the predicted pronunciation feature of each sample speech segment, and train the noise compensation model according to the difference between the predicted pronunciation feature of each sample speech segment and the second sample feature.

[0023] In another possible implementation, the sample data further includes the sample accuracy of the sample speech data, the noise compensation model includes a noise compensation network and a spoken language evaluation network; and the training module is further used to:

[0024] Calling the noise compensation network to adjust the first sample feature of each sample speech segment respectively to obtain the predicted pronunciation feature of each sample speech segment;

[0025] Calling the spoken language evaluation network to fuse the predicted pronunciation features of multiple sample speech segments to obtain the prediction accuracy of the sample speech data;

[0026] The noise compensation network is trained according to the difference between the prediction accuracy and the sample accuracy.

[0027] In another possible implementation, the sample data further includes a sample accuracy rate of the sample speech data, and the training module is further used to:

[0028] Calling a spoken language evaluation model to evaluate the predicted pronunciation features of the multiple sample character segments and the multiple sample voice segments to obtain the prediction accuracy of the sample voice data;

[0029] The noise compensation model is trained according to the difference between the prediction accuracy and the sample accuracy.

[0030] In another possible implementation, the data recognition module is used to recognize the text data and the voice data to obtain a first pronunciation feature of each voice segment and a character feature of each character segment, wherein the character feature is used to indicate a position of the character segment in the text data and a character type to which the character segment belongs;

[0031] The feature fusion module comprises:

[0032] A first fusion unit, configured to fuse the first pronunciation feature of each speech segment with the first pronunciation features of the other speech segments to obtain a fused pronunciation feature of each speech segment;

[0033] The second fusion unit is used to fuse the character features of each character segment with the character features of other character segments to obtain the fused character features of each character segment.

[0034] In another possible implementation, the data recognition module is used to recognize the text data and the voice data to obtain a first pronunciation feature of each voice segment and a noise feature of each voice segment, wherein the noise feature is used to describe the noise contained in the voice segment;

[0035] The feature adjustment module is used to adjust the first pronunciation feature of each speech segment according to the fusion feature and the noise feature of each speech segment, so as to obtain the second pronunciation feature of each speech segment.

[0036] In another possible implementation, the device further includes:

[0037] The speech evaluation module is used to evaluate the second pronunciation features of the multiple character segments and the multiple speech segments to obtain the pronunciation accuracy of the speech data.

[0038] In another possible implementation, the voice evaluation module is used to:

[0039] Evaluating the second pronunciation features of the multiple character segments and the multiple voice segments respectively to obtain pronunciation accuracy rates of the multiple voice segments;

[0040] The pronunciation accuracy rates of the multiple voice segments are merged to obtain the pronunciation accuracy rate of the voice data.

[0041] In another possible implementation, the speech evaluation module is used to call a spoken language evaluation model to evaluate the second pronunciation features of the multiple character segments and the multiple speech segments to obtain the pronunciation accuracy of the speech data.

[0042] In another possible implementation, the device further includes:

[0043] The accuracy rate sending module is used to send the pronunciation accuracy rate of the voice data to the terminal, and the terminal is used to display the pronunciation accuracy rate in the follow-up reading interface containing the text data.

[0044] In another possible implementation, the data acquisition module is used to receive the text data and the voice data sent by a terminal, and the terminal is used to display a follow-up reading interface containing the text data, and collect the voice data in response to a follow-up reading start operation.

[0045] In another possible implementation, the data recognition module is used to call a speech recognition model to recognize the text data and the speech data to obtain a first pronunciation feature of each speech segment.

[0046] On the other hand, a server is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the pronunciation feature processing method described in the above aspects.

[0047] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed in the pronunciation feature processing method described in the above aspects.

[0048] On the other hand, a computer program product or a computer program is provided, wherein the computer program product or the computer program includes a computer program code, wherein the computer program code is stored in a computer-readable storage medium, and a processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device implements the operations performed in the pronunciation feature processing method described in the above aspects.

[0049] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0050] The method, device, server and medium provided in the embodiments of the present application identify text data and voice data to obtain first pronunciation features of multiple voice segments. Since the voice data is affected by noise, the obtained first pronunciation features are difficult to accurately represent the pronunciation features of the voice segments. Therefore, the association between the multiple voice segments is comprehensively considered, the multiple first pronunciation features are fused to obtain a fused feature, and the first pronunciation feature is adjusted according to the fused feature to achieve noise compensation for the first pronunciation feature, so that the obtained second pronunciation feature can more accurately represent the pronunciation features of the voice segments, thereby improving the accuracy of the pronunciation feature. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0053] Figure 2 is a flow chart of a method for processing pronunciation features provided by an embodiment of the present application;

[0054] Figure 3 It is a structural diagram of a network model provided in an embodiment of the present application;

[0055] Figure 4 It is a structural diagram of another network model provided in an embodiment of the present application;

[0056] Figure 5 is a flow chart of another method for processing pronunciation features provided in an embodiment of the present application;

[0057] Figure 6 is a schematic diagram of a follow-up reading interface provided in an embodiment of the present application;

[0058] Figure 7 It is a schematic diagram of the relationship between noise characteristics and pronunciation characteristics provided in an embodiment of the present application;

[0059] Figure 8 is a structural schematic diagram of a noise compensation model provided in an embodiment of the present application;

[0060] Fig. 9 It is a structural diagram of another network model provided in an embodiment of the present application;

[0061] Fig.10 is a schematic diagram of another follow-up reading interface provided in an embodiment of the present application;

[0062] Fig.11 is a flow chart of another method for processing pronunciation features provided in an embodiment of the present application;

[0063] Fig.12 is a flow chart of a noise compensation model training method provided in an embodiment of the present application;

[0064] Fig.13 is a schematic diagram of pronunciation characteristics under different pronunciation qualities provided by an embodiment of the present application;

[0065] Fig.14 is a schematic diagram of a training noise compensation model provided in an embodiment of the present application;

[0066] Fig.15 is a schematic diagram of another training noise compensation model provided in an embodiment of the present application;

[0067] Fig.16 It is a structural schematic diagram of a pronunciation feature processing device provided in an embodiment of the present application;

[0068] Fig.17is a structural schematic diagram of another pronunciation feature processing device provided in an embodiment of the present application;

[0069] Fig.18 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0070] Fig.19 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0072] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of this application, a first pronunciation feature can be referred to as a second pronunciation feature, and a second pronunciation feature can be referred to as a first pronunciation feature.

[0073] The terms "at least one", "a plurality of", "each", "any" and the like used in this application include one, two or more than two, a plurality of includes two or more than two, each refers to each of the corresponding plurality, and any refers to any one of the plurality. For example, a plurality of speech segments include three speech segments, and each speech segment refers to each of the three speech segments, and any refers to any one of the three speech segments, which may be the first, the second, or the third.

[0074] In order to facilitate understanding of the embodiments of the present application, the key words involved in the embodiments of the present application are first explained:

[0075] GOP (Goodness Of Pronunciation): GOP indicates the pronunciation confidence of a certain voice segment corresponding to a certain character segment. The higher the confidence, the more accurate the pronunciation of the voice segment for the character segment.

[0076] BLSTM (Bidirectional Long Short-Term Memory): BLSTM is a neural network structure composed of forward LSTM and backward LSTM. It is used to process contextual information in text data in natural language processing.

[0077] HMM (Hidden Markov Models): A statistical model used to describe a Markov process with hidden unknown parameters.

[0078] ASR (Automatic Speech Recognition): The process of converting audio into text.

[0079] GMM (Gaussian Mixture Model): A clustering algorithm that uses a linear combination of multiple Gaussian distribution functions to fit the data distribution.

[0080] SPLICE (Stereo-based Piecewise Linear Compensation for Environments): a noise compensation algorithm for ASR based on GMM.

[0081] Phoneme: The basic unit of English pronunciation. English generally contains 48 or 39 basic phonemes.

[0082] SNR (Signal to Noise Ratio): The ratio of signal strength to noise strength. The larger the value, the smaller the noise.

[0083] PCC (Pearson Correlation Coefficient: used to measure the correlation (linear correlation) between two variables X and Y, and its value is between -1 and 1.

[0084] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0085] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0086] The key technologies of speech technology include automatic speech recognition technology (ASR), text-to-speech technology (TTS) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0087] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0088] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0089] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0090] The solutions provided in the embodiments of the present application involve artificial intelligence speech technology, natural language processing, machine learning and other technologies, which are specifically illustrated by the following embodiments.

[0091] The method provided in the embodiments of the present application can be applied in a variety of scenarios.

[0092] For example, it is applied in oral learning scenarios.

[0093] When users are learning spoken English, it is difficult to determine whether their pronunciation is accurate without the help of others. In this case, the pronunciation feature processing method provided in the embodiment of the present application can be used to identify the English paragraph and the voice data generated when the user reads the English paragraph, to obtain a first pronunciation feature of the voice data, and then to adjust the first pronunciation feature to obtain a more accurate second pronunciation feature, and then to score the user's pronunciation based on the second pronunciation feature, so that the user can understand whether his or her pronunciation is accurate based on the score.

[0094] For another example, it is used in oral assessment scenarios.

[0095] When a spoken language evaluation model is used to evaluate a user's spoken English, the evaluation result accuracy of the spoken language evaluation model is low due to the influence of noise in the user's environment. At this time, the pronunciation feature processing method provided in the embodiment of the present application can be used to identify the English paragraph and the voice data generated when the user reads the English paragraph, obtain the first pronunciation feature of the voice data, and then adjust the first pronunciation feature to obtain a more accurate second pronunciation feature. The spoken language evaluation model is then used to score the user's pronunciation based on the second pronunciation feature, so that the user can understand whether his or her pronunciation is accurate based on the score.

[0096] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. Figure 1 , the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.

[0097] The target application provided by the server 102 is installed on the terminal 101, and the terminal 101 can realize functions such as data transmission and message interaction through the target application. Optionally, the terminal 101 is a computer, a mobile phone, a tablet computer or other terminals. Optionally, the target application is a target application in the operating system of the terminal 101, or a target application provided by a third party. For example, the target application is a spoken language learning application, which has the function of spoken language assessment. Of course, the spoken language learning application can also have other functions, such as spoken language teaching function, article sharing function, etc. Optionally, the server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0098] Figure 2 This is a flowchart of a method for processing pronunciation features provided by an embodiment of the present application. The execution subject of the embodiment of the present application is a server, see Figure 2 , the method comprises the following steps:

[0099] 201. The server obtains text data and voice data corresponding to the text data.

[0100] The text data includes multiple character segments, the text data is a paragraph, a sentence, a word, etc., a character segment includes at least one letter, and the text data is a text in any language, for example, the text data is Chinese, English, Japanese or other languages. For example, if the text data is the English word "bus", then "b", "u" and "s" are each a character segment, and if the text data is the English word "word", then "w", "or" and "d" are each a character segment.

[0101] The voice data includes multiple voice segments, and the sound in each voice segment is emitted according to a character segment. The voice data corresponding to the text data is the voice data obtained by the user reading aloud according to the text data. Among them, each voice segment includes at least one phoneme, which is the smallest unit of speech, and each phoneme corresponds to a pronunciation action. For example, when the user reads the English word "word", "w", "or" and "d" correspond to a pronunciation action respectively, and multiple pronunciation actions are connected together to obtain the voice data corresponding to "word".

[0102] 202. The server recognizes the text data and the voice data to obtain a first pronunciation feature of each voice segment.

[0103] When recognizing text data and voice data, the voice data can be divided into multiple voice segments according to phonemes, and each voice segment can correspond to a character segment in the text data, so as to determine the first pronunciation feature of the voice segment corresponding to the character segment based on the template pronunciation of the character segment and the voice segment corresponding to the character segment.

[0104] The first pronunciation feature indicates the degree of match between the template pronunciation of the character segment corresponding to the voice segment and the voice segment. The template pronunciation refers to the standard pronunciation of the character segment, and the voice segment is the actual pronunciation of the user. The degree of match between the standard pronunciation and the actual pronunciation indicates whether the user's pronunciation is accurate. The higher the degree of match, the more accurate the user's pronunciation. Optionally, the first pronunciation feature is any value greater than or equal to 0 and less than or equal to 1, for example, the first pronunciation feature is 0.1, 0.8, 0.5, etc. Optionally, the first pronunciation feature is GOP.

[0105] 203. The server fuses the first pronunciation feature of each voice segment with the first pronunciation features of other voice segments to obtain a fused feature of each voice segment.

[0106] For each speech segment, other speech segments refer to speech segments other than the speech segment in the speech data, for example, for the English word "bus", for the speech segment corresponding to "b", the other speech segments are "u" and "s".

[0107] Fusing the first pronunciation feature of each speech segment with the first pronunciation features of other speech segments means fusing the first pronunciation features of other speech segments on the basis of the first pronunciation feature of the speech segment, thereby obtaining a fusion feature of the speech segment, wherein the fusion feature includes the first pronunciation feature of the speech segment and the first pronunciation features of other speech segments.

[0108] In a possible implementation, for any speech segment, considering that other speech segments of the speech segment have different degrees of influence on the speech segment, when fusion is performed, the number of speech segments between other speech segments and the speech segment is determined according to the order of the multiple speech segments in the speech data, the weight of the other speech segments is determined according to the number of speech segments, the first pronunciation features of the other speech segments are weighted according to the corresponding weights, and the first pronunciation features of the speech segment are fused with the weighted first pronunciation features of the other speech segments to obtain the fusion features of the speech segment. Among them, the smaller the number of the speech segments, the greater the weight of the other speech segments, and the larger the number of the speech segments, the smaller the weight of the other speech segments. For example, if there is one speech segment between the other speech segments and the speech segment, the weight of the other speech segments is determined to be 0.8, and if there are 10 speech segments between the other speech segments and the speech segment, the weight of the other speech segments is determined to be 0.1.

[0109] For example, the weight of the first pronunciation feature of the speech segment adjacent to the speech segment is 0.9, the weight of the first pronunciation feature of the speech segment separated from the speech segment by one speech segment is 0.8, etc. A sentence includes 5 speech segments. For the third speech segment, the first pronunciation feature of the third speech segment is 0.1, and the first pronunciation features of the first, second, fourth, and fifth speech segments are 0.4, 0.9, 0.9, and 0.7, respectively, and the weights are 0.8, 0.9, 0.9, and 0.8, respectively. The fusion is performed to obtain the fusion feature (0.32, 0.81, 0.1, 0.81, 0.56).

[0110] Optionally, the fused feature obtained by fusing multiple first features is represented by a vector, a matrix or other forms.

[0111] 204. The server adjusts the first pronunciation feature of each voice segment according to the fusion feature of each voice segment to obtain the second pronunciation feature of each voice segment.

[0112] Among them, the fusion feature can represent the connection between different voice segments in the voice data. Therefore, according to the fusion feature of each voice segment, the first pronunciation feature of the pair of voice segments is adjusted, and the obtained second pronunciation feature is the pronunciation feature after noise compensation, which can more accurately represent the pronunciation of the voice segment.

[0113] For example, the first pronunciation feature of a speech segment is 0.1, while the first pronunciation features of the speech segments before and after the speech segment are both 1, that is, the speech segments before and after the speech segment are both speech segments with relatively accurate pronunciation. Therefore, it can be determined that the first pronunciation feature may be inaccurately pronounced due to the influence of noise, so the first pronunciation feature 0.1 is adjusted.

[0114] The method provided in the embodiment of the present application identifies text data and voice data to obtain first pronunciation features of multiple voice segments. Since the voice data is affected by noise, the obtained first pronunciation features are difficult to accurately represent the pronunciation features of the voice segments. Therefore, the association between the multiple voice segments is comprehensively considered, the multiple first pronunciation features are fused to obtain a fused feature, and the first pronunciation feature is adjusted according to the fused feature to achieve noise compensation for the first pronunciation feature, so that the obtained second pronunciation feature can more accurately represent the pronunciation features of the voice segments, thereby improving the accuracy of the pronunciation feature.

[0115] In one possible implementation, a network model can be used to process text data and voice data to achieve processing of pronunciation features. The network model involved is described below:

[0116] Figure 3 This is a schematic diagram of a network model provided in an embodiment of the present application. Figure 3 The network model includes a speech recognition model 301 and a noise compensation model 302, and the speech recognition model 301 and the noise compensation model 302 are connected. The speech recognition model 301 is used to recognize text data and speech data to obtain a first pronunciation feature of each speech segment in the speech data, and the noise compensation model 302 is used to perform noise compensation on the first pronunciation feature to reduce the influence of noise on the first pronunciation feature, thereby obtaining a second pronunciation feature after noise compensation.

[0117] In one possible implementation, see Figure 4 The network model further includes a spoken language evaluation model 303, which is connected to the speech recognition model 301 and the noise compensation model 302. The spoken language evaluation model 303 is used to evaluate the speech data according to the multiple character segments output by the speech recognition model 301 and the second pronunciation features output by the noise compensation model 302, so as to obtain the pronunciation accuracy of the speech data.

[0118] Figure 5 is a flowchart of a method for processing pronunciation features provided in an embodiment of the present application. The interactive subjects in the embodiment of the present application are a terminal and a server, see Figure 5 , the method comprises the following steps:

[0119] 501. The terminal displays a follow-up reading interface including text data, and in response to a follow-up reading start operation, starts collecting voice data corresponding to the text data.

[0120] The terminal displays a follow-up reading interface containing text data, and the user performs a follow-up reading start operation in the follow-up reading interface and starts to read the text data aloud. In response to the follow-up reading start operation, the terminal starts to collect the user's voice data. The text data in the follow-up reading interface is text data automatically generated by the terminal, or is text data input by the user.

[0121] In the process of collecting voice data, the terminal not only collects the voice data sent by the user, but also collects other sounds in the user's environment. These other sounds are noise in the voice data. The existence of noise will affect the subsequent processing of voice data. Therefore, it is necessary to compensate for the noise in the pronunciation features of the semantic data so that the pronunciation features after compensation do not contain noise features. For example, the voices of other users, wind, birdsong, etc. are collected.

[0122] In a possible implementation, the terminal changes the text data in the follow-up reading interface in response to the text replacement operation. Optionally, the follow-up reading interface includes a text update control, and the user triggers the text update control to update the text data in the follow-up reading interface.

[0123] In a possible implementation, the terminal is installed with a target application, the server provides services for the target application, and the terminal displays a follow-up reading interface in the target application.

[0124] 502. In response to the follow-up reading end operation, the terminal completes the collection of voice data and sends the text data and the collected voice data to the server.

[0125] 503. The server receives the text data and the voice data corresponding to the text data sent by the terminal.

[0126] After the user finishes reading the text data aloud, he executes the follow-up reading end operation. In response to the follow-up reading end operation, the terminal ends the collection of voice data, uses the collected voice data as the voice data corresponding to the text data, and sends the text data and voice data to the server. The server subsequently processes the received text data and voice data.

[0127] In a possible implementation, the follow-up reading interface also includes a collection control, and the follow-up reading start operation is a trigger operation of the collection control, and the follow-up reading end operation is a trigger operation of the collection control again. Optionally, the collection control has a start collection function and an end collection function. Before the user triggers the collection control, the collection control is a start collection control, that is, the collection control is triggered to start collecting voice data. After the user triggers the collection control once, the collection control becomes an end collection control, that is, the collection control is triggered again to end collecting voice data. Optionally, when the collection control has different functions, different texts are displayed on the upper layer of the collection control to indicate which function the collection control currently has.

[0128] For example, see Figure 6 , the follow-up reading interface 601 displays text data "I know the real situation, do you know?" and a start reading button. The user clicks the start reading button, and the terminal starts to collect the voice data sent by the user, and the start reading button becomes an end reading button. After the user finishes reading, he clicks the end reading button to complete the collection of voice data.

[0129] 504. The server calls the speech recognition model to recognize the text data and the speech data, and obtains the first pronunciation feature of each speech segment.

[0130] The speech recognition model is used to recognize text data and speech data, divide each speech segment in the speech data, and obtain the first pronunciation feature of each speech segment. Optionally, the speech recognition model is trained by a server, or is trained by other computer devices and sent to the server.

[0131] In a possible implementation, taking a character segment p as an example, the speech recognition model uses the following formula to obtain the first pronunciation feature of the speech segment corresponding to the character segment p:

[0132]

[0133] Among them, GOP(p) represents the first pronunciation feature, P(p|O (p) ) represents the probability that the pronunciation of character segment p is the pronunciation of the reference character segment, Q represents all character segments, NF(p) represents the number of frames in which character segment p is pronounced in the speech data, P(p) represents the probability that character segment p appears in the text data, and p(O (p) |p) represents the probability that the speech segment p is pronounced as the corresponding template pronunciation.

[0134] In one possible implementation, the speech recognition model can also recognize text data to obtain each character fragment in the text data. Alternatively, the speech recognition model can also obtain the character features of each character fragment, that is, obtain the position of each character fragment in the text data and the character type to which it belongs, wherein the position of the character fragment in the word includes the start position, the middle position or the end position, and the character type includes vowels and consonants. The character features are represented by vectors, matrices or other forms. For example, for a word containing three character fragments, the character features of each character fragment are represented by a vector, and the vectors corresponding to the three character fragments are concatenated together to obtain the features of the word.

[0135] In a possible implementation, the speech recognition model can also identify the noise feature of the speech data, and the noise feature is used to describe the noise contained in the speech segment. Optionally, the noise feature is a signal-to-noise ratio, an energy feature, a signal power feature, or other features. For example, the signal-to-noise ratio includes the maximum signal-to-noise ratio or the average signal-to-noise ratio in the speech segment, and the signal power feature includes the maximum signal power, the average power, etc. For example, different noise features are shown in Table 1:

[0136] Table 1

[0137]

[0138] The first column in Table 1 is the identification of the noise feature, and the second column is the noise feature. AudioAvgPower represents the average power of the speech data, AudioPeakPower represents the maximum power of the speech data, SpeechAvgPower represents the average power of the pronounced segment in the speech data, SpeechPeakPower represents the maximum power of the pronounced segment in the speech data, SilAvgPower represents the average power of the unpronounced segment in the speech data, SilPeakPower represents the maximum power of the unpronounced segment in the speech data, AvgSnr represents the average signal-to-noise ratio, and PeakSnr represents the maximum signal-to-noise ratio.

[0139] For example, see Figure 7, the horizontal axis represents different speech segments corresponding to the same character segment, each speech segment corresponding to the same character segment corresponds to a speech segment serial number, and the vertical axis represents the GOP of each speech segment. Schematic diagram 701 represents the GOP of the character segment "AA" under the influence of different noise characteristics, and schematic diagram 702 represents the GOP of the character segment "L" under the influence of different noise characteristics. The different dotted lines and solid lines in schematic diagrams 701 and 702 represent the influence of different noise characteristics. It can be seen from schematic diagrams 701 and 702 that for the same speech segment, the GOP of the speech segment is different under the influence of different noises, and the GOP of different speech segments corresponding to the same character segment is also different under the influence of the same noise.

[0140] 505. The server calls the noise compensation model to fuse the first pronunciation feature of each speech segment with the first pronunciation features of other speech segments to obtain a fused feature of each speech segment, and adjusts the first pronunciation feature of each speech segment according to the fused feature of each speech segment to obtain a second pronunciation feature of each speech segment.

[0141] The first pronunciation feature of each speech segment output by the speech recognition model is input into the noise compensation model, and the noise compensation model is called to process the first pronunciation feature of each speech segment to obtain a processed second pronunciation feature.

[0142] In a possible implementation, the noise compensation model includes multiple noise compensation networks, each of which processes the first pronunciation feature of a speech segment, and the multiple noise compensation networks are interconnected. The first pronunciation feature of each speech segment is input into the corresponding noise compensation network, and then the noise compensation network transmits the first pronunciation feature to other noise compensation networks, and the noise compensation network also obtains the first pronunciation features of other speech segments transmitted by other noise compensation networks, and then calls the noise compensation network to fuse the input first pronunciation feature and the obtained first pronunciation features transmitted by other noise networks to obtain a fused pronunciation feature, and then adjusts the input first pronunciation feature according to the fused pronunciation feature to obtain a second pronunciation feature, thereby realizing noise compensation for the input first pronunciation feature.

[0143] Optionally, the noise compensation network further includes a dimension adjustment layer, which is connected to the noise compensation network. The dimension of the second pronunciation feature output by the noise compensation network may be different from the dimension of the first pronunciation feature, and the dimension adjustment layer is used to adjust the second pronunciation feature output by the noise compensation network to obtain a second pronunciation feature with the same dimension as the first pronunciation feature. For example, the second pronunciation feature output by the noise compensation network is a three-dimensional vector, and the dimension adjustment layer is used to adjust the three-dimensional vector to a one-dimensional value.

[0144] In one possible implementation, the character features of each character segment output by the speech recognition model and the first pronunciation features of each speech segment are input into the noise compensation model together, and the noise compensation model is called to process the first pronunciation features of each speech segment and the corresponding character features to obtain the processed second pronunciation features, that is, the influence of the character features is considered in the process of processing the first pronunciation features.

[0145] Optionally, for each character feature, the character feature corresponding to each voice segment is input into the corresponding noise compensation network, and then the noise compensation network transmits the character feature to other noise compensation networks, and the noise compensation network also obtains the character features of other voice segments transmitted by other noise compensation networks, and then calls the noise compensation network to fuse the input character features and the obtained character features transmitted by other noise networks to obtain a fused character feature. Then, the noise compensation network is called to adjust the input first pronunciation feature according to the fused pronunciation feature and the fused character feature to obtain a second pronunciation feature, thereby realizing noise compensation for the input first pronunciation feature.

[0146] Optionally, the noise compensation network is a BLSTM, LSTM or other neural network structure.

[0147] In one possible implementation, the noise features of each speech segment output by the speech recognition model and the first pronunciation features of each speech segment are input into a noise compensation model, and the noise compensation model is called to process the first pronunciation features and noise features of each speech segment to obtain a processed second pronunciation feature, that is, the influence of the noise feature is considered in the process of processing the first pronunciation feature.

[0148] Optionally, the noise compensation network is called to obtain the fused pronunciation feature of each speech segment, and then the noise compensation network is called to adjust the input first pronunciation feature according to the fused pronunciation feature and the noise feature to obtain the second pronunciation feature, thereby realizing noise compensation for the input first pronunciation feature. For example, when adjusting according to the noise feature, if the noise feature of the speech segment is large, it is determined that the speech segment is seriously affected by the noise, and the first pronunciation feature needs to be adjusted significantly.

[0149] In another possible implementation, the first pronunciation feature, the noise feature, and the character feature of the corresponding character segment of each speech segment are used as inputs of the noise compensation model.

[0150] For example, see Figure 8For the text data "bus", the GOP corresponding to the first character segment "b" is 0.1, the character features are {0.1, 0.3, 0.5}, and the noise features are {0.3, 0.5, 0.7}. The GOP corresponding to the second character segment "u" is 0.2, the character features are {0.2, 0.3, 0.5}, and the noise features are {0.3, 0.6, 0.7}. The GOP corresponding to the third character segment "s" is 0.9, the character features are {0.2, 0.5, 0.5}, and the noise features are {0.4, 0.8, 0.7}. The GOP, character features, and noise features of "b", "u", and "s" are respectively input into the corresponding noise compensation network, and the adjusted GOPs are obtained after processing. The adjusted GOPs are all 0.3. Among them, the noise compensation network 801 corresponding to "b" is connected to the noise compensation network 802 corresponding to "u", and the noise compensation network 802 corresponding to "u" is connected to the noise compensation network 803 corresponding to "s".

[0151] 506. The server calls the spoken language evaluation model to evaluate the second pronunciation features of the multiple character segments and the multiple voice segments to obtain the pronunciation accuracy of the voice data.

[0152] The server calls the noise compensation model to obtain the second pronunciation features of multiple voice segments, and uses the second pronunciation features of the multiple voice segments and the character segments corresponding to the multiple voice segments as inputs to the spoken language evaluation model. Fig. 9 , the noise compensation model 302 processes the first pronunciation feature affected by the noise to obtain a second pronunciation feature, and inputs the second pronunciation into the spoken language evaluation model 303, which outputs the pronunciation accuracy of the speech data. The pronunciation accuracy is used to indicate the accuracy of the user's pronunciation of the speech segment. The pronunciation accuracy can be expressed in the form of an evaluation score, a percentage, etc.

[0153] The oral evaluation model is trained by a server, or is trained by other computer devices and sent to the server.

[0154] In a possible implementation, the server calls the spoken language evaluation model to evaluate the second pronunciation features of multiple character segments and multiple voice segments respectively to obtain the pronunciation accuracy of the multiple voice segments; the pronunciation accuracy of the multiple voice segments is integrated to obtain the pronunciation accuracy of the voice data. Optionally, according to the weights of different voice segments in the entire voice data, the pronunciation accuracy of the multiple voice segments is weightedly integrated to obtain the pronunciation accuracy of the voice data.

[0155] 507. The server sends the pronunciation accuracy of the voice data to the terminal.

[0156] 508. The terminal displays the pronunciation accuracy rate in the follow-up reading interface.

[0157] After the server obtains the pronunciation accuracy rate of the voice data, it sends the pronunciation accuracy rate to the terminal, and the terminal displays the pronunciation accuracy rate in the follow-up reading interface. For example, see Fig.10 , the pronunciation accuracy rate is represented by an evaluation score, and the evaluation score includes 1, 2, 3, 4, and 5 points. Different scores light up the corresponding number of stars. The score displayed in the follow-up reading interface 1001 is 4 points, and 4 stars are lit.

[0158] In a possible implementation manner, the server also sends the pronunciation accuracy rate of each voice segment in the voice data, or the pronunciation accuracy rate after fusing the pronunciation accuracy rates of some voice segments among multiple voice segments, and the terminal displays the pronunciation accuracy rate in the follow-up reading interface. For example, for the word "know", the voice segments of this word include voice segments of different "zh", "i", "d", and "ao". The server fuses the pronunciation accuracy rates of "zh", "i", "d", and "ao" to obtain the pronunciation accuracy rate "4" of "know", see Fig.10 , and the pronunciation accuracy rate of "know" is displayed in the follow-up reading interface 1001.

[0159] In a possible implementation manner, see Fig.11 , the user follows up and reads the text data displayed in the target application installed on the terminal. The terminal collects voice data, sends the text data and the voice data to the server. The server uses the voice data and the text data as the input of the voice recognition model, and uses the voice recognition model to obtain the first pronunciation feature and multiple character segments in the text data. The first pronunciation feature is used as the input of the noise compensation model, and the noise compensation model is used to process the first pronunciation feature to obtain the second pronunciation feature. The second pronunciation feature and multiple character segments are used as the input of the oral evaluation model, and the oral evaluation model is used to process the second pronunciation feature and multiple character segments to obtain the pronunciation accuracy rate. The server sends the obtained pronunciation accuracy rate to the target application in the terminal, and the target application displays the pronunciation accuracy rate, so that the user can know whether the pronunciation of the text data is accurate.

[0160] The method provided by the embodiments of the present application uses a voice recognition model to recognize text data and voice data to obtain the first pronunciation feature of multiple voice segments. Since the voice data is affected by noise, the obtained first pronunciation feature is difficult to accurately represent the pronunciation feature of the voice segment. Therefore, considering the correlation relationship between multiple voice segments, a noise compensation model is used to fuse multiple first pronunciation features to obtain a fusion feature, and the first pronunciation feature is adjusted according to the fusion feature to achieve noise compensation for the first pronunciation feature, so that the obtained second pronunciation feature can more accurately represent the pronunciation feature of the voice segment, and the pronunciation feature is improved.

[0161] In addition, the influence of noise characteristics in speech data and character characteristics on speech segments is also considered, which further improves the accuracy of pronunciation features.

[0162] Furthermore, after the pronunciation accuracy is obtained, the pronunciation accuracy can be displayed in the terminal, so that the user can understand the pronunciation accuracy and help the user learn spoken pronunciation.

[0163] Above Figure 5 The first pronunciation feature of the speech segment is described using a noise compensation network. In one possible implementation, see Fig.12 , the training process of the noise compensation network includes:

[0164] 1201. The server obtains sample data.

[0165] The sample data includes sample text data and sample voice data corresponding to the sample text data. The sample text data includes multiple sample character segments. The sample voice data includes multiple sample voice segments. The sound in each sample voice segment is emitted according to a sample character segment.

[0166] 1202. The server obtains a first sample feature and a second sample feature of each sample speech segment.

[0167] The first sample feature represents the matching degree between the template pronunciation of the sample character segment corresponding to the sample voice segment and the sample voice segment, and the second sample feature is the feature obtained after denoising the first sample feature.

[0168] In a possible implementation, the first sample feature of each sample speech segment is obtained by the server using a speech recognition model to recognize each speech segment.

[0169] 1203. The server calls the noise compensation model, adjusts the first sample feature of each sample speech segment respectively, obtains the predicted pronunciation feature of each sample speech segment, and trains the noise compensation model according to the difference between the predicted pronunciation feature and the second sample feature of each sample speech segment.

[0170] For example, the loss function shown in the following formula is used to obtain the loss value of the sample speech segment, and the loss value is used to represent the difference between the predicted pronunciation feature and the second sample feature, and the noise compensation model is adjusted according to the loss value:

[0171]

[0172] Among them, L gop represents the loss value of the sample speech segment, m represents the number of sample character segments in the sample text data, n represents the total number of sample text data, and p GOP(j)represents the predicted pronunciation feature corresponding to the jth sample character segment in the i-th text data, y GOP(j) Represents the second sample feature corresponding to the j-th sample character segment in the i-th sample text data.

[0173] In one possible implementation, for example, see Fig.13 ,Under different pronunciation qualities, the GOP is affected by noise characteristics to different degrees.,The horizontal axis represents different character segments, and the vertical axis represents the average GOP of each speech segment, Fig.13 The different dotted and solid lines in the figure represent different pronunciation accuracy rates. Fig.13 It can be seen that under different pronunciation accuracy rates, the GOP is affected by noise to different degrees. Therefore, when training the noise compensation model, the pronunciation accuracy rate can be used for training.

[0174] The sample data includes the sample accuracy, the noise compensation model includes the noise compensation network and the spoken language evaluation network, and the following steps are used to replace step 1203: calling the noise compensation network, adjusting the first sample feature of each sample speech segment respectively, and obtaining the predicted pronunciation feature of each sample speech segment; calling the spoken language evaluation network, fusing the predicted pronunciation features of multiple sample speech segments, and obtaining the predicted accuracy of the sample speech data; training the noise compensation network according to the difference between the predicted accuracy and the sample accuracy. The sample accuracy is marked by the user or obtained by other means.

[0175] Optionally, character features and noise features are considered in the process of training the noise compensation model. For example, see Fig.14 , input the GOP, character features and noise features into the corresponding noise compensation network 1401, and obtain the corresponding output 1, output 2 and output 3 through each noise compensation network. The output 1, output 2 and output 3 are the predicted GOPs without dimension adjustment. The output 1, output 2 and output 3 are input into the spoken language evaluation network 1402 to obtain the prediction accuracy. The vector corresponding to the feature is input into the noise compensation model, and then the training is performed. See Fig.15 , the obtained output 1, output 2 and output 3 are vectors.

[0176] For example, the loss function shown in the following formula is used to obtain the loss value of the sample speech data, and the loss value is used to represent the difference between the prediction accuracy and the sample accuracy, and the noise compensation model is adjusted according to the loss value:

[0177]

[0178] Among them, L word represents the loss value of the sample speech data, n represents the total number of sample text data, y score(i)represents the sample accuracy of the sample speech data corresponding to the i-th text data, p score(i) Represents the prediction accuracy of the sample speech data corresponding to the i-th text data.

[0179] In another possible implementation, the noise compensation model is trained by combining the above two training methods, and the loss function is:

[0180] L total =wL gop +(1-ω)L word ;

[0181] Among them, L total Represents the loss value of the sample speech data, L gop represents the loss value obtained by the predicted pronunciation features of the sample speech segment and the second sample features, L word It represents the loss value obtained based on the prediction accuracy and sample accuracy of the sample speech data. w is the reference value, and the value range of w is 0 to 1.

[0182] In another possible implementation, after obtaining the predicted pronunciation features, the following steps are used for subsequent training: calling the spoken language evaluation model, evaluating the predicted pronunciation features of multiple sample character segments and multiple sample voice segments, and obtaining the prediction accuracy of the sample voice data; and training the noise compensation model according to the difference between the prediction accuracy and the sample accuracy. The process of training using the spoken language evaluation model is the same as the process of training using the spoken language evaluation network described above, except that the method of obtaining the prediction accuracy is different, which will not be repeated here.

[0183] In addition, after the processed pronunciation features are obtained by the pronunciation feature processing method (NN AXL) in the above embodiment, the pronunciation features are compared with the pronunciation features obtained by the SPLICE noise compensation algorithm in the related art. For example, the scoring correlation PCC and the scoring classification accuracy ACC (Accuracy) are used as evaluation indicators to evaluate which of the two methods has a higher accuracy rate.

[0184] For example, when human voice noise is added to the speech data, PCC and ACC are used to evaluate the pronunciation features obtained in the present application and the pronunciation features obtained by the related technology respectively. The evaluation results are shown in Table 2.

[0185] Table 2

[0186]

[0187]

[0188] When white noise is added to the speech data, PCC and ACC are used to evaluate the pronunciation features obtained in the present application and the pronunciation features obtained by the related technology respectively. The evaluation results are shown in Table 3.

[0189] Table 3

[0190]

[0191] For example, when human voice noise and white noise are added to the speech data, PCC and ACC are used to evaluate the pronunciation features obtained in the present application and the pronunciation features obtained by the related technology respectively. The evaluation results are shown in Table 4.

[0192] Table 4

[0193]

[0194] It can be seen from the above evaluation results that the scoring correlation corresponding to the method provided in the present application is significantly greater than the scoring correlation corresponding to the SPLICE algorithm in the related technology, and the scoring accuracy corresponding to the method provided in the present application is significantly greater than the scoring accuracy corresponding to the SPLICE algorithm in the related technology. Therefore, the pronunciation features obtained by the method provided in the embodiment of the present application have a higher accuracy.

[0195] Fig.16 is a structural diagram of a pronunciation feature processing device provided in an embodiment of the present application. Fig.16 , the device comprises:

[0196] The data acquisition module 1601 is used to acquire text data and voice data corresponding to the text data, wherein the text data includes a plurality of character segments, and the voice data includes a plurality of voice segments, and the sound in each voice segment is emitted according to a character segment;

[0197] The data recognition module 1602 is used to recognize the text data and the voice data to obtain a first pronunciation feature of each voice segment, where the first pronunciation feature indicates the matching degree between the template pronunciation of the character segment corresponding to the voice segment and the voice segment;

[0198] A feature fusion module 1603 is used to fuse the first pronunciation feature of each speech segment with the first pronunciation features of other speech segments to obtain a fusion feature of each speech segment;

[0199] The feature adjustment module 1604 is used to adjust the first pronunciation feature of each speech segment according to the fusion feature of each speech segment to obtain the second pronunciation feature of each speech segment.

[0200] The device provided in the embodiment of the present application recognizes text data and voice data to obtain first pronunciation features of multiple voice segments. Since the voice data is affected by noise, the obtained first pronunciation features are difficult to accurately represent the pronunciation features of the voice segments. Therefore, the association between the multiple voice segments is comprehensively considered, the multiple first pronunciation features are fused to obtain a fused feature, and the first pronunciation feature is adjusted according to the fused feature to achieve noise compensation for the first pronunciation feature, so that the obtained second pronunciation feature can more accurately represent the pronunciation features of the voice segments, thereby improving the accuracy of the pronunciation feature.

[0201] In a possible implementation, the feature fusion module 1603 is used to call the noise compensation model to fuse the first pronunciation feature of each speech segment with the first pronunciation features of other speech segments to obtain a fusion feature of each speech segment;

[0202] The feature adjustment module 1604 is used to call the noise compensation model to adjust the first pronunciation feature of each speech segment according to the fusion feature of each speech segment to obtain the second pronunciation feature of each speech segment.

[0203] In another possible implementation, see Fig.17 , the device further comprises:

[0204] The sample data acquisition module 1605 is used to acquire sample data, wherein the sample data includes sample text data and sample voice data corresponding to the sample text data, wherein the sample text data includes a plurality of sample character segments, and the sample voice data includes a plurality of sample voice segments, and the sound in each sample voice segment is emitted according to a sample character segment;

[0205] The sample feature acquisition module 1606 is used to acquire a first sample feature and a second sample feature of each sample speech segment, wherein the first sample feature indicates the matching degree between the template pronunciation of the sample character segment corresponding to the sample speech segment and the sample speech segment, and the second sample feature is a feature obtained after denoising the first sample feature;

[0206] The training module 1607 is used to call the noise compensation model, adjust the first sample feature of each sample speech segment respectively, obtain the predicted pronunciation feature of each sample speech segment, and train the noise compensation model according to the difference between the predicted pronunciation feature and the second sample feature of each sample speech segment.

[0207] In another possible implementation, the sample data also includes the sample accuracy of the sample speech data, and the noise compensation model includes a noise compensation network and a spoken language evaluation network; see Fig.17 The training module 1607 is also used to:

[0208] Calling the noise compensation network to adjust the first sample feature of each sample speech segment respectively to obtain the predicted pronunciation feature of each sample speech segment;

[0209] Call the spoken language evaluation network to fuse the predicted pronunciation features of multiple sample speech segments to obtain the prediction accuracy of the sample speech data;

[0210] Based on the difference between the prediction accuracy and the sample accuracy, a noise compensation network is trained.

[0211] In another possible implementation, the sample data also includes the sample accuracy of the sample speech data, see Fig.17 The training module 1607 is also used to:

[0212] Calling the spoken language evaluation model to evaluate the predicted pronunciation features of multiple sample character segments and multiple sample voice segments to obtain the prediction accuracy of the sample voice data;

[0213] Based on the difference between the prediction accuracy and the sample accuracy, a noise compensation model is trained.

[0214] In another possible implementation, see Fig.17 , a data recognition module 1602, used to recognize text data and voice data, and obtain a first pronunciation feature of each voice segment and a character feature of each character segment, wherein the character feature is used to indicate the position of the character segment in the text data and the character type to which it belongs;

[0215] The feature fusion module 1603 includes:

[0216] A first fusion unit 1613, used to fuse the first pronunciation feature of each speech segment with the first pronunciation features of other speech segments to obtain a fused pronunciation feature of each speech segment;

[0217] The second fusion unit 1623 is used to fuse the character feature of each character segment with the character features of other character segments to obtain a fused character feature of each character segment.

[0218] In another possible implementation, the data recognition module 1602 is used to recognize the text data and the voice data to obtain a first pronunciation feature of each voice segment and a noise feature of each voice segment, where the noise feature is used to describe the noise contained in the voice segment;

[0219] The feature adjustment module 1604 is used to adjust the first pronunciation feature of each speech segment according to the fusion feature and the noise feature of each speech segment to obtain the second pronunciation feature of each speech segment.

[0220] In another possible implementation, see Fig.17 , the device further comprises:

[0221] The speech evaluation module 1608 is used to evaluate the second pronunciation features of the multiple character segments and the multiple speech segments to obtain the pronunciation accuracy of the speech data.

[0222] In another possible implementation, see Fig.17 , the voice evaluation module 1608 is used to:

[0223] Evaluating the second pronunciation features of the plurality of character segments and the plurality of speech segments respectively to obtain pronunciation accuracy rates of the plurality of speech segments;

[0224] The pronunciation accuracy rates of multiple voice segments are combined to obtain the pronunciation accuracy rate of the voice data.

[0225] In another possible implementation, the speech evaluation module is used to call the spoken language evaluation model to evaluate the second pronunciation features of multiple character segments and multiple speech segments to obtain the pronunciation accuracy of the speech data.

[0226] In another possible implementation, see Fig.17 , the device further comprises:

[0227] The accuracy rate sending module 1609 is used to send the pronunciation accuracy rate of the voice data to the terminal, and the terminal is used to display the pronunciation accuracy rate in the follow-up reading interface containing text data.

[0228] In another possible implementation, the data acquisition module 1601 is used to receive text data and voice data sent by a terminal, and the terminal is used to display a follow-up reading interface containing text data, and collect voice data in response to a follow-up reading start operation.

[0229] In another possible implementation, the data recognition module 1602 is used to call the speech recognition model to recognize the text data and the speech data to obtain the first pronunciation feature of each speech segment.

[0230] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0231] It should be noted that: the pronunciation feature processing device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when processing pronunciation features. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the server is divided into different functional modules to complete all or part of the functions described above. In addition, the pronunciation feature processing device provided in the above embodiment and the pronunciation feature processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0232] An embodiment of the present application also provides a terminal, which includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the terminal in the pronunciation feature processing method of the above embodiment.

[0233] Fig.18 1 is a schematic diagram of the structure of a terminal 1800 provided in an embodiment of the present application. The terminal 1800 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 1800 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0234] The terminal 1800 includes: a processor 1801 and a memory 1802 .

[0235] The processor 1801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0236] The memory 1802 may include one or more computer-readable storage media, which may be non-transitory. The memory 1802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1802 is used to store at least one computer program, which is used to be executed by the processor 1801 to implement the pronunciation feature processing method provided in the method embodiment of the present application.

[0237] In some embodiments, the terminal 1800 may also optionally include: a peripheral device interface 1803 and at least one peripheral device. The processor 1801, the memory 1802 and the peripheral device interface 1803 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1803 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1804, a display screen 1805, a camera assembly 1806, an audio circuit 1807, a positioning assembly 1808 and a power supply 1809.

[0238] The peripheral device interface 1803 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1801 and the memory 1802. In some embodiments, the processor 1801, the memory 1802, and the peripheral device interface 1803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1801, the memory 1802, and the peripheral device interface 1803 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0239] The radio frequency circuit 1804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1804 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1804 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 1804 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1804 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0240] The display screen 1805 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1805 is a touch display screen, the display screen 1805 also has the ability to collect touch signals on the surface or above the surface of the display screen 1805. The touch signal can be input to the processor 1801 as a control signal for processing. At this time, the display screen 1805 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1805 can be one, set on the front panel of the terminal 1800; in other embodiments, the display screen 1805 can be at least two, respectively set on different surfaces of the terminal 1800 or in a folding design; in other embodiments, the display screen 1805 can be a flexible display screen, set on a curved surface or a folding surface of the terminal 1800. Even, the display screen 1805 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1805 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0241] The camera assembly 1806 is used to capture images or videos. Optionally, the camera assembly 1806 includes a front camera and a rear camera. The front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1806 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0242] The audio circuit 1807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1801 for processing, or input them into the radio frequency circuit 1804 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1800. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1801 or the radio frequency circuit 1804 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1807 may also include a headphone jack.

[0243] The positioning component 1808 is used to locate the current geographical location of the terminal 1800 to implement navigation or LBS (Location Based Service). The positioning component 1808 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas positioning system, or the European Union's Galileo positioning system.

[0244] The power supply 1809 is used to power various components in the terminal 1800. The power supply 1809 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1809 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0245] In some embodiments, the terminal 1800 further includes one or more sensors 1810 , including but not limited to: an acceleration sensor 1811 , a gyroscope sensor 1812 , a pressure sensor 1813 , a fingerprint sensor 1814 , an optical sensor 1815 , and a proximity sensor 1816 .

[0246] The acceleration sensor 1811 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1800. For example, the acceleration sensor 1811 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1801 can control the display screen 1805 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1811. The acceleration sensor 1811 can also be used to collect game or user motion data.

[0247] The gyro sensor 1812 can detect the body direction and rotation angle of the terminal 1800, and the gyro sensor 1812 can cooperate with the acceleration sensor 1811 to collect the user's 3D actions on the terminal 1800. The processor 1801 can implement the following functions based on the data collected by the gyro sensor 1812: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0248] The pressure sensor 1813 can be set on the side frame of the terminal 1800 and / or the lower layer of the display screen 1805. When the pressure sensor 1813 is set on the side frame of the terminal 1800, it can detect the user's holding signal of the terminal 1800, and the processor 1801 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1813. When the pressure sensor 1813 is set on the lower layer of the display screen 1805, the processor 1801 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1805. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0249] The fingerprint sensor 1814 is used to collect the user's fingerprint, and the processor 1801 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1814, or the fingerprint sensor 1814 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1801 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 1814 can be set on the front, back, or side of the terminal 1800. When a physical button or a manufacturer logo is set on the terminal 1800, the fingerprint sensor 1814 can be integrated with the physical button or the manufacturer logo.

[0250] The optical sensor 1815 is used to collect the ambient light intensity. In one embodiment, the processor 1801 can control the display brightness of the display screen 1805 according to the ambient light intensity collected by the optical sensor 1815. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1805 is increased; when the ambient light intensity is low, the display brightness of the display screen 1805 is reduced. In another embodiment, the processor 1801 can also dynamically adjust the shooting parameters of the camera component 1806 according to the ambient light intensity collected by the optical sensor 1815.

[0251] The proximity sensor 1816, also called a distance sensor, is disposed on the front panel of the terminal 1800. The proximity sensor 1816 is used to collect the distance between the user and the front of the terminal 1800. In one embodiment, when the proximity sensor 1816 detects that the distance between the user and the front of the terminal 1800 is gradually decreasing, the processor 1801 controls the display screen 1805 to switch from the screen-on state to the screen-off state; when the proximity sensor 1816 detects that the distance between the user and the front of the terminal 1800 is gradually increasing, the processor 1801 controls the display screen 1805 to switch from the screen-off state to the screen-on state.

[0252] Those skilled in the art will understand that Fig.18 The structure shown in the figure does not constitute a limitation on the terminal 1800, and the terminal 1800 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0253] An embodiment of the present application also provides a server, which includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the server in the pronunciation feature processing method of the above embodiment.

[0254] Fig.19 It is a structural diagram of a server provided in an embodiment of the present application. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1901 and one or more memories 1902, wherein the memory 1902 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1901 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0255] An embodiment of the present application also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed in the pronunciation feature processing method of the above embodiment.

[0256] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer program code, and the computer program code is stored in a computer-readable storage medium. The processor of the computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device implements the operations performed in the pronunciation feature processing method of the above embodiment.

[0257] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0258] The above are only optional embodiments of the embodiments of the present application and are not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A method for processing pronunciation features, characterized in that: The method comprises: Acquire text data and voice data corresponding to the text data, wherein the text data includes a plurality of character segments, the voice data includes a plurality of voice segments, and the sound in each voice segment is emitted according to a character segment; Recognize the text data and the voice data to obtain a first pronunciation feature of each voice segment, wherein the first pronunciation feature indicates a matching degree between a template pronunciation of a character segment corresponding to the voice segment and the voice segment; Fusing the first pronunciation feature of each speech segment with the first pronunciation features of other speech segments to obtain a fusion feature of each speech segment; The first pronunciation feature of each speech segment is adjusted according to the fusion feature of each speech segment to obtain the second pronunciation feature of each speech segment.

2. The method according to claim 1, characterized in that fusing the first pronunciation feature of each speech segment with the first pronunciation features of other speech segments to obtain a fusion feature of each speech segment; According to the fusion features of each speech segment, the first pronunciation features of each speech segment are adjusted to obtain the second pronunciation features of each speech segment, including: Calling the noise compensation model to fuse the first pronunciation feature of each speech segment with the first pronunciation features of the other speech segments to obtain a fusion feature of each speech segment; The first pronunciation feature of each speech segment is adjusted according to the fusion feature of each speech segment to obtain the second pronunciation feature of each speech segment.

3. The method according to claim 2, characterized in that The training process of the noise compensation model is as follows: Acquire sample data, the sample data including sample text data and sample voice data corresponding to the sample text data, the sample text data including a plurality of sample character segments, the sample voice data including a plurality of sample voice segments, and the sound in each sample voice segment is emitted according to a sample character segment; Obtaining a first sample feature and a second sample feature of each sample speech segment, wherein the first sample feature indicates a matching degree between a template pronunciation of a sample character segment corresponding to the sample speech segment and the sample speech segment, and the second sample feature is a feature obtained after denoising the first sample feature; The noise compensation model is called to adjust the first sample feature of each sample speech segment respectively to obtain the predicted pronunciation feature of each sample speech segment, and the noise compensation model is trained according to the difference between the predicted pronunciation feature of each sample speech segment and the second sample feature.

4. The method according to claim 3, characterized in that The sample data also includes the sample accuracy of the sample speech data, the noise compensation model includes a noise compensation network and a spoken language evaluation network, calling the noise compensation model, adjusting the first sample feature of each sample speech segment respectively, obtaining the predicted pronunciation feature of each sample speech segment, and training the noise compensation model according to the difference between the predicted pronunciation feature of each sample speech segment and the second sample feature, including: Calling the noise compensation network to adjust the first sample feature of each sample speech segment respectively to obtain the predicted pronunciation feature of each sample speech segment; Calling the spoken language evaluation network to fuse the predicted pronunciation features of multiple sample speech segments to obtain the prediction accuracy of the sample speech data; The noise compensation network is trained according to the difference between the prediction accuracy and the sample accuracy.

5. The method according to claim 3, characterized in that: The sample data also includes the sample accuracy of the sample speech data. After the noise compensation model is called to adjust the first sample feature of each sample speech segment respectively to obtain the predicted pronunciation feature of each sample speech segment, the method further includes: Calling a spoken language evaluation model to evaluate the predicted pronunciation features of the multiple sample character segments and the multiple sample voice segments to obtain the prediction accuracy of the sample voice data; The noise compensation model is trained according to the difference between the prediction accuracy and the sample accuracy.

6. The method according to claim 1 or 2, characterized in that: The identifying of the text data and the voice data to obtain a first pronunciation feature of each voice segment includes: Recognize the text data and the voice data to obtain a first pronunciation feature of each voice segment and a character feature of each character segment, wherein the character feature is used to indicate a position of the character segment in the text data and a character type to which the character segment belongs; The step of fusing the first pronunciation feature of each speech segment with the first pronunciation features of other speech segments to obtain the fusion feature of each speech segment includes: Fusing the first pronunciation feature of each speech segment with the first pronunciation features of the other speech segments to obtain a fused pronunciation feature of each speech segment; The character feature of each character segment is fused with the character features of other character segments to obtain a fused character feature of each character segment.

7. The method according to claim 1 or 2, characterized in that: The identifying of the text data and the voice data to obtain a first pronunciation feature of each voice segment includes: Recognize the text data and the voice data to obtain a first pronunciation feature of each voice segment and a noise feature of each voice segment, wherein the noise feature is used to describe the noise contained in the voice segment; The adjusting the first pronunciation feature of each speech segment according to the fusion feature of each speech segment to obtain the second pronunciation feature of each speech segment includes: The first pronunciation feature of each speech segment is adjusted according to the fusion feature and the noise feature of each speech segment to obtain the second pronunciation feature of each speech segment.

8. The method according to claim 1, characterized in that After adjusting the first pronunciation feature of each speech segment according to the fusion feature of each speech segment to obtain the second pronunciation feature of each speech segment, the method further includes: The second pronunciation features of the multiple character segments and the multiple voice segments are evaluated to obtain the pronunciation accuracy of the voice data.

9. The method according to claim 8, characterized in that The step of evaluating the second pronunciation features of the plurality of character segments and the plurality of voice segments to obtain the pronunciation accuracy of the voice data includes: Evaluating the second pronunciation features of the multiple character segments and the multiple voice segments respectively to obtain pronunciation accuracy rates of the multiple voice segments; The pronunciation accuracy rates of the multiple voice segments are merged to obtain the pronunciation accuracy rate of the voice data.

10. The method according to claim 8, characterized in that The step of evaluating the second pronunciation features of the plurality of character segments and the plurality of voice segments to obtain the pronunciation accuracy of the voice data includes: The spoken language evaluation model is called to evaluate the second pronunciation features of the multiple character segments and the multiple voice segments to obtain the pronunciation accuracy of the voice data.

11. The method according to claim 7, characterized in that After evaluating the second pronunciation features of the plurality of character segments and the plurality of voice segments to obtain the pronunciation accuracy of the voice data, the method further comprises: The pronunciation accuracy of the voice data is sent to a terminal, and the terminal is used to display the pronunciation accuracy in a follow-up reading interface containing the text data.

12. The method according to claim 1, characterized in that The acquiring of text data and voice data corresponding to the text data includes: The text data and the voice data sent by the receiving terminal are used to display a follow-up reading interface containing the text data, and collect the voice data in response to a follow-up reading start operation.

13. A pronunciation feature processing device, characterized in that: The device comprises: A data acquisition module, used to acquire text data and voice data corresponding to the text data, wherein the text data includes a plurality of character segments, and the voice data includes a plurality of voice segments, and the sound in each voice segment is emitted according to a character segment; A data recognition module, used to recognize the text data and the voice data to obtain a first pronunciation feature of each voice segment, wherein the first pronunciation feature indicates a matching degree between a template pronunciation of a character segment corresponding to the voice segment and the voice segment; A feature fusion module, used for fusing the first pronunciation feature of each speech segment with the first pronunciation features of other speech segments to obtain a fusion feature of each speech segment; The feature adjustment module is used to adjust the first pronunciation feature of each speech segment according to the fusion feature of each speech segment to obtain the second pronunciation feature of each speech segment.

14. A server, characterized in that: The server includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the pronunciation feature processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the pronunciation feature processing method according to any one of claims 1 to 12.

Images