User voice data processing method, device and electronic device

Abstract

The present application relates to a method, device, electronic device and computer-readable medium for processing user voice data. The method comprises: obtaining voice data from a user; inputting the voice data into an acoustic model to generate a plurality of phoneme data; inputting the phoneme data into a pronunciation model to generate a multidimensional character query matrix, wherein the multidimensional character query matrix includes a plurality of dimensions, each dimension including a plurality of characters and their corresponding probability values; inputting the multidimensional character query matrix into a language model to generate a semantic text; and generating response data based on the semantic text to feed back to the user. The method, device, electronic device and computer-readable medium for processing user voice data involved in the present application can quickly and accurately recognize the user's voice data, thereby interacting with the user more smoothly, quickly responding to the user's instructions, improving the overall user satisfaction and improving the efficiency of business processing.

Description

Technical Field

[0001] The present application relates to the field of computer information processing, and in particular, to a method, device, electronic device and computer-readable medium for processing user voice data. Background Art

[0002] Based on technologies such as speech recognition and synthesis, machine learning, and natural language understanding, intelligent voice robots automatically initiate outbound calls based on business scenarios, collect business results through voice dialogues between people and robots, and statistically process data to obtain user feedback. Intelligent voice robots are conversational intelligent voice robots for developers, which can implement intelligent conversations based on natural language processing on different message terminals, such as websites, apps, and physical robots. Users can configure their own unique knowledge base to achieve intelligent question and answer, or integrate with third-party APIs through multiple rounds of dialogue to achieve self-service, such as order inquiries, logistics tracking, and self-service return robots. Intelligent voice robots can analyze the content of conversations based on intelligent rules from conversation recordings or texts, and explore possible problems and opportunities in the conversation. They can help companies improve service quality, monitor public opinion risks, and optimize service strategies. Typical application scenarios include intelligent customer service quality inspection and sales opportunity analysis.

[0003] When intelligent voice machine technology is put into commercial use, it is often simulated and trained for a certain scenario, and then a semantic recognition machine learning model is generated to solve the problem. However, when it is put into practice, real life often causes long-tail problems in machine learning models. The emergence of long-tail problems is partly due to incomplete coverage of model training data, and partly because certain scenarios were not taken into account at the beginning of model design. These scenarios may include very loud noise in the recognition environment, or multi-person discussion, or a large number of uncommon special terms in the scene to be recognized (such as primary hyperoxaluria type 1, nusinersen sodium, etc.). For the first part of the reasons, researchers can solve it by expanding the data set, which is often the preferred solution to solve the long-tail problem, but too much data often increases the complexity of model calculations; for the second reason, it is more about solving it from the perspective of business logic and business scenarios, and there is no good solution.

[0004] Therefore, a new method, device, electronic device and computer-readable medium for processing user voice data are needed. Summary of the invention

[0005] In view of this, the present application provides a user voice data processing method, device, electronic device and computer-readable medium, which can quickly and accurately recognize the user's voice data, so as to interact with the user more smoothly, respond to the user's instructions quickly, improve the overall user satisfaction and improve business processing efficiency.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0007] According to one aspect of the present application, a method for processing user voice data is proposed, the method comprising: acquiring voice data from a user; inputting the voice data into an acoustic model to generate a plurality of phoneme data; inputting the phoneme data into a pronunciation model to generate a multidimensional character query matrix, the multidimensional character query matrix comprising a plurality of dimensions, each dimension comprising a plurality of characters and their corresponding probability values; inputting the multidimensional character query matrix into a language model to generate semantic text; and generating response data based on the semantic text to feed back to the user.

[0008] Optionally, it also includes: training a speech recognition model based on multiple historical speech data to generate the acoustic model; training a 3-gram model based on multiple historical phoneme data to generate the pronunciation model; training a GRU model based on multiple historical multi-dimensional character query matrices to generate the language model.

[0009] Optionally, the phoneme data is input into a pronunciation model to generate a multidimensional character query matrix, including: inputting the phoneme data into a pronunciation model; the pronunciation model divides the phoneme data into a plurality of short audios; counting the occurrence frequencies of the plurality of short audios to generate a plurality of groups of statistical results; assigning values ​​to characters in a multidimensional initial query matrix according to the plurality of groups of statistical results to generate the multidimensional character query matrix; wherein the character values ​​in the multidimensional initial query matrix are 0.

[0010] Optionally, the multidimensional character query matrix is ​​input into a language model to generate semantic text, including: inputting the multidimensional character query matrix into a language model; an update gate function and a reset gate function in the language model generate multiple update values ​​and multiple reset values ​​based on the multidimensional character query matrix; generating multiple character texts and their corresponding scores according to the multiple update values ​​and the multiple reset values; and using the character text with the highest score as the phonetic text.

[0011] Optionally, generating response data based on the semantic text to feed back to the user includes: generating response data of an intelligent voice robot based on the semantic text to conduct an intelligent voice conversation with the user; and / or generating user instructions based on the semantic text, and generating response data based on the user instructions to feed back to the user.

[0012] Optionally, the 3-gram model is trained based on multiple historical phoneme data to generate the pronunciation model, including: dividing the multiple historical phoneme data into multiple short audios to generate multiple short audio groups; setting a character label for each short audio in the multiple short audio groups; and training the 3-gram model based on the multiple short audio groups with character labels to generate the pronunciation model.

[0013] Optionally, the 3-gram model is trained based on multiple short audio groups with character labels to generate the pronunciation model, including: generating a multidimensional initial query matrix based on the character labels; inputting multiple short audio groups with character labels into the 3-gram model respectively; generating multiple historical multidimensional character query matrices according to the calculation results of the 3-gram model; adjusting 3-gram model parameters of the 3-gram model according to the multiple historical multidimensional character query matrices; and generating the pronunciation model when the 3-gram model parameters converge.

[0014] Optionally, generating a multidimensional initial query matrix based on character tags includes: determining the dimension of the multidimensional initial query matrix; and mapping the character tags to the multidimensional initial query matrix based on hash values ​​of the character tags.

[0015] Optionally, the GRU model is trained based on multiple historical multidimensional character query matrices to generate the language model, including: setting text labels for the multiple historical multidimensional character query matrices respectively; training the GRU model based on the multiple historical multidimensional character query matrices with text labels, and generating GRU model parameters based on the training results; and generating the language model when the GRU model parameters converge.

[0016] According to one aspect of the present application, a user voice data processing device is proposed, which includes: a voice module for acquiring voice data from a user; a phoneme module for inputting the voice data into an acoustic model to generate a plurality of phoneme data; a character module for inputting the phoneme data into a pronunciation model to generate a multi-dimensional character query matrix, wherein the multi-dimensional character query matrix includes a plurality of dimensions, each dimension includes a plurality of characters and their corresponding probability values; a text module for inputting the multi-dimensional character query matrix into a language model to generate semantic text; and a feedback module for generating response data based on the semantic text to feed back to the user.

[0017] Optionally, it also includes: an acoustic model module, which is used to train a speech recognition model based on multiple historical speech data to generate the acoustic model; a pronunciation model module, which is used to train a 3-gram model based on multiple historical phoneme data to generate the pronunciation model; and a language model module, which is used to train a GRU model based on multiple historical multi-dimensional character query matrices to generate the language model.

[0018] According to one aspect of the present application, an electronic device is proposed, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0019] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0020] According to the user voice data processing method, device, electronic device and computer-readable medium of the present application, voice data from a user is obtained; the voice data is input into an acoustic model to generate a plurality of phoneme data; the phoneme data is input into a pronunciation model to generate a multi-dimensional character query matrix, wherein the multi-dimensional character query matrix includes a plurality of dimensions, each dimension includes a plurality of characters and their corresponding probability values; the multi-dimensional character query matrix is ​​input into a language model to generate a semantic text; and response data is generated based on the semantic text to be fed back to the user, thereby being able to quickly and accurately recognize the user's voice data, thereby interacting with the user more smoothly, quickly responding to the user's instructions, improving the overall user satisfaction and improving the business processing efficiency.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects, features and advantages of the present application will become more apparent by describing in detail the exemplary embodiments thereof with reference to the accompanying drawings. The accompanying drawings described below are only some embodiments of the present application, and it is clear to a person skilled in the art that other accompanying drawings can be obtained from these accompanying drawings without creative effort.

[0023] Figure 1 The present invention is a system block diagram showing a method and device for processing user voice data according to an exemplary embodiment.

[0024] Figure 2 The figure is a flow chart of a method for processing user voice data according to an exemplary embodiment.

[0025] Figure 3 is a flow chart of a method for processing user voice data according to another exemplary embodiment.

[0026] Figure 4 is a schematic diagram of a method for processing user voice data according to another exemplary embodiment.

[0027] Figure 5 is a flow chart of a method for processing user voice data according to another exemplary embodiment.

[0028] Figure 6 The figure is a block diagram of a device for processing user voice data according to an exemplary embodiment.

[0029] Figure 7 is a block diagram of a device for processing user voice data according to another exemplary embodiment.

[0030] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment.

[0031] Fig. 9 It is a block diagram of a computer-readable medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0033] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0036] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0037] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.

[0038] The inventor of this application found that there are two ways to solve the long tail problem in the existing technical solution:

[0039] The first idea is to expand the training data set. However, in commercial use, this method will cause the solution to be uncontrollable. Uncontrollability is reflected in two aspects. The first aspect is that data set expansion is not an easy task. For the speech recognition scenario in this application, the long-tail problem often appears in rare characters and rare words in the training data set. For example, taking daily conversation data to train an academic report speech will cause inaccurate recognition of many professional terms. The fundamental reason is that these professional terms are difficult to appear in the training data. On the second hand, there will be a situation of treating the head with a headache and the foot with a foot with a foot pain, and even the "backlash" of the long-tail effect. By expanding the training data of a large number of long-tail scenarios, the weight of the long-tail scenario is greatly improved, lacking a sense of primary and secondary hierarchy. That is, the original training data is a common normal distribution that is expanded to a situation close to a uniform distribution. Still taking speech recognition as an example, assuming that the acoustic model decodes that the user is saying (bi xi), then there are the following reference words: nasal breath, pen wash, Bixi, tourmaline, and Bixi. The "Bixi" here is a typical vocabulary in the long-tail scenario, that is, the probability of appearing in the training set is extremely low. However, if we artificially expand the training data with "Bixi" to solve the long-tail problem of "bixi", the probability of Bixi being output as the recognition result will be greatly increased, which in turn reduces the probability of outputting the other "bixi".

[0040] The second idea is to add a large language model, which is also a common solution in existing technologies. However, the advantages and disadvantages of language models are obvious. The advantage is that if the computing power and data are sufficient, the ability of the entire speech recognition system to solve long-tail problems will be greatly improved. The disadvantage is also obvious, that is, in commercial scenarios, the system cost will be very sensitive, and the computing power and the size of the deployed model are proportional to the system cost, so compression is needed.

[0041] In view of various technical difficulties existing in the prior art, this application proposes a user voice data processing method, which can solve the long-tail problem existing in the end-to-end voice recognition model at a low cost. The method is described in detail below with the help of specific embodiments.

[0042] Figure 1 The present invention is a system block diagram showing a method and device for processing user voice data according to an exemplary embodiment.

[0043] like Figure 1 As shown, the system architecture 10 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0044] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as financial service applications, shopping applications, web browser applications, instant messaging tools, email clients, social platform software, etc.

[0045] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0046] Terminal devices 101, 102, 103 may, for example, obtain voice data from a user; terminal devices 101, 102, 103 may, for example, input the voice data into an acoustic model to generate a plurality of phoneme data; terminal devices 101, 102, 103 may, for example, input the phoneme data into a pronunciation model to generate a multi-dimensional character query matrix, wherein the multi-dimensional character query matrix includes a plurality of dimensions, each dimension includes a plurality of characters and their corresponding probability values; terminal devices 101, 102, 103 may, for example, input the multi-dimensional character query matrix into a language model to generate semantic text; terminal devices 101, 102, 103 may, for example, generate response data based on the semantic text to feed back to the user.

[0047] The server 105 may be a server that provides various services, such as a background management server that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received user voice data, and feed back the processing results (such as response issuance) to the terminal devices 101, 102, and 103.

[0048] The server 105 may, for example, obtain voice data from the user; the server 105 may, for example, generate a plurality of phoneme data by inputting the voice data into an acoustic model; the server 105 may, for example, input the phoneme data into a pronunciation model to generate a multi-dimensional character query matrix, wherein the multi-dimensional character query matrix includes a plurality of dimensions, each dimension includes a plurality of characters and their corresponding probability values; the server 105 may, for example, input the multi-dimensional character query matrix into a language model to generate semantic text; the server 105 may, for example, generate response data based on the semantic text to feed back to the user.

[0049] The server 10 can also, for example, train a speech recognition model based on multiple historical speech data to generate the acoustic model; the server 10 can also, for example, train a 3-gram model based on multiple historical phoneme data to generate the pronunciation model; the server 10 can also, for example, train a GRU model based on multiple historical multi-dimensional character query matrices to generate the language model.

[0050] The server 10 may also deploy, for example, an acoustic model, a pronunciation model, and a language model on the terminal devices 101 , 102 , and 103 .

[0051] The server 105 may be a physical server, or may be composed of multiple servers. It should be noted that the user voice data processing method provided in the embodiment of the present application may be executed by the server 105 and / or the terminal devices 101, 102, 103. Accordingly, the user voice data processing device may be arranged in the server 105 and / or the terminal devices 101, 102, 103. The data processing end for machine learning model training is generally located in the server 105.

[0052] Figure 2 2 is a flowchart of a method for processing user voice data according to an exemplary embodiment. The method 20 for processing user voice data at least includes steps S202 to S210.

[0053] like Figure 2 As shown, in S202, voice data from the user is obtained. The user's voice data can be obtained through a preset application or a web page, and the user's voice data can also be initially processed, which can specifically include sampling processing, data compression processing, noise filtering processing, etc.

[0054] In S204, the speech data is input into an acoustic model to generate a plurality of phoneme data. The acoustic model can identify the phonemes that are most likely to be present in the audio sample of the speech data. The acoustic model takes a waveform, divides it into smaller time periods, performs frequency analysis, and then outputs a probability distribution of all three phoneme dynamics for the particular input. The waveform frequency vector is matched with the probability distribution to identify which phonemes are more likely to be included in the audio sample, thereby providing a sequence of phonemes present in the input over a period of time.

[0055] In one embodiment, the speech recognition model may be trained based on a plurality of historical speech data to generate the acoustic model. Optionally, the speech recognition model may be specifically a Conformer model.

[0056] Specifically, for each object set, an initial acoustic model is constructed respectively, and the object information of each object in the object set is input into the initial acoustic model to obtain a predicted label, and the predicted label is compared with the corresponding real label to determine whether the predicted label is consistent with the real label, and the number of predicted labels consistent with the real label is counted, and the proportion of the number of predicted labels consistent with the real label in the number of all predicted labels is calculated. If the proportion is greater than or equal to the preset proportion value, the initial acoustic model converges to obtain a trained acoustic model. If the proportion is less than the preset proportion value, the parameters in the initial acoustic model are adjusted, and the predicted labels of each object are re-predicted by the adjusted initial acoustic model until the proportion is greater than or equal to the preset proportion value. Among them, the method of adjusting the parameters in the initial acoustic model can be performed using a stochastic gradient descent algorithm, a gradient descent algorithm, or a normal equation.

[0057] If the number of times the parameters of the initial acoustic model are adjusted exceeds a preset number, the model used to construct the initial acoustic model may be replaced to improve the model training efficiency.

[0058] In S206, the phoneme data is input into the pronunciation model to generate a multi-dimensional character query matrix, wherein the multi-dimensional character query matrix includes multiple dimensions, each dimension includes multiple characters and their corresponding probability values. The pronunciation model uses the phoneme probability distribution of the acoustic model and checks it with reference to the multi-dimensional character query matrix to define a valid phoneme sequence for a specific language word, and then limits the possible phoneme sequences to words that are meaningful in the language.

[0059] As described above, in order to solve the long-tail phenomenon as much as possible without increasing the computational burden of the entire machine learning model, the multi-dimensional character query matrix cannot contain all the valid phonemes of all words in every language in the world. The dimension of the multi-dimensional character query matrix can be U*E, where U represents the number of each character, and E represents the number of possibilities of the character. For example, if the multi-dimensional character query matrix can match 10,000 words, the number of U is 10,000. And according to different combinations of pronunciations, the possibilities of each word appearing in different combinations are different. For example, for the character "比", following the context, the possibilities are "10%, 25%, 80%, 100%", so for the word "比", the corresponding number of E is 4. Different words correspond to different E values.

[0060] In one embodiment, the phoneme data can be input into a pronunciation model; the pronunciation model divides the phoneme data into multiple short audio clips (if in the previous step, the acoustic model has already divided the speech data into multiple short audio clips, then in this step, the short audio clip data can be directly retrieved); the occurrence frequencies of the multiple short audio clips are statistically analyzed to generate multiple groups of statistical results; the characters in the multi-dimensional initial query matrix are assigned values according to the multiple groups of statistical results to generate the multi-dimensional character query matrix; where the character values in the multi-dimensional initial query matrix are 0.

[0061] The detailed content of "inputting the phoneme data into a pronunciation model to generate a multi-dimensional character query matrix" is in Figure 3 the corresponding embodiment for detailed description.

[0062] In one embodiment, the 3-gram model can be trained based on multiple historical phoneme data to generate the pronunciation model. The relevant steps will be in Figure 4 the corresponding embodiment for detailed description.

[0063] In S208, the multi-dimensional character query matrix is input into a language model to generate a semantic text. The multi-dimensional character query matrix can be input into the language model; the update gate function and the reset gate function in the language model generate multiple update values and multiple reset values based on the multi-dimensional character query matrix; multiple character texts and their corresponding scores are generated according to the multiple update values and the multiple reset values; the character text with the highest score is used as the speech text.

[0064] In one embodiment, the GRU model is trained based on multiple historical multi-dimensional character query matrices to generate the language model. More specifically, text labels can be set for the multiple historical multi-dimensional character query matrices respectively; the GRU model is trained based on the multiple historical multi-dimensional character query matrices with text labels, and GRU model parameters are generated based on the training results; when the GRU model parameters converge, the language model is generated.

[0065] In S210, response data is generated based on the semantic text to be fed back to the user. Response data of an intelligent voice robot is generated based on the semantic text to conduct an intelligent voice dialogue with the user; and / or a user instruction is generated based on the semantic text, and response data is generated based on the user instruction to be fed back to the user.

[0066] The calculation results of the speech recognition models generated using different schemes are shown in the following table:

[0067]

[0068] Among them, rare words in the model training set refer to words that appear less than 3 times in the training set used to train the model in stage a.

[0069] From the above comparison, it can be seen that the user voice data processing method of the present application solves most of the long-tail problems at a relatively low cost; while maintaining the real-time processing rate, the accuracy of the model is improved.

[0070] According to the user voice data processing method of the present application, voice data from the user is obtained; the voice data is input into an acoustic model to generate a plurality of phoneme data; the phoneme data is input into a pronunciation model to generate a multi-dimensional character query matrix, wherein the multi-dimensional character query matrix includes a plurality of dimensions, each dimension includes a plurality of characters and their corresponding probability values; the multi-dimensional character query matrix is ​​input into a language model to generate a semantic text; and response data is generated based on the semantic text to be fed back to the user, thereby being able to quickly and accurately recognize the user's voice data, thereby interacting with the user more smoothly, quickly responding to the user's instructions, improving the overall user satisfaction, and improving the business processing efficiency.

[0071] It should be clearly understood that the present application describes how to form and use specific examples, but the principles of the present application are not limited to any details of these examples. On the contrary, based on the teaching of the content disclosed in the present application, these principles can be applied to many other embodiments.

[0072] Figure 3 is a flow chart of a method for processing user voice data according to another exemplary embodiment. Figure 3 The process 30 shown is for Figure 2 The process shown is a detailed description of S206 "inputting the phoneme data into the pronunciation model to generate a multi-dimensional character query matrix".

[0073] like Figure 3As shown, in S302, the phoneme data is input into the pronunciation model. The phoneme data may contain single-phoneme information or multi-phoneme information. More specifically, what the pronunciation model outputs are the phonemes of each letter, such as "a", "b", etc., or multi-phoneme information such as "bike", "bicycle", etc. The specific situation can be fine-tuned according to the user's speech habits and the current scenario.

[0074] In S304, the pronunciation model divides the phoneme data into multiple short audio segments. If the acoustic model has already divided the speech data into multiple short audio segments, in this step, the short audio data can be directly retrieved. When the phoneme data output by the acoustic model is longer than the threshold, for example, when the phoneme data output by the acoustic model is "I will back", the phoneme data can be divided again to ensure that each short audio segment contains one character or word information as much as possible.

[0075] In S306, the occurrence frequencies of the multiple short audio segments are statistically analyzed to generate multiple sets of statistical results. The pronunciation model attempts to match this phoneme data with the words in the multi-dimensional initial query matrix and assigns values to the characters in the multi-dimensional initial query matrix according to the matching results. The character value represents the probability that this phoneme data is this character.

[0076] In S308, values are assigned to the characters in the multi-dimensional initial query matrix according to the multiple sets of statistical results to generate the multi-dimensional character query matrix. Among them, the character values in the multi-dimensional initial query matrix are 0.

[0077] In one embodiment, Figure 4 is a schematic diagram of a multi-dimensional character query matrix containing 4 characters. Depending on the context, the possibility of the character "比" (bi) appearing is: 0%, 10%, or 80%; the possibility of the character "雨" (yu) appearing is 0%, 40%, or 60%; the possibility of the character "笔" (bi) appearing is 0%, 40%, 80%; the possibility of the character "中" (zhong) appearing is 0%, 40%, 80%. And when the possibility of the character "比" is 0%, the possibility of the character "雨" is 40%, the possibility of the character "笔" appearing is 80%, and the possibility of the character "中" appearing is 0%; the above combination serves as one dimension of the multi-dimensional character query matrix. Other possibilities are arranged in sequence until all possibilities are filled. For example, when the possibility of the character "比" is 80%, the possibility of the character "雨" is 60%, the possibility of the character "笔" appearing is 0%, and the possibility of the character "中" appearing is 40%; when the possibility of the character "比" is 10%, the possibility of the character "雨" is 0%, the possibility of the character "笔" appearing is 40%, and the possibility of the character "中" appearing is 80%. Values are assigned to the characters in the first, second, and third dimensions of the multi-dimensional initial query matrix according to the above results, and the other dimensions can all maintain the original assigned value of 0.

[0078] Figure 5is a flow chart of a method for processing user voice data according to another exemplary embodiment. Figure 5 The process 50 shown is a detailed description of “training the 3-gram model based on a plurality of historical phoneme data to generate the pronunciation model”.

[0079] like Figure 5 As shown, in S502, a plurality of historical phoneme data are respectively divided into a plurality of short audios to generate a plurality of short audio groups.

[0080] In S504, a character label is set for each short audio in the plurality of short audio groups. A corresponding character label is set for each short audio, and the character label can be Chinese or foreign characters.

[0081] In S506, the 3-gram models are trained respectively based on a plurality of short audio groups with character labels to generate the pronunciation model.

[0082] In one embodiment, a multidimensional initial query matrix can be generated based on character tags; multiple short audio groups with character tags are respectively input into a 3-gram model; multiple historical multidimensional character query matrices are generated according to the calculation results of the 3-gram model; 3-gram model parameters of the 3-gram model are adjusted according to the multiple historical multidimensional character query matrices; and when the 3-gram model parameters converge, the pronunciation model is generated.

[0083] Among them, generating a multidimensional initial query matrix based on character tags includes: determining the dimension of the multidimensional initial query matrix; mapping the character tags to the multidimensional initial query matrix based on the hash values ​​of the character tags. More specifically, according to the time and space requirements for calculating the machine learning model, it is first determined that the number of characters that can be matched in the multidimensional initial query matrix is ​​10,000, and the number of character tags generated after annotation in the training data is 11,000. In this case, the 11,000 character tags can be mapped to 10,000 through the hash values. It can be seen that the fewer hash conflicts processed here, the more accurate the multidimensional initial query matrix.

[0084] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided in the present application are executed. The program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0085] In addition, it should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0086] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0087] Figure 6 FIG. 1 is a block diagram of a user voice data processing device according to another exemplary embodiment. Figure 6 As shown, the user voice data processing device 60 includes: a voice module 602 , a phoneme module 604 , a character module 606 , a text module 608 , and a feedback module 610 .

[0088] The voice module 602 is used to obtain voice data from the user;

[0089] The phoneme module 604 is used to input the speech data into the acoustic model to generate a plurality of phoneme data;

[0090] The character module 606 is used to input the phoneme data into the pronunciation model to generate a multi-dimensional character query matrix, wherein the multi-dimensional character query matrix includes multiple dimensions, each dimension includes multiple characters and their corresponding probability values;

[0091] The text module 608 is used to input the multi-dimensional character query matrix into the language model to generate semantic text;

[0092] The feedback module 610 is used to generate response data based on the semantic text to feed back to the user.

[0093] Figure 7 FIG. 1 is a block diagram of a user voice data processing device according to an exemplary embodiment. Figure 7 As shown, the user voice data processing device 70 includes: an acoustic model module 702 , a pronunciation model module 704 , and a language model module 706 .

[0094] The acoustic model module 702 is used to train a speech recognition model based on a plurality of historical speech data to generate the acoustic model.

[0095] The pronunciation model module 704 is used to train the 3-gram model based on multiple historical phoneme data to generate the pronunciation model; the pronunciation model module 704 is also used to divide the multiple historical phoneme data into multiple short audios to generate multiple short audio groups; set character labels for each short audio in the multiple short audio groups; and train the 3-gram model based on the multiple short audio groups with character labels to generate the pronunciation model.

[0096] The language model module 706 is used to train the GRU model based on multiple historical multi-dimensional character query matrices to generate the language model. The language model module 706 is also used to set text labels for the multiple historical multi-dimensional character query matrices respectively; train the GRU model based on the multiple historical multi-dimensional character query matrices with text labels, and generate GRU model parameters based on the training results; when the GRU model parameters converge, generate the language model.

[0097] According to the user voice data processing device of the present application, voice data from the user is obtained; the voice data is input into an acoustic model to generate a plurality of phoneme data; the phoneme data is input into a pronunciation model to generate a multi-dimensional character query matrix, wherein the multi-dimensional character query matrix includes a plurality of dimensions, each dimension includes a plurality of characters and their corresponding probability values; the multi-dimensional character query matrix is ​​input into a language model to generate a semantic text; and response data is generated based on the semantic text to be fed back to the user, thereby being able to quickly and accurately recognize the user's voice data, thereby interacting with the user more smoothly, quickly responding to the user's instructions, improving the overall user satisfaction, and improving the business processing efficiency.

[0098] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment.

[0099] Refer to the following Figure 8 hereinafter describes an electronic device 800 according to this embodiment of the present application. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0100] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), a display unit 840, etc.

[0101] The storage unit stores a program code, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps of various exemplary embodiments of the present application in this specification. For example, the processing unit 810 can perform the following steps: Figure 2 , Figure 3 , Figure 4 Follow the steps shown in .

[0102] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0103] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0104] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0105] The electronic device 800 may also communicate with one or more external devices 800' (e.g., keyboards, pointing devices, Bluetooth devices, etc.) to enable a user to communicate with the electronic device 800, and / or any device (e.g., routers, modems, etc.) that the electronic device 800 can communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface 850. In addition, the electronic device 800 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 860. The network adapter 860 may communicate with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0106] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Fig. 9As shown, the technical solution according to the implementation mode of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation mode of the present application.

[0107] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0108] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0109] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0110] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the following functions: obtaining voice data from a user; inputting the voice data into an acoustic model to generate a plurality of phoneme data; inputting the phoneme data into a pronunciation model to generate a multidimensional character query matrix, wherein the multidimensional character query matrix includes a plurality of dimensions, each dimension including a plurality of characters and their corresponding probability values; inputting the multidimensional character query matrix into a language model to generate a semantic text; generating response data based on the semantic text to feed back to the user. The computer-readable medium implements the following functions: training a speech recognition model based on a plurality of historical voice data to generate the acoustic model; training a 3-gram model based on a plurality of historical phoneme data to generate the pronunciation model; training a GRU model based on a plurality of historical multidimensional character query matrices to generate the language model.

[0111] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0112] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present application.

[0113] The exemplary embodiments of the present application are specifically shown and described above. It should be understood that the present application is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present application is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A method for processing user voice data, characterized in that: include: Acquire voice data from the user; Inputting the speech data into an acoustic model to generate a plurality of phoneme data; inputting the phoneme data into a pronunciation model; The pronunciation model divides the phoneme data into a plurality of short audios; Counting the occurrence frequencies of the multiple short audios to generate multiple groups of statistical results; Assigning values ​​to characters in the multidimensional initial query matrix according to the multiple sets of statistical results to generate a multidimensional character query matrix; wherein the character value in the multidimensional initial query matrix is ​​0, the multidimensional character query matrix includes multiple dimensions, each dimension includes multiple characters and a probability value corresponding to each character; Inputting the multi-dimensional character query matrix into a language model; The update gate function and the reset gate function in the language model generate multiple update values ​​and multiple reset values ​​based on the multi-dimensional character query matrix; Generate a plurality of character texts and their corresponding scores according to the plurality of update values ​​and the plurality of reset values; The character text with the highest score is taken as the semantic text; Response data is generated based on the semantic text to be fed back to the user.

2. The method according to claim 1, characterized in that Also includes: Training a speech recognition model based on a plurality of historical speech data to generate the acoustic model; Training a 3-gram model based on a plurality of historical phoneme data to generate the pronunciation model; The GRU model is trained based on a plurality of historical multi-dimensional character query matrices to generate the language model.

3. The method according to claim 1, characterized in that Generating response data based on the semantic text to feed back to the user, including: Generate response data of an intelligent voice robot based on the semantic text to conduct an intelligent voice dialogue with the user; and / or A user instruction is generated based on the semantic text, and response data is generated based on the user instruction to be fed back to the user.

4. The method according to claim 2, characterized in that The 3-gram model is trained based on a plurality of historical phoneme data to generate the pronunciation model, including: Divide the plurality of historical phoneme data into a plurality of short audios respectively, and generate a plurality of short audio groups; Setting a character label for each short audio in the plurality of short audio groups; The 3-gram models are trained respectively based on a plurality of short audio groups with character labels to generate the pronunciation model.

5. The method according to claim 4, characterized in that The 3-gram models are trained based on multiple short audio groups with character labels to generate the pronunciation model, including: Generate a multi-dimensional initial query matrix based on character labels; Multiple short audio groups with character labels are input into the 3-gram model separately; Generate multiple historical multi-dimensional character query matrices according to the calculation results of the 3-gram model; adjusting 3-gram model parameters of the 3-gram model according to a plurality of historical multi-dimensional character query matrices; When the 3-gram model parameters converge, the pronunciation model is generated.

6. The method according to claim 5, characterized in that Generate a multi-dimensional initial query matrix based on character labels, including: Determining the dimension of the multidimensional initial query matrix; The character tags are mapped into the multi-dimensional initial query matrix based on the hash values ​​of the character tags.

7. The method according to claim 2, characterized in that The GRU model is trained based on a plurality of historical multi-dimensional character query matrices to generate the language model, including: Setting text labels for the plurality of historical multi-dimensional character query matrices respectively; Training a GRU model based on the multiple historical multi-dimensional character query matrices with text labels, and generating GRU model parameters based on the training results; When the GRU model parameters converge, the language model is generated.

8. A user voice data processing device, characterized in that: include: A voice module, used to obtain voice data from users; A phoneme module, used for inputting the speech data into an acoustic model to generate a plurality of phoneme data; A character module, used for inputting the phoneme data into a pronunciation model; The pronunciation model divides the phoneme data into a plurality of short audios; Counting the occurrence frequencies of the multiple short audios to generate multiple groups of statistical results; Assigning values ​​to characters in the multidimensional initial query matrix according to the multiple sets of statistical results to generate a multidimensional character query matrix; wherein the character value in the multidimensional initial query matrix is ​​0, the multidimensional character query matrix includes multiple dimensions, each dimension includes multiple characters and a probability value corresponding to each character; A text module is used to input the multidimensional character query matrix into a language model; an update gate function and a reset gate function in the language model generate a plurality of update values ​​and a plurality of reset values ​​based on the multidimensional character query matrix; a plurality of character texts and their corresponding scores are generated according to the plurality of update values ​​and the plurality of reset values; and the character text with the highest score is used as a semantic text; A feedback module is used to generate response data based on the semantic text to feed back to the user.

9. The device according to claim 8, characterized in that Also includes: An acoustic model module, used for training a speech recognition model based on a plurality of historical speech data to generate the acoustic model; A pronunciation model module, used for training a 3-gram model based on a plurality of historical phoneme data to generate the pronunciation model; The language model module is used to train the GRU model based on multiple historical multi-dimensional character query matrices to generate the language model.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images