Collaborative recitation method and device, computer storage medium and terminal

Abstract

The application provides a collaborative recitation method and device, a computer storage medium and a terminal, and relates to the technical field of speech recognition and evaluation. The method comprises the following steps: inputting a to-be-recognized speech into a speech recognition model, determining the content corresponding to the to-be-recognized speech according to the output of the speech recognition model, and obtaining to-be-recognized content; inputting the to-be-recognized content into an intention recognition model, determining the target intention type corresponding to the to-be-recognized content according to the output of the intention recognition model, wherein the intention type output by the intention recognition model comprises at least one of the following: starting recitation, reciting, requesting help and completing recitation; and according to the target intention type, the collaborative recitation processing of the to-be-recognized content is realized. The speech recognition model and the intention recognition model in the scheme can assist students to be familiar with the content of the article and the poem in the liberal arts recitation scene, liberate manpower and improve the learning efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of speech recognition technology and evaluation, and particularly relates to a collaborative recitation method and device, a computer readable storage medium and a terminal. BACKGROUND

[0002] When learning liberal arts courses, students often need to recite articles and poems, and sometimes need classmates or parents to help listen or play back the recording to find the mistakes in the recitation process. Or in the recitation process, unfamiliar places need to be prompted by others.

[0003] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0004] The purpose of the present application is to provide a collaborative recitation method and device, a computer readable storage medium and a terminal, which at least partly improves the recitation efficiency of articles and poems.

[0005] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0006] According to a first aspect of the present application, a collaborative recitation method is provided, the method comprising: inputting a to-be-recognized speech into a speech recognition model, determining a content corresponding to the to-be-recognized speech according to an output of the speech recognition model, and obtaining a to-be-recognized content; inputting the to-be-recognized content into an intent recognition model, determining a target intent type corresponding to the to-be-recognized content according to an output of the intent recognition model, wherein the intent type output by the intent recognition model includes at least one of starting recitation, reciting, requesting help, and completing recitation; and implementing collaborative recitation processing of the to-be-recognized content according to the target intent type.

[0007] In an embodiment of the present application, the collaborative recitation of the to-be-recognized content according to the target intent type comprises: in the case that the target intent type is starting recitation, determining a name of a target recitation content in the to-be-recognized content, and obtaining the target recitation content according to the name of the target recitation content; and outputting a prompt for starting recitation.

[0008] In an embodiment of the present application, the collaborative recitation of the to-be-recognized content according to the target intent type comprises: in the case that the intent type is reciting, matching the to-be-recognized content with the target recitation content to obtain recited content and to-be-recited content.

[0009] In an embodiment of the present application, the collaborative recitation of the to-be-identified content according to the target intention type includes: in the case that the intention type is a request for help, prompting to-be-recited content in the target recitation content according to the recited content.

[0010] In an embodiment of the present application, the collaborative recitation of the to-be-identified content according to the target intention type includes: in the case that the intention type is a request for help, prompting to-be-recited content in the target recitation content according to the recited content.

[0011] In an embodiment of the present application, the method further includes: training the speech recognition model; the training of the speech recognition model includes: obtaining N training samples, wherein the i th training sample is the i th speech, N is a positive integer, and i is a positive integer less than or equal to N; performing feature extraction on the i th speech to obtain M speech feature vectors, wherein M is a positive integer; inputting the M speech feature vectors into the speech recognition model to perform speech recognition processing on each speech feature vector by the speech recognition model to obtain M character vectors output by the speech recognition model; determining a first loss function corresponding to the i th training sample according to the M character vectors and the M speech feature vectors; and optimizing the parameters of the speech recognition model according to the first loss function.

[0012] In an embodiment of the present application, the determination of the first loss function corresponding to the i th training sample according to the M character vectors and the M speech feature vectors includes: determining a m th joint probability according to a m th character vector and a m th speech feature vector, wherein m is a positive integer less than or equal to M; and determining a m th first loss function according to the m th joint probability.

[0013] In an embodiment of the present application, the speech recognition model includes: a spectrum enhancement layer, a convolution sampling layer, a linear layer, an anti-overfitting layer, and a convolution enhancement layer; wherein the spectrum enhancement layer is configured to perform random masking processing on the M feature vectors to obtain a first output; the convolution sampling layer is configured to perform convolution processing on the first output to obtain a second output; the linear layer is configured to perform linear transformation processing on the second output to obtain a third output; the anti-overfitting layer is configured to perform random feature dropout on the third output to obtain a fourth output; and the convolution enhancement layer is configured to perform convolution enhancement processing on the fourth output to obtain the M character vectors.

[0014] In an embodiment of the present application, the calculation method of the m th first loss function is f(x) =∑ (X，Y)∈D(-logP(Y|X)), wherein f(x) is the mth first loss function, X is the ith speech, Y is a character in the ith speech, D is a sample set, (X, Y) is a set of samples in the sample set D, and P(Y|X) is the mth joint probability.

[0015] In an embodiment of the present application, the method further comprises training the intent recognition model, and the training of the intent recognition model comprises: obtaining S training samples, wherein the st training sample comprises an st text and an intent type corresponding to the st text, the st text comprises K words, S is a positive integer, K is a positive integer, and s is a positive integer less than or equal to S; identifying the K words, and inputting the identified K words into the intent recognition model; obtaining K second probability vectors output by the intent recognition model according to a hidden layer and parameters of the speech recognition model; determining a second loss function of the st text according to the K second probability vectors and the K words; and determining the parameters of the intent recognition model according to the second loss function.

[0016] In an embodiment of the present application, the determining of the second loss function of the st text according to the K second probability vectors and the K words comprises: determining a kth second loss function according to a vector composed of a character in a kth second probability vector and a character in a kth word, wherein k is a positive integer less than or equal to K.

[0017] In an embodiment of the present application, the method for calculating the kth second loss function is: k =-q k log(y k )-(1-q k )log(1-y k ), wherein L k is the kth second loss function, q k is the vector composed of the character in the kth word, y k is the kth second probability vector.

[0018] According to a second aspect of the present application, a collaborative recitation device is provided, the device comprising: a speech recognition model configured to: input a to-be-recognized speech into the speech recognition model, determine content corresponding to the to-be-recognized speech according to an output of the speech recognition model, and obtain to-be-recognized content; an intent recognition model configured to: input the to-be-recognized content into the intent recognition model, and determine a target intent type corresponding to the to-be-recognized content according to an output of the intent recognition model, wherein the intent type output by the intent recognition model comprises at least one of a start recitation, an ongoing recitation, a request for help, and a complete recitation; and a processing module configured to: perform collaborative recitation processing on the to-be-recognized content according to the target intent type.

[0019] According to a third aspect of the present application, a terminal is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the collaborative recitation method of the first aspect when executing the computer program.

[0020] According to a fourth aspect of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, wherein the computer program is executable on a processor to implement the collaborative recitation method of the first aspect.

[0021] The collaborative recitation method and device, the computer readable storage medium, and the terminal provided by the embodiments of the present application have the following technical effects:

[0022] In the collaborative recitation process provided by the embodiments of the present application, the to-be-recognized speech is input into the speech recognition model, the content corresponding to the to-be-recognized speech is determined according to the output of the speech recognition model, and the to-be-recognized content is obtained; the to-be-recognized content is input into the intent recognition model, and the target intent type corresponding to the to-be-recognized content is determined according to the output of the intent recognition model, wherein the intent type output by the intent recognition model comprises at least one of a start recitation, an ongoing recitation, a request for help, and a complete recitation; and the collaborative recitation processing on the to-be-recognized content is implemented according to the target intent type. The speech recognition model and the intent recognition model in the present solution can assist students to familiarize with the content of articles and poems in the humanities recitation scene, liberate manpower, and improve learning efficiency.

[0023] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application. It is to be understood that the drawings are only schematic, and that they do not purport to be to scale with respect to one another. The embodiments presented herein are by way of example only, and various changes might be made by persons skilled in the art without departing from the scope of the application.

[0025] Figure 1 A flow chart of a method for cooperative recitation is shown according to an example embodiment of the present application;

[0026] Figure 2 A flow chart of a method for training a speech recognition model is shown according to an example embodiment of the present application;

[0027] Figure 3 A schematic diagram of acoustic wave data is shown according to an example embodiment of the present application;

[0028] Figure 4 A schematic diagram of a MFCC feature spectrum is shown according to an example embodiment of the present application;

[0029] Figure 5 A schematic diagram of a speech recognition model is shown according to an example embodiment of the present application;

[0030] Figure 6 A flow chart of a method for training an intent recognition model is shown according to an example embodiment of the present application;

[0031] Figure 7 A schematic diagram of an intent recognition model is shown according to an example embodiment of the present application;

[0032] Figure 8 A structural diagram of a cooperative recitation device is shown according to an example embodiment of the present application;

[0033] Figure 9 A block diagram of a terminal is shown according to an example embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the objects, technical solutions and advantages of the present application clearer, the following will further describe the embodiments of the present application with reference to the accompanying drawings.

[0035] The following description refers to the accompanying drawings, wherein like reference numerals refer to like elements in the several figures. The embodiments described are not meant to be all inclusive of all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0036] In the description of the present application, it should be understood that the terms "first", "second" and the like are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, "multiple" refers to two or more, unless otherwise specified. The association between the associated objects is described, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents a "or" relationship between the associated objects before and after it.

[0037] In the following, the various steps of the cooperative recitation method in the present example embodiment will be described in more detail in conjunction with the accompanying drawings and examples.

[0038] wherein, Figure 1 A flowchart of the cooperative recitation method according to an example embodiment of the present application is schematically shown.

[0039] The following will be described in detail Figure 1 The example embodiments shown are described in detail as follows:

[0040] S101, input the to-be-recognized voice to a voice recognition model, determine the content corresponding to the to-be-recognized voice according to the output of the voice recognition model, and obtain the to-be-recognized content.

[0041] For example, the cooperative recitation method provided by the present application can be applied in a system similar to human-computer cooperation. The voice recognition model and the intent recognition model and the language model in the following embodiments are all trained machine learning models and are configured in the human-computer cooperation system. In addition, the human-computer cooperation system should also be configured with a voice input module and a voice output module. The voice input module is used to obtain voice data of a user, and the voice output module is used for voice interaction with the user.

[0042] For example, when a user uses the human-computer cooperation system, the human-computer cooperation system can interact with the user. For example, when the user wants to recite, the user can say to the human-computer cooperation system: "I want to recite the text 'Jiang Jinjiu'". At this time, the human-computer cooperation system will obtain this segment of voice data of the user, i.e. the above-mentioned to-be-recognized voice, through the voice input module, and input the to-be-recognized voice to the voice recognition model. The voice recognition model will recognize the to-be-recognized voice to obtain the to-be-recognized content corresponding to the to-be-recognized voice. That is, after step S101, the system has obtained the specific content said by the user, and the specific intent of the content said by the user needs to be analyzed.

[0043] S102, input the to-be-recognized content into an intention recognition model, and determine a target intention type corresponding to the to-be-recognized content according to an output of the intention recognition model, wherein the intention type output by the intention recognition model comprises at least one of starting reciting, reciting, requesting help, and completing reciting.

[0044] S103, implement collaborative reciting processing on the to-be-recognized content according to the target intention type.

[0045] For example, after obtaining the to-be-recognized content, the to-be-recognized content is input into the intention recognition model, and the intention recognition model can recognize the intention type of the to-be-recognized content. For the example in step S101, after analyzing the sentence "I want to recite the textbook 'Drinking Alone Under the Moon'", the intention recognition model can obtain the intention type of the sentence as "starting reciting".

[0046] For example, after obtaining the intention of the user, the man-machine collaborative system can perform online collaborative reciting processing on the corresponding intention type. When the intention of the user is "starting reciting", the man-machine collaborative system can obtain the target reciting content according to the name of the target reciting content said by the user in the to-be-recognized content. For example, in the sentence "I want to recite the textbook 'Drinking Alone Under the Moon'", the name of the target reciting content is "Drinking Alone Under the Moon", and the man-machine collaborative system can obtain the content of the poem "Drinking Alone Under the Moon" and output the response "please start reciting" through the voice output module.

[0047] For example, after the user receives the response of the man-machine collaborative system, the user can start reciting the target reciting content. During the reciting process, the man-machine collaborative system synchronously performs steps S101-S103 to recognize the content and the intention of the user, and obtains the intention type as "reciting". The man-machine collaborative system can match the reciting content of the user with the target reciting content, and update the reciting progress to obtain the recited content and the to-be-recited content of the user.

[0048] For example, if the user forgets the next content after reciting to a certain place and needs to be reminded, the user can indicate the intention to the man-machine collaborative system. Still taking "Drinking Alone Under the Moon" as an example, if the user forgets the next sentence after reciting to "people who are happy should enjoy themselves, and should not let the wine jug be empty in front of the moon", the user can say to the man-machine collaborative system "what is the next sentence?", at this time, the man-machine collaborative system can perform steps S101-S103 to recognize the content and the intention type of the sentence, and obtain the intention type of the sentence as "requesting help". The man-machine collaborative system can interactively respond to the sentence, for example, can prompt the next sentence content as "the beginning of the next sentence is 'born', can you remember it?", if the user can remember it, the user can continue to recite, and the above embodiment content is executed; if the user still cannot remember it, the user can continue to be prompted.

[0049] Exemplarily, when the user recites the complete text or indicates that he has finished reciting, the man-machine collaborative system matches the recited content with the text of the target recited content, marks the wrong words, phrases, and segments, and sends the user a recitation accuracy report. And a review plan can be established for the user, for example, in the plan for weekly review in the future, according to the accuracy report, the user is prompted to review the unfamiliar passages regularly.

[0050] Through the collaborative recitation method provided in the present application, the efficiency and accuracy of the user in the recitation process can be improved, greatly helping the user to learn the articles and poems.

[0051] Figure 2 Exemplarily, a flowchart of a speech recognition model training method according to an exemplary embodiment of the present application is shown.

[0052] The following will be described in detail: Figure 2 The embodiments shown in the following will be described in detail:

[0053] S201, obtaining N training samples, wherein the i-th training sample is the i-th speech, N is a positive integer, and i is a positive integer less than or equal to N.

[0054] Exemplarily, when collecting training samples to establish a sample set of the speech recognition model, daily conversation audio, recitation audio of articles and poems, etc. can be collected. When training the speech recognition model, N training samples can be randomly obtained from the sample set for training, wherein the i-th training sample in the N samples is the i-th speech corresponding to the collected sample.

[0055] S202, feature extraction is performed on the i-th speech to obtain M speech feature vectors, wherein M is a positive integer.

[0056] Exemplarily, referring to the schematic diagram of the sound wave data shown in Figure 3 The i-th speech in the present embodiment is time series data in Waveform format. Figure 3 The waveform graph of the i-th speech in the exemplary experimental data is shown in the figure, wherein the horizontal axis represents the time length of the i-th speech, and the vertical axis represents the frequency of the i-th speech. After obtaining the i-th speech, in order to identify the information in the speech and discard other irrelevant information in the speech such as background noise, etc., the i-th speech needs to be processed by feature extraction. The feature extraction method used in the present embodiment is Mel-Frequency Cepstral Coefficients (MFCC), which is more in line with the human auditory principle. Referring to Figure 4The MFCC feature spectrum is shown in the schematic diagram, the horizontal axis also represents the time length, and the vertical axis represents the frequency. After the i-th speech is extracted, a 39-dimensional MFCC feature is obtained.

[0057] Exemplarily, when performing MFCC feature extraction, the speech data is subjected to frame processing, that is, the i-th speech is divided into several segments of fixed length, so that the change of speech data in each frame is not too drastic, facilitating the subsequent steps of MFCC feature extraction.

[0058] Exemplarily, since the i-th speech is subjected to frame processing, the obtained speech feature vector is also corresponding to each frame. In this embodiment, the number of frames of the i-th speech is denoted as M, and thus the obtained speech feature vector is also M.

[0059] S203, inputting the M speech feature vectors into the speech recognition model to perform speech recognition processing on each speech feature vector through the speech recognition model, and obtaining M character vectors output by the speech recognition model.

[0060] Exemplarily, referring to Figure 5 The schematic diagram of the speech recognition model is shown. The speech recognition model used in this embodiment is a convolution-enhanced transformation neural network structure, which is composed of a spectrum enhancement layer, a convolution sampling layer, a linear layer, an anti-overfitting layer, and a convolution enhancement layer from bottom to top.

[0061] Exemplarily, in the M speech feature vectors, each speech feature vector is sequentially input into the spectrum enhancement layer (A Simple Data Augmentation Method for Automatic Speech Recognition, SpecAug) first. The spectrum enhancement layer regards the MFCC spectrum of each speech feature vector as a picture, and performs random masking processing on a certain row or column in the picture to obtain a first output corresponding to each speech feature vector, so as to enhance the robustness of the model.

[0062] Exemplarily, after the spectrum enhancement layer, it enters the convolution sampling layer (Convolution Subsampling). Here, the first output corresponding to each speech feature vector is input into the convolution sampling layer. After convolution and sampling operation, a second output corresponding to each speech feature vector is obtained, so as to achieve the purpose of downsampling and improving operation efficiency.

[0063] Exemplarily, after the convolution sampling layer, a linear layer (also known as a full-connected layer) is entered, and each speech feature vector is input into the linear layer together with the second output corresponding to the speech feature vector, so as to classify the speech feature and obtain a third output corresponding to each speech feature vector.

[0064] Exemplarily, after the linear layer, a dropout layer is entered, and each speech feature vector is input into the dropout layer together with the third output corresponding to the speech feature vector, so as to perform random discarding processing on the speech feature and obtain a fourth output corresponding to each speech feature vector, so as to alleviate the occurrence of overfitting and achieve the effect of regularization to a certain extent.

[0065] Exemplarily, after the dropout layer, a conformer block is entered, and each speech feature vector is input into the conformer block together with the fourth output corresponding to the speech feature vector. The conformer block is a further enhancement of the transformer neural network structure in the related art, and its structure is connected by two feed forward modules (Feed Forward Module) connecting a multi-head attention module (Multi-Head Self Attention Module) and a convolution module (Convolution Module), and a last layer norm (Layernorm) is connected to the top layer for normalizing the layer.

[0066] Exemplarily, the output obtained by the conformer block is M character vectors, each of which corresponds to a vector composed of all characters in each frame of speech data recognized by the speech recognition model. Then, the probability of each frame of speech data corresponding to a certain character needs to be calculated to calculate the loss function of the speech recognition model and optimize the model. Exemplarily, the calculation method of the above probability is:

[0067] P(X, Y) = ln(∑p(y m |x m ))

[0068] wherein, for M speech feature vectors and M character vectors, y m is the mth character vector, x m is the mth speech feature vector. p(y m |x m) is the probability value of the mth character vector appearing under the condition of the mth feature vector. P(X, Y) is the mth joint probability, i.e., the probability of each frame of speech data corresponding to a certain character, X is the above-mentioned M feature vectors, and Y is the above-mentioned M character vectors. Because the joint probability corresponds to each speech feature vector, M joint probabilities are obtained.

[0069] S204, according to the M character vectors and the M speech feature vectors, determine the first loss function corresponding to the i th training sample.

[0070] S205, according to the first loss function, optimize the parameters of the speech recognition model.

[0071] Exemplarily, after obtaining the joint probability, the loss function can be calculated, and the calculation method is:

[0072] f(x) = ∑ (X，Y)∈D (-logP(Y|X))

[0073] Wherein, X is the above-mentioned i th speech, Y is the character in the i th speech, D is the sample set, (X, Y) is a group of samples in the sample set D, P(Y|X) is the above-mentioned m th joint probability, and f(x) is the m th first loss function, i.e., the first loss function corresponding to the m th feature vector in the M feature vectors.

[0074] Exemplarily, after calculating the M first loss functions, their sum can be minimized for training purposes to optimize the parameters of the speech recognition model. When the model is trained to a convergent state, the characters in the input speech to be measured can be more accurately predicted, thereby obtaining the text sequence corresponding to the speech to be measured.

[0075] After the speech recognition model performs recognition processing on the input speech data, multiple text sequences are usually generated as candidates, at which time a language model needs to be used to calculate the probability that the candidate text sequence is a meaningful sentence and to sort the probabilities to obtain the optimal text sequence. Therefore, when training the speech recognition model, a language model can be trained to enhance the prediction accuracy of the speech recognition model.

[0076] Exemplarily, for the language model, when establishing samples, the text data corresponding to the training samples of the speech recognition model can be used, for example, when establishing the sample set of the speech recognition model, daily conversation audio, recitation audio of articles and poems, etc. are collected, therefore, the text corresponding to these daily conversation audio, recitation audio of articles and poems, etc. can be collected at the same time, obtaining N training samples, and the i th training sample is used to train the language model.

[0077] Exemplarily, the language model used in the present application is a tri-gram language model, which measures the consistency of a word sequence with the language expression habit by a probability P(word sequence). In the tri-gram language model, the probability P(word sequence) is calculated by multiplying the probabilities of the occurrence of each word in the word sequence given the two words before it.

[0078]

[0079] Exemplarily, in the formula, for a text sequence with t words, the tri-gram language model calculates the probability of the occurrence of the w th word given the two words before it, w and w, and multiplies the probabilities to obtain the probability of the consistency of the text sequence with the expression habit. u u-2 u-1 u

[0080] The text sequence with the highest probability, i.e., the optimal text sequence, can be obtained by sorting the probabilities of each text sequence obtained above, thereby providing a more accurate output for the speech recognition model.

[0081] Figure 6 A flowchart of a method for training an intent recognition model according to an embodiment of the present application is schematically shown.

[0082] The following describes in detail the embodiment shown in Figure 6

[0083] S601, obtaining S training samples, wherein the s th training sample includes an s th text and an intent type corresponding to the s th text, the s th text includes K words, S is a positive integer, K is a positive integer, and s is a positive integer less than or equal to S.

[0084] Exemplarily, when establishing a sample set of the intent recognition model, a plurality of language texts and the intent types corresponding thereto can be selected to train the intent recognition model. After the sample set is established, S training samples can be randomly selected from the sample set, and the s th training sample includes an s th text and an intent type corresponding to the s th text, wherein the s th text includes K words.

[0085] S602, identifying the K words, and inputting the K words with the identification into the intent recognition model.

[0086] Exemplarily, referring to the schematic diagram of the intent recognition model shown in Figure 7 After each word in each text is identified, the id of each word is obtained. For the s th text, K words with the identification are obtained, such as x 1, x 2, …, x K in Figure 7 k , and the K words with the identification are input into the intent recognition model.​​​​​​

[0087] S603, obtaining K second probability vectors output by the intent recognition model according to the parameters of the hidden layer and the speech recognition model.

[0088] Exemplarily, as shown in the figure, Figure 7 the intent recognition model is composed of the parameters f in the speech recognition model and a plurality of hidden layers h, wherein the hidden layers save the context features above and are transmitted to the parameters f of the speech recognition model. After the first word x1 is output into the parameters f in the speech recognition model, the probability of the first word x1 appearing in the s-th text (the first second probability vector y1) and the hidden layer h1 containing the context features of the first word x1 can be obtained in combination with the hidden layer h0. Similarly, the probability of the second word x2 appearing in the text (the second second probability vector y2), the probability of the third word x3 appearing in the text (the third second probability vector y3) and so on can be obtained until the probabilities of K words appearing in the s-th text are calculated, i.e. K second probability vectors are obtained.

[0089] S604, determining a second loss function of the s-th text according to the K second probability vectors and the K words.

[0090] S605, determining the parameters of the intent recognition model according to the second loss function.

[0091] Exemplarily, the calculation method of the second loss function is:

[0092] L k =-q k log(y k )-(1-q k )log(1-y k )

[0093] wherein L k is the k-th second loss function, i.e. the second loss function corresponding to the k-th word in the K words, q k is a vector composed of characters in the k-th word, y k is the k-th second probability vector.

[0094] Exemplarily, after the K second loss functions are calculated, their sum is minimized for the purpose of training, so as to optimize the parameters of the intent recognition model. When the model is trained to the convergent state, the input text can be more accurately predicted to correspond to the intent type.

[0095] The collaborative recitation method provided in the application can assist students to be familiar with the content of the article and the poem through the speech recognition model, the intent recognition model and the language model, and the interactive module of the speech input and output, liberate manpower and improve learning efficiency in the recitation scene of the liberal arts.

[0096] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the method embodiment of the present application.

[0097] Among them, Figure 8 The structure diagram of the collaborative recitation device according to an exemplary embodiment of the present application is shown.

[0098] The collaborative recitation device 800 in the embodiment of the present application comprises a speech recognition model 801, an intention recognition model 802, and a processing module 803, wherein:

[0099] The speech recognition model 801 is configured to input the to-be-recognized speech into the speech recognition model, determine the content corresponding to the to-be-recognized speech according to the output of the speech recognition model, and obtain the to-be-recognized content.

[0100] The intention recognition model 802 is configured to input the to-be-recognized content into the intention recognition model, determine the target intention type corresponding to the to-be-recognized content according to the output of the intention recognition model, wherein the intention type output by the intention recognition model comprises at least one of starting recitation, reciting, requesting help, and completing recitation.

[0101] The processing module 803 is configured to implement collaborative recitation processing of the to-be-recognized content according to the target intention type.

[0102] It should be noted that the collaborative recitation device provided in the above embodiment is only used as an example to divide the above functions, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the collaborative recitation device and the collaborative recitation method embodiment provided in the above embodiment belong to the same concept, so for details not disclosed in the device embodiment of the present application, please refer to the above-mentioned method embodiment of the collaborative recitation method of the present application, which will not be described here.

[0103] The above sequence number of the embodiment of the present application is only for description, and does not represent the advantages and disadvantages of the embodiment.

[0104] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The program is executed by a processor to implement the steps of the method of any of the preceding embodiments. The computer readable storage medium can include, but is not limited to, any type of disk including floppy disks, optical disks, DVD, CD-ROMs, micro-drives, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nano-systems (including molecular memory ICs), or any type of media or device suitable for storing instructions and / or data.

[0105] The embodiments of the present application further provide a terminal, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor. The processor implements the steps of the method of any of the preceding embodiments when executing the program.

[0106] Figure 9 An exemplary structure diagram of a terminal according to an embodiment of the present application is shown schematically. Please refer to FIG. 9. Figure 9 As shown in FIG. 9, the terminal 900 comprises a processor 901 and a memory 902.

[0107] In the embodiments of the present application, the processor 901 is the control center of a computer system, which can be a processor of a physical machine or a processor of a virtual machine. The processor 901 can comprise one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 901 can also comprise a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state.

[0108] In the embodiments of the present application, the processor 901 is specifically configured to perform the following steps:

[0109] The voice to be recognized is input to a voice recognition model, and the content corresponding to the voice to be recognized is determined according to the output of the voice recognition model, to obtain the recognized content; the recognized content is input to an intent recognition model, and the target intent type corresponding to the recognized content is determined according to the output of the intent recognition model, wherein the intent type output by the intent recognition model includes at least one of starting reciting, reciting, requesting help, and completing reciting; and the collaborative reciting processing of the recognized content is implemented according to the target intent type.

[0110] Further, in an embodiment of the present application, the collaborative reciting of the recognized content according to the target intent type includes: in the case that the target intent type is starting reciting, determining the name of the target recited content in the recognized content, and obtaining the target recited content according to the name of the target recited content; and outputting a prompt for starting reciting.

[0111] Optionally, the collaborative reciting of the recognized content according to the target intent type includes: in the case that the intent type is reciting, matching the recognized content with the target recited content to obtain recited content and to-be-recited content.

[0112] Optionally, the collaborative reciting of the recognized content according to the target intent type includes: in the case that the intent type is requesting help, prompting the to-be-recited content in the target recited content according to the recited content.

[0113] Optionally, the collaborative reciting of the recognized content according to the target intent type includes: in the case that the intent type is completing reciting, matching the recited content with the target recited content to obtain a reciting accuracy rate, and outputting an accuracy report according to the reciting accuracy rate.

[0114] Optionally, the method further includes training the voice recognition model, and the training of the voice recognition model includes: obtaining N training samples, wherein the i th training sample is an i th voice, N is a positive integer, and i is a positive integer less than or equal to N; performing feature extraction on the i th voice to obtain M voice feature vectors, wherein M is a positive integer; inputting the M voice feature vectors to the voice recognition model to perform voice recognition processing on each voice feature vector by the voice recognition model, to obtain M character vectors output by the voice recognition model; determining a first loss function corresponding to the i th training sample according to the M character vectors and the M voice feature vectors; and optimizing the parameters of the voice recognition model according to the first loss function.

[0115] Optionally, the determining the first loss function corresponding to the i-th training sample according to the M character vectors and the M speech feature vectors comprises: determining an m-th joint probability according to an m-th character vector and an m-th speech feature vector, where m is a positive integer less than or equal to M; and determining an m-th first loss function according to the m-th joint probability.

[0116] Optionally, the speech recognition model comprises a spectrum enhancement layer, a convolution sampling layer, a linear layer, an anti-overfitting layer, and a convolution enhancement layer; the spectrum enhancement layer is configured to perform random masking processing on the M feature vectors to obtain a first output; the convolution sampling layer is configured to perform convolution processing on the first output to obtain a second output; the linear layer is configured to perform linear transformation processing on the second output to obtain a third output; the anti-overfitting layer is configured to perform random feature dropping on the third output to obtain a fourth output; and the convolution enhancement layer is configured to perform convolution enhancement processing on the fourth output to obtain the M character vectors.

[0117] Optionally, the m-th first loss function is calculated by f(x) = ∑ (X，Y)∈D (-logP(Y|X)), where f(x) is the m-th first loss function, X is the i-th speech, Y is a character in the i-th speech, D is a sample set, (X, Y) is a group of samples in the sample set D, and P(Y|X) is the m-th joint probability.

[0118] Optionally, the method further comprises training the intent recognition model, and the training the intent recognition model comprises: obtaining S training samples, where a s-th training sample comprises a s-th text and an intent type corresponding to the s-th text, the s-th text comprises K words, S is a positive integer, K is a positive integer, and s is a positive integer less than or equal to S; identifying the K words, and inputting the identified K words into the intent recognition model; obtaining K second probability vectors output by the intent recognition model according to a hidden layer and parameters of the speech recognition model; determining a second loss function of the s-th text according to the K second probability vectors and the K words; and determining the parameters of the intent recognition model according to the second loss function.

[0119] Optionally, the determining the second loss function of the s-th text according to the K second probability vectors and the K words comprises: determining a k-th second loss function according to a vector composed of characters in a k-th second probability vector and a k-th word, where k is a positive integer less than or equal to K.

[0120] Optionally, the method for calculating the k-th second loss function is L k = -qk log(y k )-(1-q k )log(1-y k ), where L k is the kth second loss function described above, q k is a vector of characters in the kth word described above, and y k is the kth second probability vector described above.

[0121] The memory 902 can include one or more computer-readable storage media. The computer-readable storage media can be non-transitory. The memory 902 can also include high-speed random access memory and can include nonvolatile memory, such as one or more magnetic disk storage devices, optical storage devices, flash memory devices, or other nonvolatile solid-state storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 902 is configured to store at least one instruction for execution by the processor 901 to implement the method in the embodiments of the present application.

[0122] In some embodiments, the terminal 900 further includes a peripheral terminal interface 903 and at least one peripheral terminal. The processor 901, the memory 902 and the peripheral terminal interface 903 can be connected through a bus or a signal line. Each peripheral terminal can be connected to the peripheral terminal interface 903 through a bus, a signal line or a circuit board. Specifically, the peripheral terminal includes at least one of a display screen 904, a camera 905 and an audio circuit 907.

[0123] The peripheral terminal interface 903 can be used to connect at least one peripheral terminal related to input / output (I / O) to the processor 901 and the memory 902. In some embodiments of the present application, the processor 901, the memory 902 and the peripheral terminal interface 903 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 901, the memory 902 and the peripheral terminal interface 903 can be implemented on a separate chip or circuit board. The embodiments of the present application do not make specific limitations in this regard.

[0124] The display screen 904 is configured to display a user interface (UI). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 904 is a touch display screen, the display screen 904 is further configured to capture touch signals on or above the surface of the display screen 904. The touch signals can be input to the processor 901 as control signals for processing. In this case, the display screen 904 can also be configured to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments of the present application, the display screen 904 can be one, arranged on the front panel of the terminal 900; in some other embodiments of the present application, the display screen 904 can be at least two, arranged on different surfaces of the terminal 900 or in a folding design; in some other embodiments of the present application, the display screen 904 can be a flexible display screen, arranged on a curved surface or a folding surface of the terminal 900. Even, the display screen 904 can also be arranged in an irregular shape, i.e., a special-shaped screen. The display screen 904 can be made of a material such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

[0125] The camera 905 is configured to capture images or videos. Optionally, the camera 905 includes a front camera and a rear camera. Generally, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, the rear camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a long-focus camera, to realize the background blurring function of the main camera and the depth-of-field camera, the panoramic shooting and virtual reality (VR) shooting function of the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments of the present application, the camera 905 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0126] The audio circuit 906 can include a microphone and a speaker. The microphone is configured to capture sound waves of a user and the environment, and convert the sound waves into an electrical signal input to the processor 901 for processing. For the purpose of stereo sound capture or noise reduction, the microphone can be multiple, arranged at different positions of the terminal 900. The microphone can also be an array microphone or an omnidirectional capture microphone.

[0127] The power supply 909 is configured to supply power to each component in the terminal 900. The power supply 907 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 907 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0128] The terminal structure diagram shown in the embodiments of the present application does not constitute a limitation on the terminal 900, and the terminal 900 can include more or fewer components than shown, or combine certain components, or use a different component arrangement.

[0129] In the present application, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or sequence; the term "multiple" refers to two or more, unless otherwise explicitly limited. The terms "mount", "connect", "connect", "fix", and the like should be broadly understood, for example, "connect" can be fixed connection, or detachable connection, or integrally connected; "connected" can be directly connected, or indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0130] In the description of the present application, it should be understood that the orientation or position relationship indicated by the terms "upper", "lower", and the like is based on the orientation or position relationship shown in the drawings, and is only for the purpose of facilitating the description of the present application and simplifying the description, and does not indicate or imply that the device or unit referred to must have a particular direction, be constructed and operated in a particular orientation, therefore, it cannot be understood as a limitation on the present application.

[0131] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, any equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A collaborative memorization method, characterized in that, include: The speech to be recognized is input into the speech recognition model, and the content corresponding to the speech to be recognized is determined based on the output of the speech recognition model to obtain the content to be recognized. The content to be identified is input into the intent recognition model. Based on the output of the intent recognition model, the target intent type corresponding to the content to be identified is determined. The intent type output by the intent recognition model includes at least one of: starting to memorize, memorizing, requesting help, and completing memorization. Based on the target intent type, collaborative memorization processing of the content to be identified is achieved; The method further includes: training the intent recognition model; The training of the intent recognition model includes: Obtain S training samples, wherein the s-th training sample includes the s-th text and the intent type corresponding to the s-th text, the s-th text includes K words, S is a positive integer, K is a positive integer, and s is a positive integer less than or equal to S; The K words are labeled, and the labeled K words are input into the intent recognition model; Based on the parameters of the hidden layer and the speech recognition model, K second probability vectors output by the intent recognition model are obtained; Based on the K second probability vectors and the K words, a second loss function is determined for the s-th text, wherein the second loss function includes the second loss function corresponding to each of the K words in the s-th text; The parameters of the intent recognition model are determined based on the second loss function, and the parameters of the intent recognition model are the parameters corresponding to the minimum sum of the second loss functions.

2. The collaborative memorization method according to claim 1, characterized in that, The step of collaboratively memorizing the content to be identified based on the target intent type includes: When the target intent type is to start memorizing, determine the name of the target memorization content in the content to be identified, and obtain the target memorization content based on the name of the target memorization content; Output a prompt to start memorizing.

3. The collaborative memorization method according to claim 2, characterized in that, The step of collaboratively memorizing the content to be identified based on the target intent type includes: When the intent type is "memorizing", the content to be identified is matched with the target memorization content to obtain the memorized content and the content to be memorized.

4. The collaborative memorization method according to claim 3, characterized in that, The step of collaboratively memorizing the content to be identified based on the target intent type includes: When the intent type is a request for help, prompts are given for the content to be memorized in the target memorization content based on the already memorized content.

5. The collaborative memorization method according to claim 4, characterized in that, The step of collaboratively memorizing the content to be identified based on the target intent type includes: When the intent type is to complete memorization, the memorized content is matched with the target memorization content to obtain the memorization accuracy rate, and an accuracy report is output based on the memorization accuracy rate.

6. The collaborative memorization method according to claim 1, characterized in that, The method further includes: training the speech recognition model; The training of the speech recognition model includes: Obtain N training samples, where the i-th training sample is the i-th speech, N is a positive integer, and i is a positive integer less than or equal to N; Feature extraction is performed on the i-th speech to obtain M speech feature vectors, where M is a positive integer; The M speech feature vectors are input into the speech recognition model, and the speech recognition model performs speech recognition processing on each speech feature vector to obtain the M character vectors output by the speech recognition model. Based on the M character vectors and the M speech feature vectors, determine the first loss function corresponding to the i-th training sample; The parameters of the speech recognition model are optimized based on the first loss function.

7. The collaborative memorization method according to claim 6, characterized in that, The step of determining the first loss function corresponding to the i-th training sample based on the M character vectors and the M speech feature vectors includes: Based on the m-th character vector and the m-th speech feature vector, determine the m-th joint probability, where m is a positive integer less than or equal to M; The m-th first loss function is determined based on the m-th joint probability.

8. The collaborative memorization method according to claim 6 or 7, characterized in that, The speech recognition model includes: a spectral enhancement layer, a convolutional sampling layer, a linear layer, an anti-overfitting layer, and a convolutional enhancement layer; The spectrum enhancement layer is used to perform random masking on the M feature vectors to obtain a first output; The convolutional sampling layer is used to perform convolution processing on the first output to obtain the second output; The linear layer is used to perform a linear transformation on the second output to obtain a third output. The anti-overfitting layer is used to randomly discard features from the third output to obtain a fourth output. The convolutional enhancement layer is used to perform convolutional enhancement processing on the fourth output to obtain the M character vectors.

9. The collaborative memorization method according to claim 7, characterized in that, The method for calculating the m-th first loss function is as follows: in, For the m-th first loss function, For the i-th voice, For the character in the i-th speech, For the sample set, For sample set A set of samples, Let be the m-th joint probability.

10. The collaborative memorization method according to claim 1, characterized in that, The step of determining the second loss function for the s-th text based on the K second probability vectors and the K words includes: The k-th second loss function is determined based on the vector composed of the k-th second probability vector and the characters in the k-th word, where k is a positive integer less than or equal to K.

11. The collaborative memorization method according to claim 10, characterized in that, The method for calculating the k-th second loss function is as follows: in, For the k-th second loss function, Let the vector be composed of the characters in the k-th word. Let be the kth second probability vector.

12. A collaborative memorization device, characterized in that, include: A speech recognition model is used to: input the speech to be recognized into the speech recognition model, determine the content corresponding to the speech to be recognized based on the output of the speech recognition model, and obtain the content to be recognized; An intent recognition model is used to: input the content to be recognized into the intent recognition model, and determine the target intent type corresponding to the content to be recognized based on the output of the intent recognition model, wherein the intent type output by the intent recognition model includes at least one of: starting to memorize, memorizing, requesting help, and completing memorization; The processing module is used to: perform collaborative memorization processing of the content to be identified based on the target intent type; The model training module is used to acquire S training samples, wherein the s-th training sample includes the s-th text and the intent type corresponding to the s-th text, the s-th text includes K words, S is a positive integer, K is a positive integer, and s is a positive integer less than or equal to S; the K words are labeled, and the labeled K words are input into the intent recognition model; based on the parameters of the hidden layer and the speech recognition model, K second probability vectors output by the intent recognition model are obtained; based on the K second probability vectors and the K words, a second loss function for the s-th text is determined, the second loss function includes the second loss function corresponding to each of the K words in the s-th text; the parameters of the intent recognition model are determined based on the second loss function, the parameters of the intent recognition model are the parameters corresponding to the minimum sum of the second loss functions.

13. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the collaborative memorization method as described in any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the collaborative memorization method as described in any one of claims 1 to 11.

Images