Text reading detection method and device and electronic equipment

Abstract

The invention relates to the technical field of computers, and discloses a text reading detection method and device and electronic equipment. The method comprises the following steps: acquiring an audio read by a user and a corresponding reference text when the user reads the audio; obtaining the matching degree of the audio and the reference text according to the audio and the reference text; identifying the audio to obtain an identification text corresponding to the audio; according to the recognition text and the reference text, detecting whether additional content of a non-reference text exists in the audio or not to obtain an additional content detection result; and detecting whether the audio read by the user meets the reading requirement of the reference text or not according to the matching degree of the audio and the reference text and the additional content detection result. According to the method, the accuracy and fairness of spoken language evaluation are improved, and the learning attitude and the actual learning effect of a learner can be better evaluated.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a text reading detection method, apparatus, and electronic device. Background Technology

[0002] Oral language assessment uses computer technology to evaluate learners' spoken pronunciation levels, and this is widely used in learning scenarios. Text reading detection is a key module in oral language assessment systems, designed to detect whether learners have read aloud the given text correctly.

[0003] Existing methods are relatively reliable when learners strictly adhere to the text during assessment. However, when learners do not read the given text, the assessment system may award low to medium scores because it lacks the ability to detect this situation. Furthermore, non-textual reading is common in oral language learning, accounting for nearly half of the low scores; however, there are also instances where high scores are obtained despite not reading the text. Learners cannot understand the root cause of their low scores when they do not read the text, while high scores due to non-textual reading affect the accuracy of the assessment. Therefore, it is essential to add a text-based reading assessment component to oral language assessments. Summary of the Invention

[0004] One objective of this application is to provide a text-to-speech detection method, apparatus, and electronic device to address the technical problem of whether a user is reading text aloud in oral assessment.

[0005] To address the aforementioned technical problems, one technical solution adopted in this application is: providing a text reading detection method, comprising: acquiring audio of a user reading aloud, and a reference text corresponding to the user reading the audio; obtaining a matching degree between the audio and the reference text based on the audio and the reference text; recognizing the audio to obtain the recognized text corresponding to the audio; detecting whether there is any additional content in the audio other than the reference text based on the recognized text and the reference text, to obtain an additional content detection result; and detecting whether the audio read by the user meets the reading requirements of the reference text based on the matching degree between the audio and the reference text and the additional content detection result. This method addresses the technical problem of whether users read aloud according to the text in oral assessment. By acquiring audio of a user reading aloud and the corresponding reference text; calculating the matching degree between the audio and the reference text, this reflects whether the user truly read aloud according to the reference text; detecting whether there is additional content, if so, indicates that the user may not have fully followed the requirements; and finally, comprehensively considering the matching degree between the audio and the reference text, as well as the presence of additional content, to obtain an overall evaluation result, determining whether the learner's reading meets the requirements. This method effectively detects whether users truly read aloud the reference text, preventing situations where users obtain high scores by not reading according to the text. This is crucial for improving the accuracy and fairness of oral assessments. Furthermore, users or guardians can promptly understand the reasons for low scores and obtain relevant statistical data, which can better evaluate learners' learning attitudes and actual learning outcomes, providing valuable references for future improvements.

[0006] Optionally, after acquiring the audio of the user's reading, the method further includes: detecting whether the audio of the user's reading contains human voice; when the audio of the user's reading contains human voice, then performing the step of detecting whether the audio of the user's reading meets the reading requirements of the reference text based on the matching degree between the audio and the reference text, and the additional content detection result. Existing solutions rely on text reading detection methods that use manual feature extraction and scoring by evaluation systems, which are easily affected by background noise or silent data, and lack emphasis on actual human voice detection. The embodiment of this method, combined with a human voice detection scheme, can effectively solve the problem of existing methods being easily affected by background noise, improving the accuracy and robustness of detection.

[0007] Optionally, obtaining the matching degree between the audio and the reference text based on the audio and the reference text includes: inputting the reference text and its corresponding positional encoding information into a decoding module to obtain a text content matrix; inputting the audio into an encoding module to obtain an audio feature matrix; and calculating the correlation between the text content matrix and the audio feature matrix through the decoding module to obtain a reference text weight matrix, wherein the reference text weight matrix is ​​used to represent the matching degree between the audio and the reference text. The embodiments of this method summarize the key processes of the model when processing audio and reference text, including input processing, feature extraction, and interactive computation. This interactive method can help the model better understand the relationship between audio and text, thereby enabling the interpretation and understanding of the audio content, and thus reflecting whether the user has truly read aloud according to the reference text.

[0008] Optionally, the step of recognizing the audio and obtaining the corresponding recognized text includes: inputting the audio into a preset large-scale audio model for recognition to obtain the recognized text, wherein the large-scale audio model includes an audio encoding module and an audio decoding module; the method further includes: acquiring training samples, wherein the training samples include batches of audio data and reference text corresponding to the audio data; preprocessing the audio data to obtain preprocessed audio data; inputting the preprocessed audio data into the large-scale audio model to be trained, extracting the acoustic features of the preprocessed audio data through the audio encoding module, and generating recognized text based on the acoustic features through the audio decoding module; comparing the consistency between the recognized text and the manually annotated text corresponding to the audio data; adjusting the parameters of the large-scale audio model to be trained to maximize the consistency between the generated recognized text and the corresponding manually annotated text; and using the large-scale audio model to be trained that has the highest consistency between the output recognized text and its corresponding manually annotated text as the preset large-scale audio model obtained through training. This method fully utilizes the capabilities of large-scale pre-trained models and, at the same time, performs targeted customized training for children's text reading scenarios, which can effectively improve the accuracy and robustness of text reading detection.

[0009] Optionally, the step of detecting whether there is additional content other than the reference text in the audio based on the identified text and the reference text to obtain an additional content detection result includes: obtaining phoneme sequences corresponding to words in the identified text and the reference text based on the identified text and the reference text; encoding each phoneme in the phoneme sequence of the identified text into a first vector based on the phoneme sequence of the identified text to obtain a first feature matrix corresponding to the phoneme sequence of the identified text, wherein the first vector includes interaction information between the phoneme corresponding to the first vector and other phonemes in the phoneme sequence of the identified text; encoding each phoneme in the phoneme sequence of the reference text into a second vector based on the phoneme sequence of the reference text to obtain a second feature matrix corresponding to the phoneme sequence of the reference text, wherein the second vector includes interaction information between the phoneme corresponding to the second vector and other phonemes in the phoneme sequence of the reference text; and obtaining an identified text weight matrix based on the first feature matrix and the second feature matrix, wherein the identified text weight matrix is ​​used to detect whether there is additional content other than the reference text in the audio to obtain an additional content detection result. This method detects the presence of extraneous non-reference text in audio by converting text into phoneme sequences, encoding them into feature matrices, and then calculating recognition text weight matrices based on these feature matrices. Compared to directly using text, this phoneme-based feature approach generalizes better to different speakers and pronunciation habits, improving the model's robustness and adaptability in practical applications. Furthermore, phoneme-based features can be extended to support reading detection for multiple languages.

[0010] Optionally, the step of detecting whether the audio read by the user meets the reading requirements of the reference text based on the matching degree between the audio and the reference text, and the additional content detection result, includes: inputting the matching degree between the audio and the reference text, and the additional content detection result, respectively, into a parallel convolutional neural network to obtain a first convolutional feature corresponding to the matching degree, and a second convolutional feature corresponding to the additional content detection result; processing the first convolutional feature through a first pooling layer to obtain a first pooling feature, and processing the second convolutional feature through a second pooling layer to obtain a second pooling feature; inputting the first pooling feature and the second pooling feature into a fully connected layer to output a prediction result of whether the audio read by the user meets the reading requirements of the reference text. Here, the matching degree and additional content of the audio are processed by the convolutional neural network, and finally, the fully connected layer is used to predict whether the audio read by the user meets the reading requirements of the reference text. This method fully integrates audio matching score and additional content detection information, which helps to comprehensively consider audio quality and content integrity to make more accurate judgments on reading accuracy. The entire model directly outputs the prediction result of reading accuracy from the input audio matching score and additional content information, avoiding manual intervention in the intermediate steps and realizing end-to-end automated learning. It has good transferability and can be applied to different speech reading detection tasks. It can quickly adapt to different application scenarios by only fine-tuning the model parameters for new datasets.

[0011] To address the aforementioned technical problems, one technical solution adopted in this application is as follows: a text reading detection device is provided, comprising: an audio and text acquisition module for acquiring audio read by a user and a reference text corresponding to the user reading the audio; a matching degree acquisition module for obtaining the matching degree between the audio and the reference text based on the audio and the reference text; a recognition text acquisition module for recognizing the audio and obtaining the recognition text corresponding to the audio; an additional content detection result module for detecting whether there is any additional content other than the reference text in the audio based on the recognition text and the reference text, thereby obtaining an additional content detection result; and a text reading detection module for detecting whether the audio read by the user meets the reading requirements of the reference text based on the matching degree between the audio and the reference text and the additional content detection result.

[0012] To solve the above-mentioned technical problems, one technical solution adopted in this application is: to provide an electronic device, including: a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory, wherein when the processor executes the one or more computer programs, the electronic device implements a text reading detection method applied to the electronic device.

[0013] To solve the above-mentioned technical problems, one technical solution adopted in this application is to provide a non-volatile computer-readable storage medium that stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device performs the text reading detection method as described above.

[0014] To solve the above-mentioned technical problems, one technical solution adopted in this application is: to provide a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by an electronic device, causing the electronic device to perform the text reading detection method as described above.

[0015] The aforementioned text reading detection device, electronic device, non-volatile computer-readable storage medium, and computer program product all possess the beneficial effects corresponding to the aforementioned text reading detection method applied to electronic devices. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a text reading detection method provided in an embodiment of this application;

[0018] Figure 2 This is a schematic diagram of the method flow for obtaining the matching degree between audio and reference text provided in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram of the large audio model corresponding to the audio model provided in the embodiments of this application for recognizing the audio read by the user and obtaining the recognized text;

[0020] Figure 4 This is a schematic diagram corresponding to the method flow for obtaining the recognition text weight matrix provided in the embodiments of this application;

[0021] Figure 5 This is a schematic diagram of the method flow for detecting whether the audio of a user's reading meets the reading requirements of a reference text, as provided in the embodiments of this application.

[0022] Figure 6 This is a flowchart of a text reading detection method provided in another embodiment of this application;

[0023] Figure 7 This is a schematic diagram of the structure of a text reading detection device provided in an embodiment of this application;

[0024] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0026] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0027] It should be noted that in the various embodiments described below, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in interchange, etc.

[0028] Oral assessment utilizes computer technology to evaluate learners' spoken pronunciation levels, and it is widely used in learning settings. Automated computer-based oral assessment avoids the need for professional teachers, improving efficiency and scalability. Computers can quickly assess each learner's pronunciation and provide immediate feedback, making it timely and effective for learners. Therefore, oral assessment is widely used in language learning and education, providing learners with continuous diagnostic and feedback services for their spoken pronunciation.

[0029] Text-to-speech detection is a crucial module in oral assessment systems. It aims to detect whether learners have read the given text correctly. This is vital because only by ensuring learners read the text correctly can the assessment system accurately evaluate their pronunciation level. If learners do not read the text correctly, the assessment results may be distorted. Therefore, text-to-speech detection is an indispensable part of oral assessment systems; it relates to the accuracy and reliability of the entire system and is the foundation for achieving automated, timely feedback in oral assessment.

[0030] Current oral language assessment systems typically rely on speech recognition and natural language processing technologies, combined with machine learning and deep learning algorithms, to evaluate a speaker's pronunciation, intonation, fluency, and other aspects. While assessments are relatively reliable when learners strictly adhere to the text, they can become unreliable in certain scenarios, even leading to user complaints and skepticism. For example, when learners deviate from the given text, the system may fail to detect this accurately, resulting in inaccurate scores, such as low to mid-range scores. This could stem from insufficient generalization ability of the acoustic model or unreliable timestamp information provided by a particular module. Furthermore, when relying on traditional manual feature engineering and scoring to detect non-textuality, the system might misidentify background noise or silence as non-textuality, indicating a lack of accurate detection of voice-over cues and insufficient generalization. Finally, non-textuality is common in early childhood oral language learning, potentially accounting for nearly half of low scores, highlighting the need for oral language assessment systems to better adapt to the characteristics and behaviors of learners at different ages. For example, sometimes deviating from the text may result in a high score for the learner. This may indicate that the assessment system is unable to accurately capture the problem of deviating from the text in certain situations, or that the scoring criteria are flawed, thus compromising accuracy and fairness.

[0031] To address the aforementioned issues, this application provides a text-to-speech detection method and apparatus. The method involves acquiring audio of a user's reading aloud and the corresponding reference text; calculating a matching degree between the audio and the reference text, reflecting whether the user truly read according to the reference text; detecting the presence of extra content, indicating that the user may not have fully followed the requirements; and finally, comprehensively considering the matching degree between the audio and the reference text, as well as the presence of extra content, to arrive at an overall evaluation result, determining whether the learner's reading meets the requirements. This method effectively detects whether learners truly read according to the reference text, preventing learners from obtaining high scores by not reading according to the text, which is crucial for improving the accuracy and fairness of oral assessment. A neural network-based end-to-end detection method for non-text reading is proposed, eliminating reliance on traditional manual features and scoring and alignment time results from the assessment system, thus improving the accuracy and generalization of non-text reading detection. A scheme combining human voice detection is also provided, effectively addressing the problem of existing methods being easily affected by background noise or silent data, improving the accuracy and robustness of detection. Furthermore, users or guardians can promptly understand the reasons for low scores and obtain relevant statistical data. This allows for a better assessment of learners' learning attitudes and actual learning outcomes, providing valuable references for future improvements. It also reduces instances where high scores are obtained despite not reading the text aloud, thus improving the accuracy and reliability of the oral assessment system.

[0032] The text-to-speech detection method of this application can be applied to oral assessment scenarios. Oral assessment scenarios include, for example, hardware devices such as microphones, speakers, headphones, recording devices, computers, and network connection devices. Microphones are used to record the user's voice input, responsible for capturing the user's spoken voice signal. Speakers are used to play reference text or oral practice content, helping the user hear the reference text and thus practice reading aloud. In some cases, users can also use headphones for oral practice to reduce external interference and better hear the reference text or system feedback. Recording devices are used to record the user's voice input; these can be recording devices integrated into the assessment system or external recording devices such as professional microphones or voice recorders. The computer is used to run the oral language assessment system, performing tasks such as speech signal processing, feature extraction, speech recognition, and evaluation. In this embodiment, the computer can be used to execute a text reading detection method. For example, the computer can acquire the audio of a user reading aloud, and the corresponding reference text; obtain the matching degree between the audio and the reference text based on the audio and the reference text; recognize the audio and obtain the recognized text corresponding to the audio; detect whether there is any additional content in the audio that is not the reference text, based on the recognized text and the reference text, to obtain additional content detection results; and detect whether the audio read by the user meets the reading requirements of the reference text based on the matching degree between the audio and the reference text and the additional content detection results. In some cases, the oral language assessment system can also run on mobile devices, such as smartphones, tablets, or learning machines, allowing users to conduct oral language assessments. The oral language assessment scenario can also include network-connected devices. If the oral language assessment system is based on cloud services, users can connect their devices (such as computers, smartphones, and other mobile devices) to the Internet to upload speech data for evaluation.

[0033] The text-to-speech detection method of this application can detect non-text-based readings in multilingual scenarios. Besides spoken language assessment, it can be applied to other scenarios, such as hot word wake-up and the evaluation of audio content and topic relevance in open-ended spoken language scenarios. Specifically, a speech recognition model for hot words is established to identify specific hot words, match them with the speech input to determine the presence of a wake-up word, and trigger corresponding operations or responses. Natural language processing technology is used to extract audio content or topic topics in open-ended spoken language scenarios, and a relevance evaluation model is established. The relevance between the speech content and topic topics is evaluated by comparing the relevance between the speech content and topic keywords. This solution integrates speech recognition, text similarity calculation, and a relevance evaluation model, enabling the detection of non-text-based readings in multilingual scenarios and allowing for reuse in other fields such as hot word wake-up and topic relevance evaluation.

[0034] Please see Figure 1, Figure 1 This is a flowchart of a text-to-speech detection method provided in an embodiment of this application. The method includes:

[0035] Step S11: Obtain the audio of the user reading aloud, and the corresponding reference text when the user reads the audio.

[0036] Users can record their reading aloud using a microphone, while the system simultaneously obtains the reference text the user is reading. For example, when a user is taking an oral assessment, the audio refers to the audio the user is currently reading aloud for the assessment, and the reference text refers to the text provided by the system for the oral assessment—the text the user is looking at while reading. By obtaining this audio and reference text, the necessary data foundation is provided for subsequent comparison, recognition, and evaluation.

[0037] The system can be designed to support multiple languages, enabling it to receive and process readings from users with different language backgrounds. It can recognize audio recordings of users reading from different languages, thereby detecting whether the audio conforms to the reading requirements of the reference text.

[0038] Step S12: Based on the audio and the reference text, obtain the matching degree between the audio and the reference text.

[0039] By comparing the content of a user's audio reading with a reference text and calculating the degree of matching, it is possible to assess whether the user's reading meets the expected requirements of the reference text. For example, these expected requirements include that the content read by the user should be identical to the content of the reference text, including the accuracy of words, phrases, and sentences; the user's reading should also conform to the rhythm and intonation of the language, neither too fast nor too slow; and, based on the logical structure of the reference text, the user should be able to pause at appropriate points to make the sentences clearer; etc.

[0040] This matching degree refers to the textual similarity between the audio content and the reference text. For example, it compares words, phrases, and sentences to determine the degree of consistency between them.

[0041] When determining the matching degree between audio and reference text, various methods can be used for comparison and evaluation. For example, text similarity comparison can be employed, using text comparison algorithms (such as edit distance algorithm, cosine similarity, etc.) to compare the similarity between the identified text in the audio and the reference text. Alternatively, a trained classifier model can be used to determine the matching degree between audio and reference text; for instance, a neural network model can be used to evaluate the matching degree between audio and text, such as a recurrent neural network (RNN) or a convolutional neural network (CNN).

[0042] In some embodiments, the matching degree is mainly used to evaluate whether the audio reads out words from the reference text. Obtaining the matching degree between the audio and the reference text includes: inputting the reference text and its corresponding positional encoding information into a decoding module to obtain a text content matrix; inputting the audio into an encoding module to obtain an audio feature matrix; and calculating the correlation between the text content matrix and the audio feature matrix through the decoding module to obtain a reference text weight matrix, which represents the matching degree between the audio and the reference text. This embodiment summarizes the key processes of the model when processing audio and reference text, including input processing, feature extraction, and interactive computation. This interactive method helps the model better understand the relationship between audio and text, thereby enabling the interpretation and understanding of the audio content and reflecting whether the user truly read out the reference text.

[0043] Specifically, please refer to Figure 2 , Figure 2 This is a schematic diagram corresponding to the method flow for obtaining the matching degree between audio and reference text provided in the embodiments of this application. For example... Figure 2 As shown, the audio input encoding module (Encoder Module) inputs the reference text and its corresponding learned positional encoding information (Learned Positional Encoding) into the decoding module (composed of multiple Decoder Blocks). The reference text serves as the text content input to the decoding module, and also incorporates learnable positional encoding information to help the model better understand the positional relationships within the text content. This positional encoding information helps the model distinguish words at different positions within the reference text. In the decoding module, a cross-attention mechanism is used to calculate the relationship between the text content matrix and the audio hidden feature matrix output by the encoding module. This cross-attention mechanism helps the model focus on the degree of correlation between the audio and the reference text, thereby calculating the reference text weight matrix, reflecting the relative importance of the audio and reference text in terms of matching degree. Through this process, the model can utilize the cross-attention mechanism to calculate the weight matrix relative to the audio and reference text, which can be used to represent the matching degree between the audio and the reference text. This method helps the model better understand the relationship between the audio and the reference text, thereby enabling the interpretation and understanding of the audio content to determine whether the user has read the reference text correctly.

[0044] The model can be a Transformer model (a deep learning model based on an attention mechanism, which includes two main components: an encoding module and a decoding module). In this embodiment, the relationship between audio and reference text is processed and their matching degree is calculated using a Transformer architecture model that includes an encoding module and a decoding module.

[0045] Figure 2 The corresponding model, on the one hand, is used to detect whether the audio reads out words from the reference text. When the reference text and audio do not match, they can be distinguished by the cross-attention mechanism matrix of the reference text and audio features (i.e., the reference text weight matrix). On the other hand, the model is also used to detect whether the audio contains human voices, in order to eliminate false detections of pure noise and silence. Therefore, after obtaining the audio read by the user, the method further includes: detecting whether the audio read by the user contains human voices. If the audio read by the user contains human voices, then step S15 is executed; if the audio read by the user does not contain human voices, then the text reading detection method of this embodiment is not required. Adding a human voice detection scheme to the text reading detection method can effectively solve the problem that existing methods are easily affected by background noise, and improve the accuracy and robustness of detection.

[0046] The process of detecting whether a user's audio reading contains human voice includes the following steps: First, the Mel-spectral characteristics (a method of audio feature representation) of the user's audio reading are input into an encoder module. This encoder module can contain various layers, such as convolutional layers, recurrent neural network (RNN) layers, or Transformer layers, to extract hidden acoustic features. These hidden acoustic features are feature representations containing important acoustic information extracted from the Mel-spectral characteristics of the user's audio reading by the encoder module. The output is a hidden acoustic feature matrix, which captures important acoustic information from the audio data, such as speech features, audio spectrum features, speech content, rhythm, and prosody. Then, the hidden acoustic feature matrix is ​​input into an average pooling layer. The average pooling operation helps eliminate temporal dimension information, converting time-series data into a fixed-length feature vector. This step transforms the hidden acoustic feature matrix into a fixed-dimensional audio hidden feature vector for subsequent processing and classification. Next, the audio hidden layer feature vector is input into a fully connected layer for binary classification. The obtained audio hidden layer feature vector is fed into the fully connected layer, which contains multiple neurons. After processing by activation functions and other methods, the model can perform binary classification to determine the presence or absence of human voices. In this embodiment, the model can extract acoustic features from the audio Mel-spectral features, then convert them into a fixed-length feature vector through an average pooling layer, and finally perform binary classification through a fully connected layer to determine whether human voices are present in the audio. This model can be a deep learning-based neural network model used for human voice detection tasks in audio processing.

[0047] In some embodiments, during execution via Figure 2Before the corresponding model can detect whether words in the reference text are read aloud in the audio, and whether the audio contains human voices, it can be trained. The training dataset includes two types of audio data: The first type is audio data containing human voices. This type of data is used to train the model to identify whether audio contains human voices. This data can be obtained from Automatic Speech Recognition (ASR) datasets, which typically contain various audio segments containing human voices, such as speaking or reading aloud. The second type is non-human voice audio data. This type of data is used to train the model to identify non-human voice audio. This type of data can be obtained from two sources: The first is by using an event detection model (such as CLAP (Contrastive Learning of Audios and Texts by Pretraining)) to filter non-human voice audio from the evaluation data. This method involves using an event detection model to identify non-human voice audio, such as ambient sounds and noise. The second approach is to utilize non-human audio data from publicly available audio understanding and event detection datasets, such as FSD50k, ClothoV2, and AudioCaps. These datasets contain various types of audio data that can be used to train models to recognize non-human audio.

[0048] The steps for training this model may include: organizing the acquired audio data containing human voices and non-human voice audio data into a training set; extracting Mel-spectral features from the audio data and converting it into an input format acceptable to the model; dividing the dataset into training and validation sets for training and evaluating model performance; designing a deep learning model suitable for this task, including encoding modules, decoding modules, average pooling layers, and fully connected layers, for human voice detection and word content detection; training the model using the training set data, continuously adjusting the model parameters through backpropagation to enable it to accurately predict human voices and word content in the audio data; evaluating the model's performance using the validation set and adjusting the model parameters to improve performance; and finally, deploying the trained model in a real-world application to detect whether words from a reference text are read aloud in the audio and whether human voices are present.

[0049] Step S13: Recognize the audio and obtain the corresponding recognized text.

[0050] This can be achieved by using speech recognition technology to convert the user's audio into text representation. For example, a speech recognition engine can be used to process the audio and generate corresponding text output. This recognized text can then be used for subsequent analysis and comparison, providing a usable text format for the system's later processing.

[0051] In some embodiments, recognizing audio and obtaining corresponding recognized text includes: inputting the audio into a preset audio model for recognition to obtain recognized text, wherein the audio model includes an audio encoding module and an audio decoding module.

[0052] Specifically, please refer to Figure 3 , Figure 3 This is a schematic diagram corresponding to the large audio model provided in this application embodiment for recognizing audio read aloud by a user and obtaining the recognized text. For example... Figure 3 As shown, the audio input for user reading is "2*Conv1D+GELU". This "2*Conv1D+GELU" represents the use of two consecutive one-dimensional convolutional layers (Conv1D) in a neural network, followed by a non-linear transformation of their outputs through the GELU (Gaussian Error Linear Units) activation function. Sinusoidal Positional Encoding is a positional encoding method that uses sine and cosine functions to encode information at each position in a sequence. Figure 3 The part within the dashed box on the left is the audio encoding module, which is used to encode the deep acoustic features of the audio. The part on the right is the decoding module for recognizing text, and finally outputs the recognized text corresponding to the audio read aloud by the user. In this embodiment, the method further includes: obtaining the preset audio model. Obtaining the preset audio model includes: obtaining training samples, which include batches of audio data and corresponding reference texts. The batches of audio data can be collected in a children's text reading scenario; preprocessing the audio data to obtain preprocessed audio data; inputting the preprocessed audio data into the audio model to be trained, extracting the acoustic features of the preprocessed audio data through the audio encoding module, and generating recognized text based on the acoustic features through the audio decoding module; comparing the consistency between the recognized text and the manually annotated text corresponding to the audio data; adjusting the parameters of the audio model to be trained to maximize the consistency between the generated recognized text and the corresponding manually annotated text; and using the audio model to be trained that has the highest consistency between the output recognized text and its corresponding manually annotated text as the preset audio model obtained through training. In this model, the manually labeled text corresponding to the audio data can be tags selected from the reference text. These tags help the model learn the correspondence between the recognized text and the standard text. The purpose of this training process is to optimize the model parameters using training data, so that the generated recognized text is as consistent as possible with the manually labeled text. This model can be used in audio-text recognition tasks, such as those involving children reading aloud.

[0053] The audio data includes manually labeled data and weakly labeled data. The batch audio data and corresponding reference texts for children's text reading scenarios are obtained in two ways: one is manually labeled data, which includes a small portion of audio data from children's text reading scenarios and corresponding accurate reference texts, used as part of the training set. The other is weakly labeled data, obtained through model selection, which includes a large amount of audio data from children's text reading scenarios. The acquisition of weakly labeled data mainly involves the following steps: the original reading data contains audio and reference text, but the two may not match, so a trained recognition model is used to obtain the recognition text; the original audio and reference text, and the original audio and recognition text are respectively sent to a scoring system to obtain pronunciation accuracy scores; audio data where the accuracy scores of words in both the reference text and the recognition text are greater than a preset score (e.g., a preset score of 85 out of 100) are selected, while also satisfying the condition that the phoneme-level edit distance between the reference text and the recognition text is small, meaning that after audio recognition, the difference between the obtained recognition text and the reference text is small.

[0054] The preprocessing of the audio data includes data augmentation of the original audio data by adjusting speech rate, volume, and noise levels to increase the diversity of the training set.

[0055] This method fully utilizes the capabilities of large-scale pre-trained models and conducts targeted customized training for children's text reading scenarios, which can effectively improve the accuracy and robustness of text reading detection.

[0056] Step S14: Based on the identified text and the reference text, detect whether there is any additional content in the audio that is not the reference text, so as to obtain the additional content detection result.

[0057] For example, if the reference text is "I like apple" and the audio content is "The weather is nice today, where to go, apple", although the audio reads the word "apple" from the reference text, it also contains some content unrelated to the reference text. That is, there is extra content that is not related to the reference text. Therefore, this audio is also data that is not read from the text.

[0058] The additional content detection result refers to the detection result of extra content that is not referenced text in the audio.

[0059] Specifically, please refer to Figure 4Based on the identified text and the reference text, the phoneme sequences corresponding to the words in each text are obtained. Based on the phoneme sequence of the identified text, each phoneme in the phoneme sequence is encoded into a first vector, resulting in a first feature matrix corresponding to the phoneme sequence of the identified text. The first vector represents the feature vector of each phoneme in the phoneme sequence of the identified text and includes interaction information between the phoneme corresponding to the first vector and other phonemes in the phoneme sequence of the identified text. Based on the phoneme sequence of the reference text, each phoneme in the phoneme sequence of the reference text is encoded into a second vector, resulting in a second feature matrix corresponding to the phoneme sequence of the reference text. The second vector represents the feature vector of each phoneme in the phoneme sequence of the reference text and includes interaction information between the phoneme corresponding to the second vector and other phonemes in the phoneme sequence of the reference text. Based on the first and second feature matrices, an identified text weight matrix is ​​obtained. The identified text weight matrix is ​​used to detect whether there is additional content in the audio that is not part of the reference text, thus obtaining additional content detection results.

[0060] The phoneme sequence in the identified text is processed by treating each phoneme as a basic unit and converting it into a mathematical vector. This generated mathematical vector is the first vector corresponding to that phoneme. Each phoneme is represented as a feature vector, which is also a first vector. Therefore, the entire identified text phoneme sequence will consist of multiple such first vectors. Similarly, the phoneme sequence in the reference text is also processed, and the entire reference text phoneme sequence will consist of multiple second vectors as described above.

[0061] The first vector includes the interaction information between the phoneme corresponding to the first vector and other phonemes in the phoneme sequence of the recognized text. When generating the first vector for each phoneme, in addition to including the feature information of the phoneme itself, the relationship between the phoneme and its surrounding phonemes can also be considered. Therefore, the interaction information between the phoneme corresponding to the first vector and other phonemes in the phoneme sequence of the recognized text refers to the fact that the first vector contains the relationship information between the phoneme and its surrounding phonemes. This helps to capture the speech feature performance of phonemes in context, further improving the description and understanding of the phoneme sequence. Similarly, the second vector includes the interaction information between the phoneme corresponding to the second vector and other phonemes in the phoneme sequence of the reference text. This interaction information refers to the fact that the second vector contains the relationship information between the phoneme and its surrounding phonemes.

[0062] Among them, such as Figure 4As shown, the reference text and the recognized text are processed by a glyph-to-phoneme module (G2P Module). The G2P Module can be a grapheme-to-phoneme model used to convert letters into phonemes. The reference text and the recognized text are processed by the G2P Module to obtain the phoneme sequences corresponding to the words in the respective texts. The phoneme sequences processed by the G2P Module are then passed through a Learned Positional Encoding module. This module learns a positional code for each position in the input sequence, enabling the model to better understand information from different positions within the sequence. Next, the two phoneme sequences after Learned Positional Encoding each pass through a Self-Attention layer. Self-Attention is an attention mechanism that helps the model focus on the relationships between different positions in the sequence when processing sequence data. The representations processed by Self-Attention are then passed through a Cross-Attention layer. Cross-Attention is an attention mechanism used to establish a relationship between the two sequences, i.e., between the reference text and the recognized text, so that the model can better compare the similarities and differences between them. Finally, after processing with Self Attention and Cross Attention, the model outputs a weight matrix for the recognized text. This weight matrix can contain the importance weight of each word or phoneme in the final recognized text. The recognized text weight matrix is ​​used in this process to detect whether there is any extra content in the audio that is not part of the reference text, thus obtaining additional content detection results. This recognized text weight matrix helps the model identify and quantify extra content in the audio that does not match the reference text, as this extra content may appear as parts with lower weights or unexpected content in the final recognized text. By analyzing the recognized text weight matrix, the model can weight the importance of each part in the audio and detect whether any extra content is incorrectly included in the recognized text. If some parts show lower weights in the recognized text weight matrix, but these parts do not match the reference text, then there is likely extra content from the non-reference text. In this embodiment, the recognized text weight matrix can be used for additional content detection to improve the accuracy and reliability of the audio recognition system. Furthermore, the phoneme-level feature-based method, compared to directly using text, can generalize better to different speakers and pronunciation habits, improving the robustness and adaptability of the model in practical applications. Furthermore, based on phoneme-level features, it can be extended to support reading detection for multiple languages.

[0063] Step S15: Based on the matching degree between the audio and the reference text, and the results of the additional content detection, detect whether the audio read by the user meets the reading requirements of the reference text.

[0064] In this embodiment, the detection of whether the user's audio reading meets the reading requirements of the reference text combines the matching degree between the audio and the reference text with the results of additional content detection. This application embodiment designs a non-text-based reading detection model. This non-text-based reading detection model can comprehensively consider the matching degree between the audio and the reference text, as well as the results of additional content detection. It uses convolutional neural networks and fully connected layers for feature extraction and prediction to determine whether the user's audio reading meets the specified reading requirements.

[0065] Specifically, please refer to Figure 5 The matching degree between the user's audio reading and the reference text, as well as the additional content detection results obtained above, are input into a parallel convolutional neural network. This parallel convolutional neural network can be two networks with the same network structure, obtaining a first convolutional feature corresponding to the matching degree and a second convolutional feature corresponding to the additional content detection results. The first convolutional feature is processed by a first pooling layer to obtain a first pooling feature, and the second convolutional feature is processed by a second pooling layer to obtain a second pooling feature. The first and second pooling features are then input into a fully connected layer to output a prediction result indicating whether the user's audio reading meets the reading requirements of the reference text. Here, "3*Conv1D+GELU" means that three one-dimensional convolutional layers (Conv1D) are used consecutively in the neural network, and their outputs are then nonlinearly transformed through the GELU (Gaussian Error Linear Units) activation function. The process involves processing the first convolutional features through a first pooling layer and the second convolutional features through a second pooling layer. This process involves pooling the feature maps generated by two different convolutional kernels. Pooling operations use max pooling or average pooling to reduce the size of the feature maps, thereby reducing computation and the number of parameters while retaining important feature information. The first and second pooling layers refer to the specific layers applied after the convolutional layers. Both the first and second pooling layers perform pooling operations on the feature maps output by the convolutional layers, reducing the spatial size of the feature maps and extracting the most salient features. The first and second pooling features refer to the features obtained after processing by the corresponding pooling layers. These features contain the most salient feature information extracted in the convolutional layers, which helps subsequent fully connected layers perform classification or prediction tasks. The fully connected layer integrates and processes the features extracted by the preceding convolutional and pooling layers to output the final classification or regression result. The fully connected layer is a basic layer type in deep learning neural networks.

[0066] The prediction result can be a binary classification problem, meaning the prediction result includes one of two cases: it meets the reading requirements of the reference text, or it does not. Meeting the reading requirements of the reference text can include a high degree of matching between the words in the reference text and the words in the user's audio reading, meaning the user read the reference text correctly. It can also include the user's reading speed and rhythm matching the required reading speed and rhythm of the reference text. Furthermore, it can include the user's accurate pronunciation, correctly pronouncing the words and sentences in the reference text. Finally, it can include the user's reading conveying the correct emotion and intonation, conforming to the context and emotion required by the reference text. Not meeting the reading requirements of the reference text can be due to, for example, the user's audio reading containing incorrect or missing words, inconsistent with the reference text; the user's reading speed and rhythm not matching the required reading speed and rhythm, possibly too fast or too slow; inaccurate pronunciation, inconsistent with the pronunciation in the reference text; or the user's reading lacking the necessary emotion and intonation, inconsistent with the required emotion and intonation of the reference text, etc.

[0067] This method employs a convolutional neural network to process audio matching and additional content, ultimately using a fully connected layer to predict whether the user's audio reading meets the reading requirements of the reference text. This approach fully integrates audio matching and additional content detection information, helping to comprehensively consider audio quality and content completeness to make more accurate judgments on reading compliance. The entire model directly outputs the predicted reading compliance result from the input audio matching and additional content information, avoiding manual intervention in intermediate steps and achieving end-to-end automated learning, thus improving the evaluation and detection of reading quality. Furthermore, it exhibits good transferability, applicable to different speech reading detection tasks; simply fine-tuning the model parameters for new datasets allows for rapid adaptation to different application scenarios.

[0068] The non-text-reading detection model in this application embodiment can specifically be a deep learning model combining convolutional neural networks and fully connected layers. In some embodiments, the model can be pre-trained. During training, a training set is first acquired, which can be constructed into two types of data. One type is non-text-reading data selected based on a scoring model and a recognition model. The selection criteria for this type of data include: low scores for words in the reference text and high scores for words in the recognized text; and a low proportion of words in the reference text appearing in the recognized text. Samples are selected from such audio and text data as the training set; these samples are considered to be non-text-reading data. The other type of data is text simulation data, which can be selected from audio recordings with high consistency between the reference text and the recognized text, and high scores for both the reference text and the audio, to obtain data matching the reference text and audio. Through the training sets constructed in the above two ways, the model will be able to learn to distinguish different types of reading situations, thereby improving the accuracy of the assessment of the user's reading quality. After determining the training set, the model can be trained to obtain the non-text-reading detection model.

[0069] The text-to-speech detection method provided in this application can not only effectively detect whether a user has actually read the reference text, but also reduce the occurrence of high scores despite not reading the text correctly. Furthermore, knowing whether the user's audio reading meets the requirements of the reference text can help the user or guardian understand the reasons for a low score.

[0070] Please see Figure 6 , Figure 6This is a flowchart of a text reading detection method provided in another embodiment of this application. It mainly includes: a reference text content reading detection module, an audio large model module, a text recognition and reference text relevance detection module, and a non-text reading detection module. The reference text content reading detection module receives audio read aloud by the user and reference text as input. By comparing the speech in the audio with the content of the reference text, it calculates the weight matrix of the reference text in the entire audio. The reference text content reading detection module achieves the detection of reference text content and human voice in the audio by constructing a human voice detection training set and training a model. This module can be used to determine whether the audio is read according to the reference text requirements and eliminate false detections of pure noise and silence. The audio large model module receives audio as input, processes the audio using the audio large model, and outputs the corresponding recognized text. The audio large model is trained using the constructed audio training set data. The model includes an audio encoding module and a recognized text decoding module. The audio encoding module is used to perform deep acoustic feature encoding on the audio to extract important feature information. The text decoding module decodes encoded audio features into recognized text. It constructs a training set of data for children's scenes and trains a large audio model to achieve the audio recognition task. The text-to-reference-text relevance detection module receives phoneme sequences of the recognized and reference texts as input. By comparing the relevance between the two, it calculates the weight matrix of the recognized text. The purpose of this module is to determine the similarity between the recognized and reference texts. This module extracts the weight matrix of the recognized text to determine whether the audio contains content unrelated to the reference text. The non-text-based reading detection module receives the weight matrix of the recognized text, the weight matrix of the reference text, and the voice detection results as input. By comprehensively considering this information, it determines whether the audio is non-text-based reading. The output of this module is whether the audio meets the reading requirements of the given text. The training set for the non-text-based reading detection module is divided into two categories: non-text-based reading data in reading scenarios filtered according to the scoring and recognition models; and text simulation data. This module uses a reference text weight matrix and a recognition text weight matrix as input, extracts features using a parallel convolutional neural network, and performs prediction through a fully connected layer to determine whether the current audio is non-textualized reading data.

[0071] This embodiment's method utilizes a neural network in an end-to-end manner to detect non-text-based reading, eliminating reliance on traditional manual features, scoring systems, and alignment time results. This improves the accuracy and generalization of non-text-based reading detection. It not only detects non-text-based reading but also identifies the root causes of low scores, providing valuable feedback for improving reading quality. Furthermore, it offers data statistics and learns user attitudes, reducing false detections of non-human voice data and enhancing user experience. It can detect non-text-based reading in multilingual scenarios and can be reused in the fields of hot word activation, relevance evaluation of audio content and topic themes in open-ended spoken language scenarios.

[0072] In some embodiments, after detecting whether the audio read by the user meets the reading requirements of the reference text, the method further includes: generating feedback information based on the result of the detection of whether the audio meets the reading requirements of the reference text, the feedback information being used to indicate the locations where the reading does not meet the requirements and the existing pronunciation problems; analyzing the user's pronunciation quality, pointing out problematic phonemes and tones, and providing suggestions to the user based on the analysis results, such as recommending pronunciation practice methods and correcting inappropriate reading habits, to guide the user on how to make targeted improvements. The method may also include: establishing an interactive feedback mechanism that allows users or guardians to ask questions and provide professional answers, or tracking and analyzing previous problems in subsequent evaluations.

[0073] The feedback information can be presented in the form of charts or graphs to aid in understanding and analysis.

[0074] After users have used the oral assessment system for a period of time, the system can also analyze their historical data, provide trend analysis reports, summarize progress and point out areas for improvement, and provide optimization suggestions for users or guardians.

[0075] The system can also record learners' learning attitude data, such as practice frequency, duration, and initiative, and integrate and present this data to help users or guardians understand their learning status.

[0076] Through the above steps, the system can provide more comprehensive feedback and analysis, helping users or guardians to fully understand the user's reading performance and providing valuable references for improving and enhancing learning outcomes.

[0077] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a text reading detection device provided in an embodiment of this application. The text reading detection device 20 includes:

[0078] The audio and text acquisition module 21 is used to acquire the audio read by the user and the reference text corresponding to the user reading the audio.

[0079] The matching degree acquisition module 22 is used to obtain the matching degree between the audio and the reference text based on the audio and the reference text;

[0080] The text recognition acquisition module 23 is used to recognize the audio and obtain the recognized text corresponding to the audio.

[0081] The additional content detection result module 24 is used to detect whether there is any additional content in the audio that is not the reference text, based on the identified text and the reference text, so as to obtain the additional content detection result;

[0082] The text reading detection module 25 is used to detect whether the audio read by the user meets the reading requirements of the reference text based on the matching degree between the audio and the reference text, as well as the detection result of the additional content.

[0083] The text reading detection device 20 described above can be a software module. The software module includes several instructions, which are stored in a memory. The processor can access the memory and call the instructions to execute them in order to complete the text reading detection method described in the above embodiments.

[0084] In some embodiments, the text reading detection device 20 described above can also be constructed from hardware devices. For example, the text reading detection device 20 can be constructed from one or more chips, and the chips can work together to complete the text reading detection method described in the various embodiments above. As another example, the text reading detection device 20 can also be constructed from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM (Acorn RISC Machine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0085] It should be noted that the text reading detection device 20 described above can execute the text reading detection method for electronic devices provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the embodiments of the text reading detection device 20 can be found in the text reading detection method for electronic devices provided in the embodiments of this application.

[0086] See Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 30 includes one or more processors 31 and a memory 32. The memory 32 is connected to one or more processors 31, for example, via a bus.

[0087] Processor 31 is configured to support the electronic device 30 in performing the corresponding functions in the text reading detection method described in the above method embodiments. Processor 31 may be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0088] Memory 32 is used to store program code, etc. Memory 32 may include volatile memory (VM), such as random access memory (RAM); memory may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 32 may also include combinations of the above types of memory.

[0089] The memory 32 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the text reading detection method in the embodiments of this application. The processor 31 executes various functional applications and data processing of the text reading detection method and the text reading detection device by running the non-volatile software programs, instructions, and modules stored in the memory 32, thereby realizing the functions of each module or unit of the text reading detection method and the text reading detection device provided in the above method embodiments.

[0090] The memory 32 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on the use of the text-to-speech detection device. In some embodiments, the memory 32 may include memory remotely located relative to the processor 31, and such remote memory may be connected to the text-to-speech detection device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0091] The one or more modules are stored in the memory 32. When executed by the one or more processors 31, they execute the text reading detection method in any of the above method embodiments. For example, they execute the method steps described in the above method embodiments to realize the functions of the modules described in the above device embodiments.

[0092] The electronic device 30 in this application embodiment may specifically be an ultra-mobile personal computer device, a smart display or all-in-one machine, a server or server cluster, etc.

[0093] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors 31, enabling the one or more processors 31 to execute the text reading detection method in any of the above method embodiments.

[0094] This application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by the electronic device 30, enable the electronic device 30 to perform the text reading detection method in any of the above method embodiments.

[0095] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0096] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A text reading detection method, characterized in that, include: Obtain the audio of the user reading aloud, and the reference text corresponding to the user reading the audio; Based on the audio and the reference text, obtain the matching degree between the audio and the reference text; Identify the audio and obtain the corresponding identified text. Based on the identified text and the reference text, detect whether there is any additional content in the audio that is not the reference text, so as to obtain the additional content detection result; Based on the matching degree between the audio and the reference text, and the results of the additional content detection, it is determined whether the audio read by the user meets the reading requirements of the reference text.

2. The method according to claim 1, characterized in that, After acquiring the audio of the user's reading, the method further includes: detecting whether the audio of the user's reading contains a human voice; If it is confirmed that the audio read by the user contains human voice, then the step of detecting whether the audio read by the user meets the reading requirements of the reference text is performed based on the matching degree between the audio and the reference text, and the additional content detection result.

3. The method according to claim 1, characterized in that, The step of obtaining the matching degree between the audio and the reference text based on the audio and the reference text includes: The reference text and its corresponding position encoding information are input into the decoding module to obtain a text content matrix. The audio is input into the encoding module to obtain the audio feature matrix; The decoding module calculates the correlation between the text content matrix and the audio feature matrix to obtain a reference text weight matrix, which represents the matching degree between the audio and the reference text.

4. The method according to claim 1, characterized in that, The process of recognizing the audio and obtaining the corresponding recognized text includes: The audio input is recognized by a preset audio model to obtain the recognized text. The audio model includes an audio encoding module and an audio decoding module. The method further includes: Obtain training samples, which include batches of audio data and reference text corresponding to the audio data; The audio data is preprocessed to obtain preprocessed audio data; The preprocessed audio data is input into the audio model to be trained. The acoustic features of the preprocessed audio data are extracted by the audio encoding module, and the recognition text is generated by the audio decoding module based on the acoustic features. Compare the consistency between the identified text and the manually annotated text corresponding to the audio data; The parameters of the audio model to be trained are adjusted to maximize the consistency between the generated recognized text and the corresponding manually annotated text. The audio model to be trained that has the highest consistency between the output recognized text and its corresponding manually annotated text is used as the preset audio model obtained through training.

5. The method according to claim 1, characterized in that, The step of detecting whether there is additional content in the audio that is not the reference text, based on the identified text and the reference text, to obtain additional content detection results, includes: Based on the identified text and the reference text, obtain the phoneme sequence corresponding to each word in the identified text and the reference text; Based on the phoneme sequence of the identified text, each phoneme in the phoneme sequence of the identified text is encoded into a first vector to obtain a first feature matrix corresponding to the phoneme sequence of the identified text, wherein the first vector includes the interaction information between the phoneme corresponding to the first vector and other phonemes in the phoneme sequence of the identified text. Based on the phoneme sequence of the reference text, each phoneme in the phoneme sequence of the reference text is encoded into a second vector to obtain a second feature matrix corresponding to the phoneme sequence of the reference text, wherein the second vector includes the interaction information between the phoneme corresponding to the second vector and other phonemes in the phoneme sequence of the reference text. Based on the first feature matrix and the second feature matrix, a recognition text weight matrix is ​​obtained. The recognition text weight matrix is ​​used to detect whether there is any additional content in the audio that is not the reference text, so as to obtain additional content detection results.

6. The method according to any one of claims 1 to 5, characterized in that, The step of detecting whether the audio read by the user meets the reading requirements of the reference text based on the matching degree between the audio and the reference text, and the result of the additional content detection, includes: The matching degree between the audio and the reference text, and the additional content detection result are respectively input into a parallel convolutional neural network to obtain the first convolutional feature corresponding to the matching degree and the second convolutional feature corresponding to the additional content detection result. The first convolutional feature is obtained by processing the first pooling layer, and the second convolutional feature is obtained by processing the second pooling layer. The first pooling feature and the second pooling feature are input into a fully connected layer to output a prediction result of whether the audio read by the user meets the reading requirements of the reference text.

7. A text reading detection device, characterized in that, include: The audio and text acquisition module is used to acquire the audio of the user reading aloud, and the reference text corresponding to the user reading the audio. The matching degree acquisition module is used to obtain the matching degree between the audio and the reference text based on the audio and the reference text; The text recognition acquisition module is used to recognize the audio and obtain the recognized text corresponding to the audio. The additional content detection result module is used to detect whether there is any additional content in the audio that is not the reference text, based on the identified text and the reference text, so as to obtain the additional content detection result; The text reading detection module is used to detect whether the audio read by the user meets the reading requirements of the reference text based on the matching degree between the audio and the reference text, as well as the detection results of the additional content.

8. An electronic device, characterized in that, include: A memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor, when executing the one or more computer programs, causing the electronic device to implement the text reading detection method as described in any one of claims 1 to 6.

9. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by an electronic device, cause the electronic device to perform the text reading detection method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions that, when executed by an electronic device, cause the electronic device to perform the text reading detection method according to any one of claims 1 to 6.

Images