Text similarity identification method, text reading detection method, and related apparatuses

Abstract

This application discloses a text similarity recognition method, a text reading detection method, and related apparatus. The method involves acquiring an audio recording of a user reading aloud, and the corresponding reference text; recognizing the audio to obtain the corresponding recognized text; extracting features from both the reference and recognized texts to obtain letter sequence feature information for both the reference and recognized texts; and performing text similarity recognition based on these letter sequence feature information to obtain the similarity information between the reference and recognized texts. This application reduces resource dependence and complexity in text similarity recognition, improves efficiency, and simplifies the process. It also detects whether the user's audio reading meets requirements, improving the accuracy and fairness of oral assessment and effectively evaluating learners' learning attitudes and actual learning outcomes.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a text similarity recognition method, a text reading detection method, and related apparatus. Background Technology

[0002] Oral assessment uses computer technology to evaluate learners' spoken pronunciation levels. Text reading detection, a key module of any oral assessment system, aims to determine whether learners can correctly read aloud a given text.

[0003] In traditional methods, assessment results are relatively reliable when learners strictly adhere to the text during reading aloud. However, there are instances where learners do not read according to the text, causing the assessment system to fail to accurately detect these instances and potentially resulting in inaccurate scores. This is particularly prevalent in oral language learning, where learners often fail to read according to the text, and existing systems often cannot effectively distinguish between these situations. Therefore, it is essential to add a text-reading aptitude check component to oral language assessments. Summary of the Invention

[0004] One objective of this application is to provide a text similarity recognition method, a text reading detection method, and related apparatus to address how to reduce resource dependence and complexity when performing text similarity recognition, thereby accurately detecting whether a user is reading aloud from text in oral assessment.

[0005] To address the aforementioned technical problems, one technical solution adopted in this application is to provide a text similarity recognition method, comprising: acquiring audio of a user reading aloud, and reference text corresponding to the audio; recognizing the audio to obtain recognized text corresponding to the audio; extracting features from the reference text and the recognized text respectively to obtain letter sequence feature information corresponding to the reference text and the recognized text; and performing text similarity recognition based on letter sequence feature information to obtain similarity information between the reference text and the recognized text. Specifically, by using letter sequence-based feature extraction, it eliminates the need to rely on word pronunciation dictionary resources and related models for phoneme sequence conversion, instead directly using letter sequences for feature extraction and similarity calculation, reducing dependence on external resources. Directly extracting letter sequence feature information from the reference text and the recognized text avoids complex phoneme sequence conversion and multi-sound processing steps, simplifying the entire process and reducing complexity. Therefore, this method, employing letter sequence-based feature extraction and similarity calculation, can reduce resource dependence and complexity to a certain extent, while improving efficiency and simplifying the process.

[0006] Optionally, the step of extracting features from the reference text and the identified text respectively to obtain letter sequence feature information corresponding to the reference text and the identified text, and performing text similarity recognition based on letter sequence feature information to obtain similarity information between the reference text and the identified text, includes: obtaining a first letter sequence mapping feature corresponding to the reference text and a second letter sequence mapping feature corresponding to the identified text; adding position information to the first letter sequence mapping feature to obtain a third letter sequence mapping feature containing content information and position information of the letter sequence; adding position information to the second letter sequence mapping feature to obtain a fourth letter sequence mapping feature containing content information and position information of the letter sequence; inputting the third letter sequence mapping feature and the fourth letter sequence mapping feature into a self-attention mechanism feature encoding network respectively to output a first letter sequence feature matrix corresponding to the reference text and a second letter sequence feature matrix corresponding to the identified text; inputting the first letter sequence feature matrix and the second letter sequence feature matrix into a neural network encoding layer for feature fusion and encoding to output similarity information between the reference text and the identified text. By adding location information, a self-attention mechanism feature encoding network, and a neural network encoding layer to predict text similarity information, it is helpful to capture the semantic and structural information between texts, thereby enabling similarity prediction.

[0007] Optionally, the step of extracting features from the reference text and the identified text respectively to obtain letter sequence feature information corresponding to the reference text and the identified text, and performing text similarity recognition based on letter sequence feature information to obtain similarity information between the reference text and the identified text, is executed by a preset text similarity recognition model; the method further includes: acquiring training samples, the training samples including training data of the reference text corresponding to the training audio, training data of the identified text corresponding to the training audio, and the reference text... The training data is compared with the target similarity result of the text recognition training data; features are extracted from the training samples to obtain letter sequence feature information for training; the letter sequence feature information for training includes the content information, position information, and relationship between letters of the training samples; the letter sequence feature information for training is input into the text similarity recognition model to be trained to output the similarity prediction result corresponding to the training sample; the similarity difference information between the similarity prediction result and the target similarity result is determined; the text similarity recognition model to be trained is trained according to the similarity difference information until the preset text similarity recognition model is obtained. This method trains a model capable of text similarity recognition based on letter sequence feature information, which can improve the performance and accuracy of text similarity recognition.

[0008] Optionally, obtaining training samples includes: obtaining reference text training data corresponding to the training audio, and obtaining recognition text training data with content consistent with the reference text training data, wherein the recognition text training data is obtained based on the training audio generated when the reference text training data is read aloud; perturbing the training audio corresponding to the recognition text training data to obtain perturbed audio; and recognizing the perturbed audio to obtain new recognition text training data; wherein the reference text training data corresponding to the training audio, the recognition text training data with content consistent with the reference text training data, and the new recognition text training data constitute the first training sample. By constructing positive examples and utilizing consistent reference text training data, corresponding recognition text training data, and perturbed audio, the quality of training data can be effectively improved, data diversity increased, and model performance and generalization ability enhanced.

[0009] Optionally, obtaining training samples further includes: acquiring recognition text training data that does not match the reference text training data based on the reference text training data corresponding to the training audio; wherein, the reference text training data corresponding to the training audio and the recognition text training data that does not match the reference text training data constitute a second training sample. By constructing negative examples, the model can effectively learn the features of correct matching and non-matching, improving the model's generalization ability and accuracy, enabling it to better handle text matching problems in various situations.

[0010] To address the aforementioned technical problems, one technical solution adopted in this application is to provide a text reading detection method, comprising: acquiring an audio recording of a user reading aloud, and a reference text corresponding to the audio recording; obtaining a matching degree between the audio and the reference text based on the audio and the reference text; recognizing the audio to obtain the recognized text corresponding to the audio; acquiring similarity information between the recognized text and the reference text, wherein the similarity information is obtained through the aforementioned text similarity recognition method; and detecting whether the audio recording of the user reading aloud meets the reading requirements of the reference text based on the matching degree between the audio and the reference text, and the similarity information between the recognized text and the reference text. This method improves the accuracy and fairness of oral assessment, better evaluates learners' learning attitudes and actual learning outcomes, and reduces the complexity and resource dependence of text similarity recognition.

[0011] Optionally, obtaining the matching degree between the audio and the reference text based on the audio and the reference text includes: inputting the reference text and its corresponding positional encoding information into a decoding module to obtain a text content matrix; inputting the audio into an encoding module to obtain an audio feature matrix; and calculating the correlation between the text content matrix and the audio feature matrix through the decoding module to obtain a reference text weight matrix, which represents the matching degree between the audio and the reference text. This summarizes the key processes of the model in processing audio and reference text, including input processing, feature extraction, and interactive computation. This interactive method helps the model better understand the relationship between the audio and the reference text, thereby enabling the interpretation and understanding of the audio content and reflecting whether the user truly read the reference text aloud.

[0012] Optionally, detecting whether the audio read by the user meets the reading requirements of the reference text based on the matching degree between the audio and the reference text, and the similarity information between the identified text and the reference text, includes: inputting the matching degree between the audio and the reference text, and the similarity information between the identified text and the reference text, into a parallel convolutional neural network to obtain a first convolutional feature corresponding to the matching degree, and a second convolutional feature corresponding to the similarity information; processing the first convolutional feature through a first pooling layer to obtain a first pooling feature, and processing the second convolutional feature through a second pooling layer to obtain a second pooling feature; inputting the first pooling feature and the second pooling feature into a fully connected layer to output a prediction result of whether the audio read by the user meets the reading requirements of the reference text. Here, by processing the matching degree and similarity information of the audio through a convolutional neural network, and finally predicting whether the audio read by the user meets the reading requirements of the reference text through a fully connected layer, this method can be applied to different speech reading detection tasks. It only requires fine-tuning the model parameters for new datasets to quickly adapt to different application scenarios.

[0013] To address the aforementioned technical problems, one technical solution adopted in this application is to provide a text similarity recognition device, comprising: a first data acquisition module for acquiring audio read aloud by a user and reference text corresponding to the audio read aloud by the user; a first text recognition module for recognizing the audio and obtaining the recognized text corresponding to the audio; and a first similarity recognition module for extracting features from the reference text and the recognized text respectively, obtaining letter sequence feature information corresponding to the reference text and the recognized text, and performing text similarity recognition based on the letter sequence feature information of the reference text and the recognized text to obtain similarity information between the reference text and the recognized text. This text similarity recognition device has the same beneficial effects as the aforementioned text similarity recognition method.

[0014] To address the aforementioned technical problems, one technical solution adopted in this application is as follows: a text reading detection device is provided, comprising: a second data acquisition module for acquiring audio read aloud by a user and reference text corresponding to the audio read aloud by the user; a matching degree detection module for obtaining the matching degree between the audio and the reference text based on the audio and the reference text; a second text recognition module for recognizing the audio and obtaining the recognized text corresponding to the audio; a second similarity recognition module for acquiring similarity information between the recognized text and the reference text, wherein the similarity information is obtained by the second similarity recognition module through the text similarity recognition method described above; and a reading detection module for detecting whether the audio read aloud by the user meets the reading requirements of the reference text based on the matching degree between the audio and the reference text, and the similarity information between the recognized text and the reference text. This text reading detection device has the same beneficial effects as the aforementioned text reading detection method.

[0015] To solve the above-mentioned technical problems, one technical solution adopted in this application is: to provide an electronic device, including: a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory, wherein when the processor executes the one or more computer programs, the electronic device implements a text similarity recognition method or a text reading detection method applied to the electronic device.

[0016] To solve the above-mentioned technical problems, one technical solution adopted in this application is to provide a non-volatile computer-readable storage medium that stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device performs the text similarity recognition method or text reading detection method as described above.

[0017] To solve the above-mentioned technical problems, one technical solution adopted in this application is: to provide a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by an electronic device, causing the electronic device to perform the text similarity recognition method or text reading detection method as described above.

[0018] The aforementioned electronic devices, non-volatile computer-readable storage media, and computer program products all possess the beneficial effects corresponding to the aforementioned text similarity recognition methods or text reading detection methods applied to electronic devices. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a text similarity recognition method provided in an embodiment of this application;

[0021] Figure 2 This is a flowchart of a method for obtaining similarity information between reference text and identified text, provided in an embodiment of this application.

[0022] Figure 3 This is a schematic diagram of the process for calculating the similarity information between the reference text and the identified text, provided in an embodiment of this application.

[0023] Figure 4 This is a flowchart of a text reading detection method provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the method flow for obtaining the matching degree between audio and reference text provided in the embodiments of this application;

[0025] Figure 6 This is a schematic diagram of the large audio model corresponding to the audio model provided in the embodiments of this application for recognizing the audio read by the user and obtaining the recognized text;

[0026] Figure 7 This is a schematic diagram of the method flow for detecting whether the audio of a user's reading meets the reading requirements of a reference text, as provided in the embodiments of this application.

[0027] Figure 8 This is a flowchart of a text reading detection method provided in another embodiment of this application;

[0028] Figure 9 This is a schematic diagram of the structure of a text similarity recognition device provided in an embodiment of this application;

[0029] Figure 10 This is a schematic diagram of the structure of a text reading detection device provided in an embodiment of this application;

[0030] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0032] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0033] It should be noted that in the various embodiments described below, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in interchange, etc.

[0034] Oral assessment utilizes computer technology to evaluate learners' spoken pronunciation levels, and it is widely used in learning settings. Automated computer-based oral assessment avoids the need for professional teachers, improving efficiency and scalability. Computers can quickly assess each learner's pronunciation and provide immediate feedback, making it timely and effective for learners. Therefore, oral assessment is widely used in language learning and education, providing learners with continuous diagnostic and feedback services for their spoken pronunciation.

[0035] Text-to-speech detection is a crucial module in oral assessment systems. It aims to detect whether learners have read the given text correctly. This is vital because only by ensuring learners read the text correctly can the assessment system accurately evaluate their pronunciation level. If learners do not read the text correctly, the assessment results may be distorted. Therefore, text-to-speech detection is an indispensable part of oral assessment systems; it relates to the accuracy and reliability of the entire system and is the foundation for achieving automated, timely feedback in oral assessment.

[0036] Text reading detection primarily assesses whether learners have read the given reference text correctly. This process involves two aspects: firstly, detecting whether learners have read the content of the reference text, i.e., assessing the matching degree between the learner's audio and the reference text; and secondly, detecting whether learners have read content outside the reference text.

[0037] A key step in detecting whether text is read aloud is identifying text similarity, and related technologies often employ phoneme sequence-based methods. Phoneme sequence-based text similarity recognition relies on word pronunciation dictionaries and corresponding word-to-phoneme models. Furthermore, when a word has multiple pronunciations, an additional module is needed to select the correct pronunciation sequence, increasing system complexity. Additionally, pronunciation differences exist between British and American English, requiring additional modules to handle these differences. Therefore, related technologies suffer from resource dependence and high complexity in text similarity recognition.

[0038] To address the aforementioned issues, this application provides a text similarity recognition method. By extracting features based on letter sequences, it eliminates the need for word pronunciation dictionaries and related models for phoneme sequence conversion. Instead, it directly uses letter sequences for feature extraction and similarity calculation, reducing reliance on external resources. By directly extracting letter sequence feature information from both the reference and recognition texts, it avoids complex phoneme sequence conversion and multi-sound processing steps, simplifying the entire process and reducing complexity. The text similarity recognition method of this application, employing letter sequence-based feature extraction and similarity calculation, can to some extent solve the problems associated with phoneme sequence-based text similarity calculation, reducing resource dependence and complexity while improving efficiency and simplifying the process.

[0039] In addition, based on this text similarity recognition method, this application embodiment also provides a text reading detection method, which obtains the audio of the user's reading and the corresponding reference text; calculates the matching degree based on the audio and the reference text, which can reflect whether the user has really read the reference text; detects whether there is content other than the reference text through the above text similarity recognition method. If such a situation exists, it means that the user may not have read the text completely as required; finally, taking into account the matching degree between the audio and the reference text, as well as whether there is any additional content, an overall evaluation result is obtained to determine whether the learner's reading meets the requirements.

[0040] This text-to-speech detection method effectively detects whether learners truly read the reference text aloud, preventing learners from obtaining high scores by not reading according to the text. This is crucial for improving the accuracy and fairness of oral assessments. Specifically, the detection of whether additional content was read aloud is performed using the aforementioned text similarity recognition method. The letter sequence-based feature extraction method does not rely on word pronunciation dictionaries or related models for phoneme sequence conversion, reducing dependence on external resources and making the system more independent and flexible. Directly using letter sequences for feature extraction and similarity calculation avoids complex phoneme sequence conversion and multi-sound processing steps, simplifying the entire text similarity recognition process and improving system efficiency. Moreover, the letter sequence-based feature extraction and similarity calculation method reduces the complexity of text similarity calculation, making the system easier to implement and maintain, while improving system performance and stability. Therefore, this text similarity recognition method reduces system complexity and resource dependence, simplifies the process, and improves accuracy and efficiency, thus providing a better solution for text-to-speech detection.

[0041] Through the text-to-speech detection method of this application, users or guardians can promptly understand the reasons for low scores and obtain relevant statistical data. This enables a better assessment of learners' learning attitudes and actual learning outcomes, providing valuable references for subsequent improvements. It also reduces instances where high scores are obtained despite not reading the text aloud, thus improving the accuracy and reliability of the oral assessment system.

[0042] The text similarity recognition method and text reading detection method of this application embodiment can be applied to oral assessment scenarios. Oral assessment scenarios include, for example, hardware devices such as microphones, speakers, headphones, recording devices, computers, and network connection devices. Microphones are used to record the user's voice input, responsible for capturing the user's spoken voice signal. Speakers are used to play reference text or oral practice content, helping the user hear the reference text and thus practice reading aloud. In some cases, users can also use headphones for oral practice to reduce external interference and better hear the reference text or system feedback. Recording devices are used to record the user's voice input; these can be recording devices integrated into the assessment system or external recording devices such as professional microphones or voice recorders. Computers are used to run the oral assessment system, performing tasks such as speech signal processing, feature extraction, speech recognition, and evaluation. In this application embodiment, the computer can be used to execute the text similarity recognition method and the text reading detection method. In some cases, the oral assessment system can also run on mobile devices, such as smartphones, tablets, or learning machines, allowing users to perform oral assessments through these devices. Oral assessment scenarios can also include network-connected devices. If the oral assessment system is based on cloud services, users can connect their devices (such as computers, smartphones, and other mobile devices) to the Internet to upload voice data for evaluation.

[0043] The text similarity recognition method and text reading detection method of this application can detect non-text-based readings in multilingual scenarios. Besides spoken language assessment, they can also be applied to other scenarios, such as hot word wake-up and the evaluation of audio content and topic relevance in open spoken language scenarios. Specifically, a speech recognition model for hot words is established to identify specific hot words, match them with the voice input to determine the presence of a wake-up word, and trigger corresponding operations or responses. Natural language processing technology is used to extract audio content or topic topics in open spoken language scenarios, and a relevance evaluation model is established to evaluate the relevance between the voice content and the topic topic by comparing the relevance between the voice content and topic keywords. The above scheme integrates speech recognition, text similarity calculation, and relevance evaluation models, enabling the detection of non-text-based readings in multilingual scenarios and allowing for reuse in other fields such as hot word wake-up and topic relevance evaluation.

[0044] Please see Figure 1 , Figure 1 This is a flowchart of a text similarity recognition method provided in an embodiment of this application. The method includes:

[0045] Step S11: Obtain the audio of the user reading aloud, and the corresponding reference text.

[0046] Users can record their reading aloud using a microphone, while the system simultaneously obtains the reference text the user is reading aloud. For example, when a user is taking an oral assessment, the audio refers to the audio the user is currently reading aloud for the oral assessment, and the reference text refers to the text provided by the system for the oral assessment—that is, the text the user is looking at while reading aloud, which can be considered the standard text. By obtaining this audio and reference text, the necessary data foundation is provided for subsequent comparison, recognition, and evaluation.

[0047] The system can be designed to support multiple languages, enabling it to receive and process readings from users with different language backgrounds. It can recognize audio recordings of users reading from different languages, thereby detecting whether the audio conforms to the reading requirements of the reference text.

[0048] Step S12: Recognize the audio and obtain the corresponding recognized text.

[0049] Text recognition refers to the extraction of text content from audio using speech recognition technology, which is the process of converting the speech signal in the audio into text form.

[0050] This can be achieved by using speech recognition technology to convert the user's audio into text representation. For example, a speech recognition engine can be used to process the audio and generate corresponding text output. This recognized text can then be used for subsequent analysis and comparison, providing a usable text format for the system's subsequent processing. In this embodiment, it can be achieved through... Figure 6 The audio model shown identifies the audio read by the user to obtain the identified text. For details, please refer to the following method implementation examples.

[0051] Step S13: Extract features from the reference text and the identified text respectively to obtain the letter sequence feature information corresponding to the reference text and the letter sequence feature information corresponding to the identified text. Then, perform text similarity recognition based on the letter sequence feature information to obtain the similarity information between the reference text and the identified text.

[0052] In this embodiment, text similarity recognition is based on letters. This method omits complex steps such as phoneme conversion and directly uses letter sequences for feature extraction and similarity calculation, simplifying the entire process and reducing the complexity of implementation and maintenance. Furthermore, since the method based on letter sequence feature information does not involve specific linguistic knowledge such as phoneme conversion and relies more on general letter sequence feature information, it has better generalization and wider applicability when dealing with different languages ​​or accents.

[0053] Specifically, please refer to Figure 2 Step S13 specifically includes:

[0054] Step S131: Obtain the first letter sequence mapping feature corresponding to the reference text, and the second letter sequence mapping feature corresponding to the recognition text.

[0055] The reference text and the recognition text can be input into the letter encoding mapping module respectively. The letter encoding mapping module can obtain the letter sequence mapping representation of the reference text, i.e., the first letter sequence mapping feature; and also obtain the letter sequence mapping representation of the recognition text, i.e., the second letter sequence mapping feature.

[0056] The letter encoding mapping module is a module that processes input text. It encodes and maps the letters in the text, converting the letter sequence into a feature representation. For example, this letter encoding mapping module maps the letter sequence to a vector space or other representation form for subsequent feature extraction, similarity calculation, or other text processing tasks.

[0057] Step S132: Add position information to the first letter sequence mapping feature to obtain a third letter sequence mapping feature containing both the content information and position information of the letter sequence.

[0058] Specifically, a learnable positional encoding mapping representation can be added to the first letter sequence mapping feature. This representation is primarily used to introduce positional information of the letter sequence into the first letter sequence mapping feature. Positional encoding mapping representations are used to introduce positional information when processing sequence data (such as natural language text).

[0059] Step S133: Add position information to the second letter sequence mapping feature to obtain a fourth letter sequence mapping feature containing content information and position information of the letter sequence.

[0060] Among them, a learnable positional encoding mapping representation can be added to the second letter sequence mapping feature, which is mainly used to introduce the positional information of the letter sequence in the first letter sequence mapping feature.

[0061] In this model, each letter in the first and second letter sequences has its corresponding positional information, which is introduced into the model in the form of positional encoding. The positional encoding can be fixed, i.e., predefined, and the specific encoding method could be, for example, sinusoidal positional encoding. The positional information can be a vector, whose value represents the relative position of the letter in the sequence.

[0062] In the process of adding positional encoding to the mapping features of the letter sequence between the reference text and the recognized text, the positional encoding is used to add positional information to the representation of each letter, enabling the model to understand the relative positional relationships of the letters in the sequence. This helps improve the model's ability to understand sequence data, especially when dealing with long sequences, helping the model accurately capture information from different positions within the sequence.

[0063] Step S134: Input the third letter sequence mapping feature and the fourth letter sequence mapping feature into the self-attention mechanism feature encoding network to output the first letter sequence feature matrix corresponding to the reference text and the second letter sequence feature matrix corresponding to the recognized text.

[0064] Self-attention is a mechanism that learns the dependencies between elements within a sequence. It allows each element in the input sequence to interact with other elements, thereby better capturing important information and connections within the sequence. In this embodiment, the dependencies between elements can be represented as relationships between letters. The self-attention feature encoding network generates a first-letter sequence feature matrix corresponding to the reference text and a second-letter sequence feature matrix corresponding to the identified text. These matrices contain text feature information encoded by the self-attention mechanism.

[0065] The first letter sequence feature matrix contains the feature representation of the letter sequence corresponding to the reference text. The relationships between the letters are learned in the feature encoding network through a self-attention mechanism, reflecting the dependencies and correlations between letters in the reference text. Similarly, the second letter sequence feature matrix contains the feature representation of the letter sequence corresponding to the recognition text. The relationships between the letters are also learned in the feature encoding network through a self-attention mechanism, reflecting the dependencies and correlations between letters in the recognition text.

[0066] The text feature information is an abstract representation obtained through model learning. It contains important features of the input text sequence (i.e., the letter sequences corresponding to the reference text and the recognition text respectively). These features can characterize the semantics, structure or other important information in the text, which helps the model understand the text and perform subsequent tasks, such as similarity calculation.

[0067] Step S135: Input the first letter sequence feature matrix and the second letter sequence feature matrix into the neural network encoding layer to perform feature fusion and encoding, so as to output the similarity information between the reference text and the identified text.

[0068] The first and second letter sequence feature matrices are input into the neural network encoding layer. In this layer, these feature matrices undergo feature fusion and encoding to better capture textual similarity information. This may include feature combination, dimensionality transformation, and nonlinear transformation. Finally, the neural network encoding layer outputs similarity information between the reference text and the identified text, which can be used to assess the degree of similarity between the two texts.

[0069] In a neural network, the encoding layer refers to one or more layers that perform feature extraction, feature fusion, and encoding on the input data. This layer can be various types of neural network layers, such as fully connected layers, convolutional neural network layers, and recurrent neural network layers.

[0070] This application's embodiments predict text similarity information by adding location information, a self-attention mechanism feature encoding network, and a neural network encoding layer. This helps capture the semantic and structural information between texts, thereby enabling similarity prediction. This process can be referenced... Figure 3 .

[0071] In some embodiments, step S13 is performed by a preset text similarity recognition model. This text similarity recognition model utilizes deep learning techniques to extract high-level features from the reference text and the recognized text, and calculates the similarity between the texts by learning how to compare these features. The training process of this text similarity recognition model is provided below, through which the text similarity recognition model is obtained.

[0072] Acquire training samples, which include reference text training data corresponding to the training audio, recognition text training data corresponding to the training audio, and target similarity results between the reference text training data and the recognition text training data; extract features from the training samples to obtain letter sequence feature information for training, wherein the letter sequence feature information for training includes the content information, position information, and relationship between letters of the training samples; input the letter sequence feature information for training into the text similarity recognition model to be trained to output the similarity prediction result corresponding to the training samples; determine the similarity difference information between the similarity prediction result and the target similarity result; train the text similarity recognition model to be trained based on the similarity difference information until the text similarity recognition model is obtained.

[0073] The training samples include the reference text training data corresponding to the training audio, the recognition text training data corresponding to the training audio, and the target similarity results between the reference text training data and the recognition text training data. These training samples can be manually labeled and used to train the model.

[0074] The reference text training data consists of pre-written text templates for specific audio content. These texts are typically accurate and standardized, serving as a standard reference. By reading aloud or transcribing these reference text training data, an audio dataset for training—the training audio—can be generated. The recognition text training data is the text data obtained through speech recognition of the training audio.

[0075] Feature extraction is performed on the training samples, including content information, positional information, and relationships between letters. This process can be based on... Figure 3 The illustrated process involves feature extraction through a letter encoding mapping module, adding positional information, and a self-attention mechanism feature encoding network to obtain content information, positional information, and relationships between letters. The letter sequence feature information used for training is input into the text similarity recognition model to be trained. This model can be a previously defined neural network structure used to learn the similarity relationships between texts. The model outputs the similarity prediction result corresponding to the training samples. This result represents the model's prediction of the similarity between the reference text and the recognized text, and this prediction result is compared with the target similarity result. By comparing the similarity prediction result with the target similarity result, the error or difference information in the similarity prediction is determined, and this error is quantified using a loss function. Using the similarity difference information, the model parameters are adjusted through a backpropagation algorithm and an optimizer, enabling the model to more accurately predict the similarity between texts. This process can be accomplished by minimizing the difference between the similarity prediction result calculated by the loss function and the target similarity result. The text similarity recognition model in this embodiment continuously adjusts its parameters to better capture the similarity information between texts, thereby improving the model's accuracy and generalization ability in text similarity recognition tasks.

[0076] In some embodiments, obtaining training samples includes: obtaining reference text training data corresponding to the training audio, and obtaining recognition text training data with content consistent with the reference text training data, wherein the recognition text training data is obtained based on the training audio generated when reading the reference text training data; perturbing the training audio corresponding to the recognition text training data to obtain perturbed audio; and recognizing the perturbed audio to obtain new recognition text training data; wherein the reference text training data corresponding to the training audio, the recognition text training data with content consistent with the reference text training data, and the new recognition text training data constitute the first training sample. Perturbing the training audio corresponding to the recognition text training data includes adding speech rate, noise, volume, etc., to the audio corresponding to the recognition text training data. Perturbation helps the model better adapt to various environments and noise conditions in the real world, making the trained model more robust and capable of recognizing more varied audio data. This embodiment, by constructing positive examples and utilizing consistent reference text training data, corresponding recognition text training data, and perturbed audio, can effectively improve the quality of training data, increase data diversity, and enhance the model's performance and generalization ability.

[0077] In some embodiments, obtaining training samples further includes: obtaining recognition text training data that does not match the reference text training data based on the reference text training data corresponding to the training audio; wherein, the reference text training data corresponding to the training audio and the recognition text training data that does not match the reference text training data constitute the second training sample. By constructing negative examples, the model can effectively learn the features of correct matching and non-matching, improving the model's generalization ability and accuracy, enabling it to better handle text matching problems in various situations.

[0078] Both the first and second training samples can be input into the text similarity recognition model. These two types of training samples provide the positive and negative examples needed to train the model, helping it learn the features of correct matches and mismatches, thus improving its generalization ability and accuracy.

[0079] In some embodiments, the training samples may specifically be a dataset of children's text readings. This dataset may contain reference texts read aloud by children and corresponding recognition texts. Training with children's text reading data allows the model to better adapt to the characteristics of children's speech and language habits, thereby improving the model's performance and effectiveness when processing children's text reading data.

[0080] Please see Figure 4 , Figure 4 This is a flowchart of a text-to-speech detection method provided in an embodiment of this application. The method includes:

[0081] Step S21: Obtain the audio of the user reading aloud, and the reference text corresponding to the audio of the user reading aloud.

[0082] Users can record their reading aloud using a microphone, while the system simultaneously obtains the reference text the user is reading. For example, when a user is taking an oral assessment, the audio refers to the audio the user is currently reading aloud for the assessment, and the reference text refers to the text provided by the system for the oral assessment—the text the user is looking at while reading. By obtaining this audio and reference text, the necessary data foundation is provided for subsequent comparison, recognition, and evaluation.

[0083] The system can be designed to support multiple languages, enabling it to receive and process readings from users with different language backgrounds. It can recognize audio recordings of users reading from different languages, thereby detecting whether the audio conforms to the reading requirements of the reference text.

[0084] Step S22: Based on the audio and the reference text, obtain the matching degree between the audio and the reference text.

[0085] By comparing the content of a user's audio reading with a reference text and calculating the degree of matching, it is possible to assess whether the user's reading meets the expected requirements of the reference text. For example, these expected requirements include that the content read by the user should be identical to the content of the reference text, including the accuracy of words, phrases, and sentences; the user's reading should also conform to the rhythm and intonation of the language, neither too fast nor too slow; and, based on the logical structure of the reference text, the user should be able to pause at appropriate points to make the sentences clearer; etc.

[0086] This matching degree refers to the textual similarity between the audio content and the reference text. For example, it compares words, phrases, and sentences to determine the degree of consistency between them.

[0087] When determining the matching degree between audio and reference text, various methods can be used for comparison and evaluation. For example, text similarity comparison can be employed, using text comparison algorithms (such as edit distance algorithm, cosine similarity, etc.) to compare the similarity between the identified text in the audio and the reference text. Alternatively, a trained classifier model can be used to determine the matching degree between audio and reference text; for instance, a neural network model can be used to evaluate the matching degree between audio and text, such as a recurrent neural network (RNN) or a convolutional neural network (CNN).

[0088] In some embodiments, the matching degree is mainly used to evaluate whether the audio reads out words from the reference text. Obtaining the matching degree between the audio and the reference text includes: inputting the reference text and its corresponding positional encoding information into a decoding module to obtain a text content matrix; inputting the audio into an encoding module to obtain an audio feature matrix; and calculating the correlation between the text content matrix and the audio feature matrix through the decoding module to obtain a reference text weight matrix, which represents the matching degree between the audio and the reference text. This embodiment summarizes the key processes of the model when processing audio and reference text, including input processing, feature extraction, and interactive computation. This interactive method helps the model better understand the relationship between audio and text, thereby enabling the interpretation and understanding of the audio content and reflecting whether the user truly read out the reference text.

[0089] Specifically, please refer to Figure 5 , Figure 5 This is a schematic diagram corresponding to the method flow for obtaining the matching degree between audio and reference text provided in the embodiments of this application. For example... Figure 5 As shown, the audio input encoding module (Encoder Module) inputs the reference text and its corresponding learned positional encoding information (Learned Positional Encoding) into the decoding module (composed of multiple Decoder Blocks). The reference text serves as the text content input to the decoding module, and also incorporates learnable positional encoding information to help the model better understand the positional relationships within the text. This positional encoding information helps the model distinguish words at different positions within the reference text. In the decoding module, a cross-attention mechanism is used to calculate the relationship between the text content matrix and the audio hidden feature matrix output by the encoding module. This cross-attention mechanism helps the model focus on the degree of correlation between the audio and the reference text, thereby calculating the reference text weight matrix, reflecting the relative importance of the audio and reference text in terms of matching degree. Through this process, the model can utilize the cross-attention mechanism to calculate the weight matrix relative to the audio and reference text, which can be used to represent the matching degree between the audio and the reference text. This method helps the model better understand the relationship between the audio and the reference text, thereby enabling the interpretation and understanding of the audio content to determine whether the user has read the reference text correctly.

[0090] The model can be a Transformer model (a deep learning model based on an attention mechanism, which includes two main components: an encoding module and a decoding module). In this embodiment, the relationship between audio and reference text is processed and their matching degree is calculated using a Transformer architecture model that includes an encoding module and a decoding module.

[0091] Figure 5 The corresponding model, on the one hand, is used to detect whether the audio reads out words from the reference text. When the reference text and audio do not match, they can be distinguished by the cross-attention mechanism matrix of the reference text and audio features (i.e., the reference text weight matrix). On the other hand, the model is also used to detect whether the audio contains human voices, in order to eliminate false detections of pure noise and silence. Therefore, after obtaining the audio read by the user, the method further includes: detecting whether the audio read by the user contains human voices. If the audio read by the user contains human voices, then step S25 is executed; if the audio read by the user does not contain human voices, then the text reading detection method of this embodiment of the application is not required. Adding a human voice detection scheme to the text reading detection method can effectively solve the problem that existing methods are easily affected by background noise, and improve the accuracy and robustness of detection.

[0092] The process of detecting whether a user's audio reading contains human voice includes the following steps: First, the Mel-spectral characteristics (a method of audio feature representation) of the user's audio reading are input into an encoder module. This encoder module can contain various layers, such as convolutional layers, recurrent neural network (RNN) layers, or Transformer layers, to extract hidden acoustic features. These hidden acoustic features are feature representations containing important acoustic information extracted from the Mel-spectral characteristics of the user's audio reading by the encoder module. The output is a hidden acoustic feature matrix, which captures important acoustic information from the audio data, such as speech features, audio spectrum features, speech content, rhythm, and prosody. Then, the hidden acoustic feature matrix is ​​input into an average pooling layer. The average pooling operation helps eliminate temporal dimension information, converting time-series data into a fixed-length feature vector. This step transforms the hidden acoustic feature matrix into a fixed-dimensional audio hidden feature vector for subsequent processing and classification. Next, the audio hidden layer feature vector is input into a fully connected layer for binary classification. The obtained audio hidden layer feature vector is fed into the fully connected layer, which contains multiple neurons. After processing by activation functions and other methods, the model can perform binary classification to determine the presence or absence of human voices. In this embodiment, the model can extract acoustic features from the audio Mel-spectral features, then convert them into a fixed-length feature vector through an average pooling layer, and finally perform binary classification through a fully connected layer to determine whether human voices are present in the audio. This model can be a deep learning-based neural network model used for human voice detection tasks in audio processing.

[0093] In some embodiments, during execution via Figure 5Before the corresponding model can detect whether words in the reference text are read aloud in the audio, and whether the audio contains human voices, it can be trained. The training dataset includes two types of audio data: The first type is audio data containing human voices. This type of data is used to train the model to identify whether audio contains human voices. This data can be obtained from Automatic Speech Recognition (ASR) datasets, which typically contain various audio segments containing human voices, such as speaking or reading aloud. The second type is non-human voice audio data. This type of data is used to train the model to identify non-human voice audio. This type of data can be obtained from two sources: The first is by using an event detection model (such as CLAP (Contrastive Learning of Audios and Texts by Pretraining)) to filter non-human voice audio from the evaluation data. This method involves using an event detection model to identify non-human voice audio, such as ambient sounds and noise. The second approach is to utilize non-human audio data from publicly available audio understanding and event detection datasets, such as FSD50k, ClothoV2, and AudioCaps. These datasets contain various types of audio data that can be used to train models to recognize non-human audio.

[0094] The steps for training this model may include: organizing the acquired audio data containing human voices and non-human voice audio data into a training set; extracting Mel-spectral features from the audio data and converting it into an input format acceptable to the model; dividing the dataset into training and validation sets for training and evaluating model performance; designing a deep learning model suitable for this task, including encoding modules, decoding modules, average pooling layers, and fully connected layers, for human voice detection and word content detection; training the model using the training set data, continuously adjusting the model parameters through backpropagation to enable it to accurately predict human voices and word content in the audio data; evaluating the model's performance using the validation set and adjusting the model parameters to improve performance; and finally, deploying the trained model in a real-world application to detect whether words from a reference text are read aloud in the audio and whether human voices are present.

[0095] Step S23: Recognize the audio and obtain the corresponding recognized text.

[0096] This can be achieved by using speech recognition technology to convert the user's audio into text representation. For example, a speech recognition engine can be used to process the audio and generate corresponding text output. This recognized text can then be used for subsequent analysis and comparison, providing a usable text format for the system's later processing.

[0097] In some embodiments, recognizing audio and obtaining corresponding recognized text includes: inputting the audio into a preset audio model for recognition to obtain recognized text, wherein the audio model includes an audio encoding module and an audio decoding module.

[0098] Specifically, please refer to Figure 6 , Figure 6 This is a schematic diagram corresponding to the large audio model provided in this application embodiment for recognizing audio read aloud by a user and obtaining the recognized text. For example... Figure 6 As shown, the audio input for user reading is "2*Conv1D+GELU". This "2*Conv1D+GELU" indicates that two consecutive one-dimensional convolutional layers (Conv1D) are used in the neural network, and their outputs are then non-linearly transformed using the GELU activation function. Sinusoidal Positional Encoding is a positional encoding method that uses sine and cosine functions to encode information at each position in a sequence. Figure 6 The part within the dashed box on the left is the audio encoding module, which is used to encode the deep acoustic features of the audio. The part on the right is the decoding module for recognizing text, and finally outputs the recognized text corresponding to the audio read aloud by the user. In this embodiment, the method further includes: obtaining the preset audio model. Obtaining the preset audio model includes: obtaining training samples, which include batches of audio data and corresponding reference text in a children's text reading scenario; the audio data includes manually labeled data and weakly labeled data; preprocessing the audio data to obtain preprocessed audio data; inputting the preprocessed audio data into the audio model to be trained; extracting the acoustic features of the preprocessed audio data through the audio encoding module; and generating the recognized text based on the acoustic features through the audio decoding module; comparing the consistency between the recognized text and the manually labeled text corresponding to the audio data; adjusting the parameters of the audio model to be trained to maximize the consistency between the generated recognized text and the corresponding manually labeled text; and using the audio model to be trained that has the highest consistency between the output recognized text and its corresponding manually labeled text as the preset audio model obtained through training. In this model, the manually labeled text corresponding to the audio data can be tags selected from the reference text. These tags help the model learn the correspondence between the recognized text and the standard text. The purpose of this training process is to optimize the model parameters using training data, so that the generated recognized text is as consistent as possible with the manually labeled text. This model can be used in audio-text recognition tasks, such as those involving children reading aloud.

[0099] The batch of audio data and corresponding reference texts for children's text reading scenarios were obtained in two ways. One was manually labeled data, which included a small portion of audio data from children's text reading scenarios and their corresponding accurate reference texts, used as part of the training set. The other was weakly labeled data, obtained through model filtering, which included a large amount of audio data from children's text reading scenarios. The acquisition of weakly labeled data mainly involved the following steps: The original reading data contained audio and reference text, but the two might not match, so a trained recognition model was used to obtain the recognition text; the original audio and reference text, and the original audio and recognition text were respectively fed into a scoring system to obtain pronunciation accuracy scores; audio data where the accuracy scores of words in both the reference text and the recognition text were greater than a preset score (e.g., a preset score of 85 out of 100) were selected. This means that after audio recognition, the difference between the obtained recognition text and the reference text is small.

[0100] The preprocessing of the audio data includes data augmentation of the original audio data by adjusting speech rate, volume, and noise levels to increase the diversity of the training set.

[0101] This method fully utilizes the capabilities of large-scale pre-trained models and conducts targeted customized training for children's text reading scenarios, which can effectively improve the accuracy and robustness of text reading detection.

[0102] Step S24: Obtain similarity information between the identified text and the reference text.

[0103] For a detailed process of obtaining similarity information between the identified text and the reference text, please refer to the above method implementation examples.

[0104] Step S25: Based on the matching degree between the audio and the reference text, and the similarity information between the identified text and the reference text, detect whether the audio read by the user meets the reading requirements of the reference text.

[0105] This application embodiment designs a non-text-based reading detection model. This non-text-based reading detection model can comprehensively consider the matching degree between the audio and the reference text, as well as the similarity information between the identified text and the reference text. It uses convolutional neural networks and fully connected layers to extract and predict features to determine whether the audio read by the user meets the specified reading requirements.

[0106] Specifically, please refer to Figure 7The matching degree between the user's audio reading and the reference text, as well as the similarity information between the identified text and the reference text obtained above, are input into a parallel convolutional neural network. This parallel convolutional neural network can be two networks with the same network structure, obtaining a first convolutional feature corresponding to the matching degree and a second convolutional feature corresponding to the similarity information between the identified text and the reference text. The first convolutional feature is processed by a first pooling layer to obtain a first pooling feature, and the second convolutional feature is processed by a second pooling layer to obtain a second pooling feature. The first and second pooling features are then input into a fully connected layer to output a prediction result indicating whether the user's audio reading meets the reading requirements of the reference text. The process of processing the first convolutional feature through the first pooling layer and the second convolutional feature through the second pooling layer involves first and second pooling operations. This is a pooling operation performed on the feature maps generated by two different convolutional kernels. The pooling operation uses max pooling or average pooling to reduce the size of the feature maps, thereby reducing the computational load and the number of parameters while retaining important feature information. The first and second pooling layers refer to the specific layers applied after the convolutional layers for the pooling operation. The first and second pooling layers both perform pooling operations on the feature maps output by the convolutional layers, reducing the spatial size of the feature maps and thus extracting the most salient features. The first and second pooling features refer to the features obtained after processing by the corresponding pooling layers. These features contain the most salient feature information extracted in the convolutional layers, which helps subsequent fully connected layers perform classification or prediction tasks. The fully connected layer integrates and processes the features extracted by the preceding convolutional and pooling layers to output the final classification or regression result. The fully connected layer is a basic layer type in deep learning neural networks.

[0107] The prediction result can be a binary classification problem, meaning the prediction result includes one of two cases: it meets the reading requirements of the reference text, or it does not. Meeting the reading requirements of the reference text can include a high degree of matching between the words in the reference text and the words in the user's audio reading, meaning the user read the reference text correctly. It can also include the user's reading speed and rhythm matching the required reading speed and rhythm of the reference text. Furthermore, it can include the user's accurate pronunciation, correctly pronouncing the words and sentences in the reference text. Finally, it can include the user's reading conveying the correct emotion and intonation, conforming to the context and emotion required by the reference text. Not meeting the reading requirements of the reference text can be due to, for example, the user's audio reading containing incorrect or missing words, inconsistent with the reference text; the user's reading speed and rhythm not matching the required reading speed and rhythm, possibly too fast or too slow; inaccurate pronunciation, inconsistent with the pronunciation in the reference text; or the user's reading lacking the necessary emotion and intonation, inconsistent with the required emotion and intonation of the reference text, etc.

[0108] This method employs a convolutional neural network to process audio matching and additional content, ultimately using a fully connected layer to predict whether the user's audio reading meets the reading requirements of the reference text. This approach fully integrates audio matching and additional content detection information, helping to comprehensively consider audio quality and content completeness to make more accurate judgments on reading compliance. The entire model directly outputs the predicted reading compliance result from the input audio matching and additional content information, avoiding manual intervention in intermediate steps and achieving end-to-end automated learning, thus improving the evaluation and detection of reading quality. Furthermore, it exhibits good transferability, applicable to different speech reading detection tasks; simply fine-tuning the model parameters for new datasets allows for rapid adaptation to different application scenarios.

[0109] The non-text-reading detection model in this application embodiment can specifically be a deep learning model combining convolutional neural networks and fully connected layers. In some embodiments, the model can be pre-trained. During training, a training set is first acquired, which can be constructed into two types of data. One type is non-text-reading data selected based on a scoring model and a recognition model. The selection criteria for this type of data include: low scores for words in the reference text and high scores for words in the recognized text; and a low proportion of words in the reference text appearing in the recognized text. Samples are selected from such audio and text data as the training set; these samples are considered to be non-text-reading data. The other type of data is text simulation data, which can be selected from audio recordings with high consistency between the reference text and the recognized text, and high scores for both the reference text and the audio, to obtain data matching the reference text and audio. Through the training sets constructed in the above two ways, the model will be able to learn to distinguish different types of reading situations, thereby improving the accuracy of the assessment of the user's reading quality. After determining the training set, the model can be trained to obtain the non-text-reading detection model.

[0110] The text-to-speech detection method provided in this application can not only effectively detect whether a user has actually read the reference text, but also reduce the occurrence of high scores despite not reading the text correctly. Furthermore, knowing whether the user's audio reading meets the requirements of the reference text can help the user or guardian understand the reasons for a low score.

[0111] Please see Figure 8 , Figure 8This is a flowchart of a text reading detection method provided in another embodiment of this application. It mainly includes: a reference text content reading detection module, an audio large model module, a text recognition and reference text relevance detection module, and a non-text reading detection module. The reference text content reading detection module receives audio read aloud by the user and reference text as input. By comparing the speech in the audio with the content of the reference text, it calculates the weight matrix of the reference text in the entire audio. The reference text content reading detection module achieves the detection of reference text content and human voice in the audio by constructing a human voice detection training set and training a model. This module can be used to determine whether the audio is read according to the reference text requirements and eliminate false detections of pure noise and silence. The audio large model module receives audio as input, processes the audio using the audio large model, and outputs the corresponding recognized text. The audio large model is trained using the constructed audio training set data. The model includes an audio encoding module and a recognized text decoding module. The audio encoding module is used to perform deep acoustic feature encoding on the audio to extract important feature information. The text decoding module decodes encoded audio features into recognized text. It constructs a training set of data for children's scenarios and trains a large audio model to achieve the audio recognition task. The text-to-reference-text relevance detection module determines the similarity between the recognized and reference texts based on letters, simplifying the calculation process and improving the model's generalization ability. The non-text-reading detection module receives the weight matrix of the recognized text, the weight matrix of the reference text, and the voice detection results as input. By comprehensively considering this information, it determines whether the audio is non-text-reading. The output of this module is whether the audio meets the reading requirements of the given text. The training set for the non-text-reading detection module is divided into two categories: non-text-reading data in reading scenarios filtered according to the scoring and recognition models; and simulated text data. This module uses the reference text weight matrix and the recognized text weight matrix as input, extracts features using a parallel convolutional neural network, and performs prediction through a fully connected layer to determine whether the current audio is non-text-reading data.

[0112] This embodiment's method utilizes a neural network in an end-to-end manner to detect non-text-based reading, eliminating reliance on traditional manual features, scoring systems, and alignment time results. This improves the accuracy and generalization of non-text-based reading detection. It not only detects non-text-based reading but also identifies the root causes of low scores, providing valuable feedback for improving reading quality. Furthermore, it offers data statistics and learns user attitudes, reducing false detections of non-human voice data and enhancing user experience. It can detect non-text-based reading in multilingual scenarios and can be reused in the fields of hot word activation, relevance evaluation of audio content and topic themes in open-ended spoken language scenarios.

[0113] Please see Figure 9 , Figure 9 This is a schematic diagram of a text similarity recognition device provided in an embodiment of this application. The text similarity recognition device 30 includes: a first data acquisition module 31, used to acquire audio read aloud by a user and a reference text corresponding to the audio read aloud by the user; a first text recognition module 32, used to recognize the audio and obtain the recognized text corresponding to the audio; and a first similarity recognition module 33, used to extract features from the reference text and the recognized text respectively, obtain letter sequence feature information corresponding to the reference text and the recognized text, and perform text similarity recognition based on the letter sequence feature information to obtain similarity information between the reference text and the recognized text.

[0114] It should be noted that the text similarity recognition device 30 described above can execute the text similarity recognition method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the method. For technical details not described in detail in the embodiments of the text similarity recognition device 30, please refer to the text similarity recognition method applied to electronic devices provided in the embodiments of this application.

[0115] Please see Figure 10 , Figure 10 This is a schematic diagram of a text reading detection device provided in an embodiment of this application. The text reading detection device 40 includes: a second data acquisition module 41, used to acquire audio read aloud by a user and a reference text corresponding to the audio read aloud; a matching degree detection module 42, used to obtain the matching degree between the audio and the reference text based on the audio and the reference text; a second text recognition module 43, used to recognize the audio and obtain the recognized text corresponding to the audio; a second similarity recognition module 44, used to acquire similarity information between the recognized text and the reference text, the similarity information being obtained by the second similarity recognition module through the text similarity recognition method described in the above embodiment; and a reading detection module 45, used to detect whether the audio read aloud by the user meets the reading requirements of the reference text based on the matching degree between the audio and the reference text, and the similarity information between the recognized text and the reference text.

[0116] It should be noted that the text reading detection device 40 described above can execute the text reading detection method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in the embodiments of the text reading detection device 40 can be found in the above-described text reading detection method applied to electronic devices provided in the embodiments of this application.

[0117] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 50 includes one or more processors 51 and a memory 52. ​​The memory 52 is connected to one or more processors 51, for example, via a bus.

[0118] Processor 51 is configured to support the electronic device 50 in performing the corresponding functions of the text similarity recognition method and text reading detection method in the above method embodiments. Processor 51 may be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0119] Memory 52 is used to store program code, etc. Memory 52 may include volatile memory (VM), such as random access memory (RAM); memory may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory may also include combinations of the above types of memory.

[0120] The memory 52 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the text similarity recognition method and the text reading detection method in the embodiments of this application. The processor 51 executes various functional applications and data processing of the text similarity recognition method and the text reading detection method by running the non-volatile software programs, instructions, and modules stored in the memory 52, that is, it implements the text similarity recognition method and the text reading detection method provided in the above method embodiments.

[0121] The memory 52 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on the use of the text similarity recognition device and the text reading detection device.

[0122] The one or more modules are stored in the memory 52. ​​When executed by the one or more processors 51, they perform the text similarity recognition method and text reading detection method in any of the above method embodiments, for example, they perform the method steps described in the above method embodiments.

[0123] The electronic devices in this application embodiment may specifically be ultra-mobile personal computer devices, smart displays or all-in-one machines, servers or server clusters, etc.

[0124] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that can be executed by one or more processors, enabling the one or more processors to execute the text similarity recognition method and text reading detection method in any of the above method embodiments.

[0125] This application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by the electronic device, enable the electronic device to perform the text similarity recognition method and the text reading detection method in any of the above method embodiments.

[0126] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0127] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A text similarity recognition method, characterized in that, include: Obtain the audio of the user reading aloud, and the reference text corresponding to the audio of the user reading aloud; Identify the audio and obtain the corresponding identified text. Feature extraction is performed on the reference text and the identified text respectively to obtain the letter sequence feature information corresponding to the reference text and the letter sequence feature information corresponding to the identified text. Then, text similarity recognition based on letter sequence feature information is performed to obtain the similarity information between the reference text and the identified text.

2. The method according to claim 1, characterized in that, The step involves extracting features from both the reference text and the identified text to obtain letter sequence feature information corresponding to the reference text and the identified text, and then performing text similarity recognition based on the letter sequence feature information to obtain similarity information between the reference text and the identified text. Obtain the first letter sequence mapping feature corresponding to the reference text, and the second letter sequence mapping feature corresponding to the identified text; Add positional information to the first letter sequence mapping feature to obtain a third letter sequence mapping feature that contains both the content information and positional information of the letter sequence. Add positional information to the second letter sequence mapping feature to obtain a fourth letter sequence mapping feature that contains both the content information and positional information of the letter sequence. The third letter sequence mapping feature and the fourth letter sequence mapping feature are respectively input into a self-attention mechanism feature encoding network to output the first letter sequence feature matrix corresponding to the reference text and the second letter sequence feature matrix corresponding to the recognized text; The first letter sequence feature matrix and the second letter sequence feature matrix are input into the neural network encoding layer for feature fusion and encoding to output the similarity information between the reference text and the identified text.

3. The method according to claim 1, characterized in that, The steps of extracting features from the reference text and the identified text respectively to obtain the letter sequence feature information corresponding to the reference text and the letter sequence feature information corresponding to the identified text, and performing text similarity recognition based on the letter sequence feature information to obtain the similarity information between the reference text and the identified text are executed by a preset text similarity recognition model. The method further includes: Obtain training samples, which include reference text training data corresponding to the training audio, recognition text training data corresponding to the training audio, and target similarity results between the reference text training data and the recognition text training data; The training samples are subjected to feature extraction to obtain letter sequence feature information for training; the letter sequence feature information for training includes the content information, position information, and relationship between letters of the training samples. The letter sequence feature information used for training is input into the text similarity recognition model to be trained, so as to output the similarity prediction result corresponding to the training sample; Determine the similarity difference information between the similarity prediction result and the target similarity result; The text similarity recognition model to be trained is trained based on the similarity difference information until the preset text similarity recognition model is obtained.

4. The method according to claim 3, characterized in that, The acquisition of training samples includes: Obtain reference text training data corresponding to the training audio, and obtain recognition text training data that is consistent with the content of the reference text training data. The recognition text training data is obtained based on the training audio generated when reading the reference text training data. The training audio corresponding to the recognized text training data is perturbed to obtain the perturbed audio. The perturbated audio is then recognized to obtain new training data for the recognized text. The training data consists of the reference text training data corresponding to the training audio, the recognition text training data whose content is consistent with the reference text training data, and the new recognition text training data.

5. The method according to claim 4, characterized in that, The acquisition of training samples also includes: Based on the reference text training data corresponding to the training audio, recognition text training data that does not match the reference text training data is obtained; wherein, the reference text training data corresponding to the training audio and the recognition text training data that does not match the reference text training data constitute the second training sample.

6. A text reading detection method, characterized in that, include: Obtain the audio of the user reading aloud, and the reference text corresponding to the audio of the user reading aloud; Based on the audio and the reference text, obtain the matching degree between the audio and the reference text; Identify the audio and obtain the corresponding identified text. The similarity information between the identified text and the reference text is obtained by the text similarity recognition method according to any one of claims 1 to 5; Based on the matching degree between the audio and the reference text, and the similarity information between the identified text and the reference text, it is determined whether the audio read by the user meets the reading requirements of the reference text.

7. The method according to claim 6, characterized in that, The step of obtaining the matching degree between the audio and the reference text based on the audio and the reference text includes: The reference text and its corresponding position encoding information are input into the decoding module to obtain a text content matrix. The audio is input into the encoding module to obtain the audio feature matrix; The decoding module calculates the correlation between the text content matrix and the audio feature matrix to obtain a reference text weight matrix, which represents the matching degree between the audio and the reference text.

8. The method according to claim 6 or 7, characterized in that, The step of detecting whether the audio read by the user meets the reading requirements of the reference text based on the matching degree between the audio and the reference text, and the similarity information between the identified text and the reference text, includes: The matching degree between the audio and the reference text, and the similarity information between the identified text and the reference text are respectively input into a parallel convolutional neural network to obtain the first convolutional feature corresponding to the matching degree and the second convolutional feature corresponding to the similarity information. The first convolutional feature is obtained by processing the first pooling layer, and the second convolutional feature is obtained by processing the second pooling layer. The first pooling feature and the second pooling feature are input into a fully connected layer to output a prediction result of whether the audio read by the user meets the reading requirements of the reference text.

9. A text similarity recognition device, characterized in that, include: The first data acquisition module is used to acquire the audio of the user reading aloud, and the reference text corresponding to the audio of the user reading aloud; The first text recognition module is used to recognize the audio and obtain the recognized text corresponding to the audio. The first similarity recognition module is used to extract features from the reference text and the recognition text respectively, to obtain the letter sequence feature information corresponding to the reference text and the letter sequence feature information corresponding to the recognition text, and to perform text similarity recognition based on the letter sequence feature information to obtain the similarity information between the reference text and the recognition text.

10. A text reading detection device, characterized in that, include: The second data acquisition module is used to acquire the audio of the user reading aloud, and the reference text corresponding to the audio of the user reading aloud; The matching degree detection module is used to obtain the matching degree between the audio and the reference text based on the audio and the reference text; The second text recognition module is used to recognize the audio and obtain the recognized text corresponding to the audio. The second similarity recognition module is used to obtain similarity information between the recognized text and the reference text, wherein the similarity information is obtained by the second similarity recognition module through the text similarity recognition method according to any one of claims 1 to 5; The reading detection module is used to detect whether the audio read by the user meets the reading requirements of the reference text based on the matching degree between the audio and the reference text, as well as the similarity information between the identified text and the reference text.

11. An electronic device, characterized in that, include: A memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, wherein, when executing the one or more computer programs, the processor causes the electronic device to implement the text similarity recognition method as described in any one of claims 1 to 5, or the text reading detection method as described in any one of claims 6 to 8.

Images