Low-consumption learning obstacle intelligent screening method and system based on multi-modal data

Abstract

The invention discloses a low-consumption learning obstacle intelligent screening method and system based on multi-modal data, and relates to the technical field of data analysis. The method comprises the following steps: detecting whether a face image is abnormal or not by using a Haar wavelet transform-based multi-level sparse coding image matching model and a reference abnormal face image; de-noising the voice signal to be detected by using a differentiated voice de-noising model based on the combination of Wiener filtering and Transform; recognizing the de-noised voice signal to be detected by using a result evaluation type voice recognition model based on the combination of HMM and Transform; obtaining a student language expression abnormity evaluation result; identifying the answer sheet image of the student by using a definition multi-dimensional analysis type character identification model based on the combination of a CNN (Convolutional Neural Network) and a Transform; obtaining a student writing expression abnormity evaluation result; and judging whether the students have learning obstacles or not according to the plurality of results. According to the method, the data of multiple modes are fully utilized, so that learning obstacle screening with relatively high precision and relatively low consumption is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a low-cost intelligent screening method and system for learning disabilities based on multimodal data. Background Art

[0002] With the development of the times and the progress of society, students' education issues have received increasing attention from all walks of life. However, there are still many students with certain learning disabilities, which not only seriously affects their learning quality, but also easily leads to more psychological problems.

[0003] Although some artificial intelligence technologies have been applied to the screening of students with learning disabilities, these technologies often only use data from a single modality to obtain screening results, resulting in low accuracy of the screening results, which reduces the practical application value of intelligent screening for learning disabilities. At the same time, the high consumption of computing resources further limits the wide and in-depth application of intelligent screening for learning disabilities.

[0004] Therefore, it is of great value and significance to make full use of data from multiple modalities to achieve low-cost learning disability screening. Summary of the invention

[0005] In order to overcome the above problems or at least partially solve the above problems, the present invention provides a low-cost intelligent screening method and system for learning disabilities based on multimodal data, which makes full use of data from multiple modalities to achieve high-precision and low-cost learning disability screening.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: In a first aspect, the present invention provides a low-cost intelligent screening method for learning disabilities based on multimodal data, comprising the following steps: According to the preset extraction time period, collect the face images of students during the learning process; Using the multi-level sparse coding image matching model based on Haar wavelet transform and the preset benchmark abnormal face image, each face image is tested for abnormality and the face detection result is generated; When the student finishes learning, the student's question and answer voice signal is obtained and used as the voice signal to be detected; The discriminative speech denoising model based on the combination of Wiener filter and Transformer is used to denoise the speech signal to be detected, and the denoised speech signal to be detected is obtained; Using the result-evaluation speech recognition model based on the combination of HMM and Transformer, the denoised speech signal to be detected is recognized, and the speech signal recognition result is obtained and sent to the corresponding evaluator; Obtain the assessment results of students’ language expression abnormality from the assessor; When the student completes the voice question and answer, obtain the student answer sheet image; Using the clarity multi-dimensional analysis text recognition model based on the combination of CNN and Transformer, the student answer sheet images are recognized, and the answer sheet text recognition results are obtained and sent to the corresponding evaluators; Obtain the assessment results of students’ written expression abnormalities from the assessors; Based on the face detection results, the student's language expression abnormality assessment results and the student's writing expression abnormality assessment results, it is judged whether the student has learning disabilities and the final student learning disability screening results are generated.

[0007] First, the present invention proposes a multi-level sparse coding image matching model based on Haar wavelet transform to match face images; the model performs wavelet transform on the extracted face images and the benchmark abnormal face images, performs sparse coding and matching on multiple image levels, and significantly improves the accuracy of matching. Secondly, the present invention proposes a discriminative speech denoising model based on the combination of Wiener filter and Transformer to denoise the speech signal; on the basis of equal division detection of the speech signal, the model selectively uses Wiener filter or Transformer for denoising each segment of the equally divided speech signal, which not only ensures the quality of speech denoising, but also significantly reduces the consumption of computing resources. Thirdly, the present invention proposes a result evaluation speech recognition model based on the combination of HMM and Transformer to recognize the speech signal; the model first uses a simple HMM model to recognize the speech signal, and evaluates the recognition result. If the recognition result does not contain easily confused words, the recognition result is directly output; if the speech recognition result contains easily confused words, the Transformer model is used to recognize the speech signal. This speech recognition method not only ensures the accuracy of speech recognition, but also significantly reduces the consumption of computing resources. Finally, the present invention proposes a clarity multi-dimensional analysis text recognition model based on the combination of CNN and Transformer to recognize text; the model first uses a variety of methods to analyze the clarity of the image. If the evaluation result of each method is that the image clarity is high, the CNN model is directly used to recognize the text; otherwise, the Transformer model is used to recognize the text. This text recognition method can not only ensure the accuracy of text recognition, but also significantly reduce the consumption of computing resources.

[0008] In summary, the present invention makes full use of data from multiple modalities to achieve high-precision and low-cost learning disability screening, and has good practical application value.

[0009] Based on the first aspect, further, the low-cost intelligent screening method for learning disabilities based on multimodal data further includes the following steps: The denoised speech signal to be detected is encoded using a reconstruction detection speech coding model based on multiple loss functions to obtain a high-quality denoised speech signal encoding result.

[0010] Based on the first aspect, further, the method for encoding the denoised speech signal to be detected using the reconstruction detection speech coding model based on multiple loss functions comprises the following steps: The de-noised speech signal to be detected is encoded by using an automatic encoder to obtain an encoding result; the loss function is used to constrain the encoding process; The encoding result is reconstructed into a speech signal, and the distortion of the reconstructed speech signal is detected; if the distortion of the reconstructed speech signal is less than a preset distortion threshold, the encoding result is used as the encoding result of the high-quality denoised speech signal to be detected; otherwise, the denoised speech signal to be detected is continuously encoded using the automatic encoder, and a new loss function is used for constraint to obtain a new encoding result; the new encoding result is reconstructed into a new speech signal, and the distortion of the reconstructed new speech signal is detected; if the distortion of the new speech signal is less than a preset distortion threshold, the encoding result is used as the encoding result of the high-quality denoised speech signal to be detected; otherwise, the above steps are continued until the encoding result of the high-quality denoised speech signal to be detected is obtained.

[0011] Based on the first aspect, further, the low-cost intelligent screening method for learning disabilities based on multimodal data further includes the following steps: Obtain and upload student identity information, facial images, student answer sheet images, high-quality denoised speech signal encoding results to be tested, and student learning disability screening results to the blockchain for on-chain storage.

[0012] Based on the first aspect, further, the method for detecting whether each face image has an abnormality by using the multi-level sparse coding image matching model based on Haar wavelet transform and a preset reference abnormal face image comprises the following steps: Perform Haar wavelet transform on the face image and the benchmark abnormal face image; Sparse coding and matching are performed on each level of the image after wavelet decomposition of the face image and the reference abnormal face image to obtain the matching degree of each level of the image; If the matching degree of each level of images is greater than the preset matching degree threshold, the corresponding face image is deemed to be abnormal; otherwise, the corresponding face image is deemed to be normal.

[0013] Based on the first aspect, further, the method for denoising the speech signal to be detected by using the discriminative speech denoising model based on the combination of Wiener filtering and Transformer includes the following steps: The speech signal to be detected is processed in equal parts to obtain a plurality of equally divided speech signals; Performing peak signal-to-noise ratio detection on each equally divided speech signal to obtain the peak signal-to-noise ratio of the corresponding equally divided speech signal; If the peak signal-to-noise ratio of the equally divided speech signal is higher than the preset peak signal-to-noise ratio threshold, the Wiener filter model is used to perform speech denoising on the corresponding equally divided speech signal; otherwise, the Transformer model is used to perform speech denoising on the corresponding equally divided speech signal.

[0014] Based on the first aspect, further, the method for recognizing the denoised speech signal to be detected by using the result evaluation speech recognition model based on the combination of HMM and Transformer includes the following steps: Use the HMM model to recognize the denoised speech signal to be detected and generate a recognition result; If the recognition result does not contain the preset easily confused words, the recognition result is directly output as the final speech signal recognition result; otherwise, the Transformer model is used to recognize the denoised speech signal to be detected to obtain the final speech signal recognition result.

[0015] Based on the first aspect, further, the method for recognizing student answer sheet images using the clarity multi-dimensional analysis text recognition model based on the combination of CNN and Transformer includes the following steps: Analyze the clarity of the student answer sheet image using multiple image clarity analysis methods to generate multiple corresponding image clarity analysis results; If all image clarity analysis results are greater than the preset image clarity threshold, the CNN model is used to recognize the text to obtain the answer sheet text recognition result; otherwise, the Transformer model is used to recognize the text to obtain the final answer sheet text recognition result.

[0016] Based on the first aspect, further, the above-mentioned image clarity analysis method includes Brenner gradient method, Tenegrad gradient method, Laplace gradient method, variance method, and quantity gradient method.

[0017] In a second aspect, the present invention provides a low-cost intelligent screening system for learning disabilities based on multimodal data, including a face acquisition module, a face anomaly detection module, a voice acquisition module, a voice detection module, a voice recognition module, a language evaluation acquisition module, a drawing acquisition module, a text recognition module, a writing evaluation acquisition module and a learning disability judgment module, wherein: The face collection module is used to collect face images of students during their learning process according to a preset extraction time period; The face anomaly detection module is used to detect whether each face image has an abnormality by using a multi-level sparse coding image matching model based on Haar wavelet transform and a preset reference abnormal face image, and generate a face detection result; The voice acquisition module is used to obtain the student's question and answer voice signal as the voice signal to be detected after the student finishes learning; The speech detection module is used to denoise the speech signal to be detected by using a discriminative speech denoising model based on the combination of Wiener filtering and Transformer to obtain the denoised speech signal to be detected; The speech recognition module is used to recognize the denoised speech signal to be detected by using a result-evaluated speech recognition model based on a combination of HMM and Transformer, and obtain and send the speech signal recognition result to the corresponding evaluator; The language assessment acquisition module is used to obtain the assessment results of abnormal language expression of students from the assessor; The drawing acquisition module is used to obtain the student answer sheet image after the student completes the voice question and answer; The text recognition module is used to recognize the student answer sheet image using a clarity multi-dimensional analysis text recognition model based on a combination of CNN and Transformer, and obtain and send the answer sheet text recognition results to the corresponding evaluator; A writing assessment acquisition module is used to obtain the assessment results of abnormal writing expressions of students from the assessors; The learning disability judgment module is used to judge whether a student has a learning disability based on the face detection results, the student's language expression abnormality assessment results and the student's writing expression abnormality assessment results, and generate the final student learning disability screening results.

[0018] The present invention has at least the following advantages or beneficial effects: 1. The present invention proposes a multi-level sparse coding image matching model based on Haar wavelet transform to match face images; the model performs wavelet transform on the extracted face images and the benchmark abnormal face images, performs sparse coding and matching at multiple image levels, and significantly improves the matching accuracy.

[0019] 2. The present invention proposes a discriminative speech denoising model based on the combination of Wiener filtering and Transformer to denoise speech signals. Based on the equal-division detection of speech signals, the model selectively uses Wiener filtering or Transformer to denoise each equally divided speech signal, which not only ensures the quality of speech denoising, but also significantly reduces the consumption of computing resources.

[0020] 3. The present invention proposes a result-evaluation speech recognition model based on the combination of HMM and Transformer to recognize speech signals; the model first uses a simple HMM model to recognize speech signals and evaluates the recognition results. If the recognition results do not contain easily confused words, the recognition results are directly output; if the speech recognition results contain easily confused words, the Transformer model is used to recognize the speech signal. This speech recognition method not only ensures the accuracy of speech recognition, but also significantly reduces the consumption of computing resources.

[0021] 4. The present invention proposes a reconstruction detection speech coding model based on multiple loss functions to encode speech signals; the model is based on a variety of different loss functions and selects high-quality coding results by means of reconstruction detection, which significantly improves the quality of speech coding.

[0022] 5. The present invention proposes a multi-dimensional analysis text recognition model based on the combination of CNN and Transformer to recognize text. The model first uses multiple methods to analyze the clarity of the image. If the evaluation result of each method is that the image clarity is high, the CNN model is directly used to recognize the text; otherwise, the Transformer model is used to recognize the text. This text recognition method can not only ensure the accuracy of text recognition, but also significantly reduce the consumption of computing resources.

[0023] 6. The present invention utilizes blockchain technology to store the student's identity information, the extracted facial image of the student, the answer sheet image, the encoding result of the voice signal to be detected after high-quality denoising, and the learning disability screening result on the chain, ensuring the security of the system.

[0024] 7. The present invention makes full use of data from multiple modalities to achieve high-precision, low-cost screening for learning disabilities, and has good practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 This is a flow chart of a low-cost intelligent screening method for learning disabilities based on multimodal data according to an embodiment of the present invention; Figure 2 This is a principle block diagram of a low-power intelligent screening system for learning disabilities based on multimodal data according to an embodiment of the present invention; Figure 3 A structural block diagram of an electronic device provided by an embodiment of the present invention.

[0027] Explanation of the accompanying drawings: 100, face acquisition module; 200, face anomaly detection module; 300, voice acquisition module; 400, voice detection module; 500, voice recognition module; 600, language assessment acquisition module; 700, drawing acquisition module; 800, text recognition module; 900, writing assessment acquisition module; 1000, learning disability judgment module; 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0031] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0032] In the description of the embodiments of the present invention, "plurality" means at least 2.

[0033] Example like Figure 1 As shown, in a first aspect, an embodiment of the present invention provides a low-cost intelligent screening method for learning disabilities based on multimodal data, comprising the following steps: S1. According to a preset extraction time period, facial images of students in the learning process are collected; a student studies a specific learning content under the guidance of a tester. During the learning process, the image acquisition device extracts the facial images of the student at fixed intervals.

[0034] S2, using a multi-level sparse coding image matching model based on Haar wavelet transform and a preset benchmark abnormal face image, detecting whether each face image has an abnormality and generating a face detection result; Furthermore, it includes: performing Haar wavelet transform on the face image and the benchmark abnormal face image; sparse coding and matching each level of image after wavelet decomposition of the face image and the benchmark abnormal face image to obtain the matching degree of each level of image; if the matching degree of each level of image is greater than a preset matching degree threshold, it is determined that the corresponding face image is abnormal; otherwise, it is determined that the corresponding face image is normal.

[0035] In some embodiments of the present invention, a multi-level sparse coding image matching model based on Haar wavelet transform is used to detect whether each face image has abnormalities. If most face images have abnormalities, it is determined that the student's facial expression has abnormalities. The above-mentioned benchmark abnormal face image refers to a typical face image of a student with learning disabilities.

[0036] S3. When the student finishes learning, the student's question and answer voice signal is obtained and used as the voice signal to be detected; S4, using a discriminative speech denoising model based on the combination of Wiener filtering and Transformer to denoise the speech signal to be detected, and obtaining a denoised speech signal to be detected; Furthermore, it includes: dividing the speech signal to be detected into equal parts to obtain multiple equally divided speech signals; performing peak signal-to-noise ratio detection on each equally divided speech signal to obtain the peak signal-to-noise ratio of the corresponding equally divided speech signal; if the peak signal-to-noise ratio of the equally divided speech signal is higher than a preset peak signal-to-noise ratio threshold, using the Wiener filter model to perform speech denoising on the corresponding equally divided speech signal; otherwise, using the Transformer model to perform speech denoising on the corresponding equally divided speech signal.

[0037] Furthermore, it also includes: using a reconstruction detection speech coding model based on multiple loss functions to encode the denoised speech signal to be detected, and obtaining a high-quality denoised speech signal coding result. Specifically, it includes: using an automatic encoder to encode the denoised speech signal to be detected to obtain a coding result; using a loss function to constrain during the coding process; reconstructing the coding result into a speech signal, and performing distortion detection on the reconstructed speech signal; if the distortion of the reconstructed speech signal is less than a preset distortion threshold, then using this coding result as a high-quality denoised speech signal coding result; otherwise, continuing to use the automatic encoder to encode the denoised speech signal to be detected, using a new loss function to constrain, to obtain a new coding result; reconstructing the new coding result into a new speech signal, and performing distortion detection on the reconstructed new speech signal; if the distortion of the new speech signal is less than the preset distortion threshold, then using this coding result as a high-quality denoised speech signal coding result; otherwise, continuing the above steps until a high-quality denoised speech signal coding result is obtained.

[0038] In some embodiments of the present invention, the denoised speech signal to be detected is encoded using an automatic encoder (a certain loss function is used to constrain the process) to obtain an encoding result, and the encoding result is reconstructed into a speech signal. If the reconstructed speech signal does not have a large distortion, the encoding result is used as a high-quality encoding result of the denoised speech signal to be detected. If the reconstructed speech signal has a large distortion, the denoised speech signal to be detected is further encoded using an automatic encoder (a new loss function is used to constrain the process) to obtain an encoding result, and the encoding result is reconstructed into a speech signal. If the reconstructed speech signal has a large distortion, the encoding result is used as a high-quality encoding result of the denoised speech signal to be detected. If the reconstructed speech signal has a large distortion, the above process is continued until a high-quality encoding result of the denoised speech signal to be detected is obtained.

[0039] S5. Using the result evaluation speech recognition model based on the combination of HMM and Transformer, the denoised speech signal to be detected is recognized, and the speech signal recognition result is obtained and sent to the corresponding evaluator; Furthermore, it includes: using the HMM model to recognize the denoised speech signal to be detected and generate a recognition result; if the recognition result does not contain preset easily confused words, the recognition result is directly output as the final speech signal recognition result; otherwise, the Transformer model is used to recognize the denoised speech signal to be detected and obtain the final speech signal recognition result.

[0040] In some embodiments of the present invention, for the speech signal to be detected after denoising, the HMM model is used to recognize the speech signal. If the recognition result does not contain easily confused words, the recognition result is directly output; if the recognition result contains easily confused words (for example, the word "poet", which has a highly similar pronunciation to words such as "ten people" and is prone to recognition errors), the Transformer model is reused to recognize the speech signal to obtain the final recognition result.

[0041] S6. Obtain the assessment results of the student's language expression abnormality from the evaluator; professional evaluators evaluate the recognition results of the voice signal to be detected after denoising to determine whether the student has language expression abnormalities, providing effective support for subsequent judgments on students' learning disabilities.

[0042] S7. After the student completes the voice question and answer session, obtain the student's answer sheet image; after the student completes the voice question answering session, the student answers the relevant questions on the answer sheet in handwriting as required under the guidance of the tester. After the student writes the answer, use the image acquisition device to take a photo of the student's answer sheet to obtain the answer sheet image.

[0043] S8. Use the clarity multi-dimensional analysis text recognition model based on the combination of CNN and Transformer to recognize the student answer sheet image, obtain the answer sheet text recognition result and send it to the corresponding evaluator; Furthermore, it includes: using multiple image clarity analysis methods to analyze the clarity of the student answer sheet image, and generating multiple corresponding image clarity analysis results; if all the image clarity analysis results are greater than the preset image clarity threshold, then using the CNN model to recognize the text, and obtaining the answer sheet text recognition result; otherwise, using the Transformer model to recognize the text, and obtaining the final answer sheet text recognition result. The above-mentioned image clarity analysis methods include Brenner gradient method, Tenegrad gradient method, Laplace gradient method, variance method, and quantity gradient method.

[0044] S9. Obtain the assessment results of the student's abnormal writing expression from the assessor; the professional staff evaluates the text recognition results on the answer sheet to determine whether the student has abnormal writing expression.

[0045] S10. Determine whether the student has learning disabilities based on the face detection results, the student's language expression abnormality assessment results, and the student's writing expression abnormality assessment results, and generate the final student learning disability screening results.

[0046] In some embodiments of the present invention, if the student does not have abnormal facial expressions, abnormal language expression, or abnormal writing expression, it is determined that the student has no obvious learning disabilities; if the student has one of abnormal facial expressions, abnormal language expression, or abnormal writing expression, it is determined that the student has general learning disabilities; if the student has two or more of abnormal facial expressions, abnormal language expression, or abnormal writing expression, it is determined that the student has severe learning disabilities.

[0047] Furthermore, the method also includes: obtaining and uploading student identity information, face images, student answer sheet images, encoding results of high-quality denoised voice signals to be detected, and student learning disability screening results to the blockchain for on-chain storage. Note: The number of images is relatively small, so no encoding is done; the speech is generally longer, so it is encoded.

[0048] First, the present invention proposes a multi-level sparse coding image matching model based on Haar wavelet transform to match face images; the model performs wavelet transform on the extracted face images and the benchmark abnormal face images, performs sparse coding and matching on multiple image levels, and significantly improves the accuracy of matching. Secondly, the present invention proposes a discriminative speech denoising model based on the combination of Wiener filter and Transformer to denoise the speech signal; on the basis of equal division detection of the speech signal, the model selectively uses Wiener filter or Transformer for denoising each segment of the equally divided speech signal, which not only ensures the quality of speech denoising, but also significantly reduces the consumption of computing resources. Thirdly, the present invention proposes a result evaluation speech recognition model based on the combination of HMM and Transformer to recognize the speech signal; the model first uses a simple HMM model to recognize the speech signal, and evaluates the recognition result. If the recognition result does not contain easily confused words, the recognition result is directly output; if the speech recognition result contains easily confused words, the Transformer model is used to recognize the speech signal. This speech recognition method not only ensures the accuracy of speech recognition, but also significantly reduces the consumption of computing resources. Next, the present invention proposes a reconstruction detection speech coding model based on multiple loss functions to encode speech signals; the model selects high-quality coding results in a reconstruction detection manner based on a variety of unused loss functions, which significantly improves the quality of speech coding. Subsequently, the present invention proposes a clarity multi-dimensional analysis text recognition model based on the combination of CNN and Transformer to recognize text; the model first uses a variety of methods to analyze the clarity of the image. If the evaluation result of each method is that the image clarity is high, the CNN model is directly used to recognize the text; otherwise, the Transformer model is used to recognize the text. This text recognition method can not only ensure the accuracy of text recognition, but also significantly reduce the consumption of computing resources. Finally, the present invention uses blockchain technology to store the student's identity information, the extracted face image of the student, the answer sheet image, the encoding result of the speech signal to be detected after high-quality denoising, and the learning disability screening result on the chain, ensuring the security of the system. In summary, the present invention makes full use of data from multiple modalities to achieve high-precision and low-consumption learning disability screening, and has good practical application value.

[0049] like Figure 2As shown, in the second aspect, an embodiment of the present invention provides a low-cost intelligent screening system for learning disabilities based on multimodal data, including a face acquisition module 100, a face anomaly detection module 200, a voice acquisition module 300, a voice detection module 400, a voice recognition module 500, a language evaluation acquisition module 600, a drawing acquisition module 700, a text recognition module 800, a writing evaluation acquisition module 900 and a learning disability judgment module 1000, wherein: The face collection module 100 is used to collect face images of students during their learning process according to a preset extraction time period; The face anomaly detection module 200 is used to detect whether each face image has an abnormality by using a multi-level sparse coding image matching model based on Haar wavelet transform and a preset reference abnormal face image, and generate a face detection result; The voice collection module 300 is used to obtain the student's question and answer voice signal as the voice signal to be detected after the student finishes learning; The speech detection module 400 is used to denoise the speech signal to be detected by using a discriminative speech denoising model based on a combination of Wiener filtering and Transformer to obtain a denoised speech signal to be detected; The speech recognition module 500 is used to recognize the denoised speech signal to be detected by using a result evaluation speech recognition model based on a combination of HMM and Transformer, and obtain and send the speech signal recognition result to the corresponding evaluator; The language evaluation acquisition module 600 is used to obtain the evaluation results of abnormal language expression of students by the evaluator; The drawing acquisition module 700 is used to obtain the student answer sheet image after the student completes the voice question and answer; The text recognition module 800 is used to recognize the student answer sheet image by using a clarity multi-dimensional analysis text recognition model based on a combination of CNN and Transformer, and obtain and send the answer sheet text recognition result to the corresponding evaluator; The writing evaluation acquisition module 900 is used to obtain the evaluation results of abnormal writing expression of students by the evaluator; The learning disability judgment module 1000 is used to judge whether a student has a learning disability based on the face detection results, the student's language expression abnormality assessment results and the student's writing expression abnormality assessment results, and generate the final student learning disability screening results.

[0050] This system, through the cooperation of multiple modules such as face acquisition module 100, face anomaly detection module 200, voice acquisition module 300, voice detection module 400, voice recognition module 500, language assessment acquisition module 600, drawing acquisition module 700, text recognition module 800, writing assessment acquisition module 900 and learning disability judgment module 1000, makes full use of data of multiple modalities to achieve high-precision and low-cost learning disability screening, and has good practical application value. First, the present invention proposes a multi-level sparse coding image matching model based on Haar wavelet transform to match face images; the model performs wavelet transform on the extracted face image and the benchmark abnormal face image, performs sparse coding and matching at multiple image levels, and significantly improves the matching accuracy. Secondly, the present invention proposes a discriminative speech denoising model based on the combination of Wiener filter and Transformer to denoise the speech signal; on the basis of equally divided detection of the speech signal, the model selectively uses Wiener filter or Transformer to denoise each equally divided speech signal, which not only ensures the quality of speech denoising, but also significantly reduces the consumption of computing resources. Thirdly, the present invention proposes a result evaluation speech recognition model based on the combination of HMM and Transformer to recognize the speech signal; the model first uses a simple HMM model to recognize the speech signal, and evaluates the recognition result. If the recognition result does not contain easily confused words, the recognition result is directly output; if the speech recognition result contains easily confused words, the Transformer model is used to recognize the speech signal. This speech recognition method not only ensures the accuracy of speech recognition, but also significantly reduces the consumption of computing resources. Finally, the present invention proposes a clarity multi-dimensional analysis text recognition model based on the combination of CNN and Transformer to recognize text; the model first uses multiple methods to analyze the clarity of the image. If the evaluation result of each method is that the image clarity is high, the CNN model is directly used to recognize the text; otherwise, the Transformer model is used to recognize the text. This text recognition method can not only ensure the accuracy of text recognition, but also significantly reduce the consumption of computing resources.

[0051] like Figure 3 As shown, in a third aspect, an embodiment of the present application provides an electronic device, which includes a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, any method in the first aspect described above is implemented.

[0052] The memory 101, the processor 102 and the communication interface 103 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0053] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0054] The processor 102 may be an integrated circuit chip with signal processing capability. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0055] In the embodiments provided in the present application, it should be understood that the disclosed method and system can also be implemented in other ways. The method and system embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the method and system, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0056] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0057] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by the processor 102, the method as in any one of the first aspects described above is implemented. If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0058] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0059] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A low-cost intelligent screening method for learning disabilities based on multimodal data, characterized in that: The following steps are involved: According to the preset extraction time period, collect the face images of students during the learning process; Using the multi-level sparse coding image matching model based on Haar wavelet transform and the preset benchmark abnormal face image, each face image is tested for abnormality and the face detection result is generated; When the student finishes learning, the student's question and answer voice signal is obtained and used as the voice signal to be detected; The discriminative speech denoising model based on the combination of Wiener filtering and Transformer is used to denoise the speech signal to be detected, and the denoised speech signal to be detected is obtained; Using the result-evaluation speech recognition model based on the combination of HMM and Transformer, the denoised speech signal to be detected is recognized, and the speech signal recognition result is obtained and sent to the corresponding evaluator; Obtain the assessment results of students’ language expression abnormality from the assessor; When the student completes the voice question and answer, obtain the student answer sheet image; Using the clarity multi-dimensional analysis text recognition model based on the combination of CNN and Transformer, the student answer sheet images are recognized, and the answer sheet text recognition results are obtained and sent to the corresponding evaluators; Obtain the assessment results of students’ written expression abnormalities from the assessors; Based on the face detection results, the student's language expression abnormality assessment results and the student's writing expression abnormality assessment results, it is judged whether the student has learning disabilities and the final student learning disability screening results are generated.

2. According to claim 1, a low-cost intelligent screening method for learning disabilities based on multimodal data is characterized in that: The following steps are also included: The denoised speech signal to be detected is encoded using a reconstruction detection speech coding model based on multiple loss functions to obtain a high-quality denoised speech signal encoding result.

3. According to claim 2, a low-cost intelligent screening method for learning disabilities based on multimodal data is characterized in that: The method for encoding the denoised speech signal to be detected by using a reconstruction detection speech coding model based on multiple loss functions comprises the following steps: The de-noised speech signal to be detected is encoded by using an automatic encoder to obtain an encoding result; the loss function is used to constrain the encoding process; The encoding result is reconstructed into a speech signal, and the distortion of the reconstructed speech signal is detected; if the distortion of the reconstructed speech signal is less than a preset distortion threshold, the encoding result is used as the encoding result of the high-quality denoised speech signal to be detected; otherwise, the denoised speech signal to be detected is continuously encoded using the automatic encoder, and a new loss function is used for constraint to obtain a new encoding result; the new encoding result is reconstructed into a new speech signal, and the distortion of the reconstructed new speech signal is detected; if the distortion of the new speech signal is less than a preset distortion threshold, the encoding result is used as the encoding result of the high-quality denoised speech signal to be detected; otherwise, the above steps are continued until the encoding result of the high-quality denoised speech signal to be detected is obtained.

4. According to claim 2, a low-cost intelligent screening method for learning disabilities based on multimodal data is characterized in that: The following steps are also included: Obtain and upload student identity information, facial images, student answer sheet images, high-quality denoised speech signal encoding results to be tested, and student learning disability screening results to the blockchain for on-chain storage.

5. According to claim 1, a low-cost intelligent screening method for learning disabilities based on multimodal data is characterized in that: The method for detecting whether each face image has an abnormality by using a multi-level sparse coding image matching model based on Haar wavelet transform and a preset reference abnormal face image comprises the following steps: Perform Haar wavelet transform on the face image and the benchmark abnormal face image; Sparse coding and matching are performed on each level of the image after wavelet decomposition of the face image and the reference abnormal face image to obtain the matching degree of each level of the image; If the matching degree of each level of images is greater than the preset matching degree threshold, the corresponding face image is deemed to be abnormal; otherwise, the corresponding face image is deemed to be normal.

6. The low-cost intelligent screening method for learning disabilities based on multimodal data according to claim 1, characterized in that: The method for denoising the speech signal to be detected by using a distinguishable speech denoising model based on a combination of Wiener filtering and Transformer comprises the following steps: The speech signal to be detected is processed in equal parts to obtain a plurality of equally divided speech signals; Performing peak signal-to-noise ratio detection on each equally divided speech signal to obtain the peak signal-to-noise ratio of the corresponding equally divided speech signal; If the peak signal-to-noise ratio of the equally divided speech signal is higher than the preset peak signal-to-noise ratio threshold, the Wiener filter model is used to perform speech denoising on the corresponding equally divided speech signal; otherwise, the Transformer model is used to perform speech denoising on the corresponding equally divided speech signal.

7. The low-cost intelligent screening method for learning disabilities based on multimodal data according to claim 1, characterized in that: The method for recognizing the denoised speech signal to be detected by using the result evaluation speech recognition model based on the combination of HMM and Transformer comprises the following steps: Use the HMM model to recognize the denoised speech signal to be detected and generate a recognition result; If the recognition result does not contain the preset easily confused words, the recognition result is directly output as the final speech signal recognition result; otherwise, the Transformer model is used to recognize the denoised speech signal to be detected to obtain the final speech signal recognition result.

8. The low-cost intelligent screening method for learning disabilities based on multimodal data according to claim 1, characterized in that: The method for identifying student answer sheet images using a clarity multi-dimensional analysis text recognition model based on a combination of CNN and Transformer comprises the following steps: Analyze the clarity of the student answer sheet image using multiple image clarity analysis methods to generate multiple corresponding image clarity analysis results; If all the image clarity analysis results are greater than the preset image clarity threshold, the CNN model is used to recognize the text to obtain the answer sheet text recognition result; otherwise, the Transformer model is used to recognize the text to obtain the final answer sheet text recognition result.

9. The low-cost intelligent screening method for learning disabilities based on multimodal data according to claim 8, characterized in that: The image clarity analysis methods include Brenner gradient method, Tenegrad gradient method, Laplace gradient method, variance method and quantity gradient method.

10. A low-cost intelligent screening system for learning disabilities based on multimodal data, characterized in that: It includes a face acquisition module, a face anomaly detection module, a voice acquisition module, a voice detection module, a voice recognition module, a language assessment acquisition module, a drawing acquisition module, a text recognition module, a writing assessment acquisition module and a learning disability judgment module, among which: The face collection module is used to collect face images of students during their learning process according to a preset extraction time period; The face anomaly detection module is used to detect whether each face image has an abnormality by using a multi-level sparse coding image matching model based on Haar wavelet transform and a preset reference abnormal face image, and generate a face detection result; The voice acquisition module is used to obtain the student's question and answer voice signal as the voice signal to be detected after the student finishes learning; The speech detection module is used to denoise the speech signal to be detected by using a discriminative speech denoising model based on the combination of Wiener filtering and Transformer to obtain the denoised speech signal to be detected; The speech recognition module is used to recognize the denoised speech signal to be detected by using a result-evaluated speech recognition model based on a combination of HMM and Transformer, and obtain and send the speech signal recognition result to the corresponding evaluator; The language assessment acquisition module is used to obtain the assessment results of abnormal language expression of students from the assessor; The drawing acquisition module is used to obtain the student answer sheet image after the student completes the voice question and answer; The text recognition module is used to recognize the student answer sheet image using a clarity multi-dimensional analysis text recognition model based on a combination of CNN and Transformer, and obtain and send the answer sheet text recognition results to the corresponding evaluator; A writing assessment acquisition module is used to obtain the assessment results of abnormal writing expressions of students from the assessors; The learning disability judgment module is used to judge whether a student has a learning disability based on the face detection results, the student's language expression abnormality assessment results and the student's writing expression abnormality assessment results, and generate the final student learning disability screening results.

Images