Teaching and research management system and method

Abstract

The invention relates to the field of teaching and research management, and particularly discloses a teaching and research management system and method.The teaching and research management method comprises the steps that firstly, an audio file of English listening and speaking practice submitted by a student and a video file of English listening and speaking practice submitted by the student are obtained, then feature extraction and correlation analysis are conducted on the audio file and the video file through the deep learning technology, and finally through a generator, a learning result is obtained; therefore, the workload of the teacher is reduced, the teacher is assisted to pay attention to the personalized guidance of the student, the teaching process is optimized, the overall teaching quality is improved, and the more comprehensive listening and speaking of the student is cultivated.

Description

Technical Field

[0001] The present application relates to the field of teaching and research management, and more specifically, to a teaching and research management system and method. Background Art

[0002] In modern English teaching, although the four basic skills of listening, speaking, reading and writing complement each other, the traditional teaching model often focuses on grammar and writing. Teachers and students spend most of their time memorizing grammar rules and analyzing sentence structures, resulting in significant deficiencies in students' listening, speaking and reading.

[0003] With the rapid development of educational technology, English listening and speaking skills have gradually received more attention. When students finish reciting English after class, they usually publish their audio and video materials to teachers for review. However, faced with a large amount of submitted practice materials, teachers often find it difficult to review them one by one due to the huge workload. This not only increases the burden on teachers, but may also lead to subjectivity and omissions in grading, thus affecting students' learning effects and enthusiasm.

[0004] Therefore, a teaching and research management system and method are desired. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed. The embodiment of this application provides a teaching and research management system and method, which first obtains the audio files of English listening and speaking exercises submitted by students and the video files of English listening and speaking exercises submitted by students, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally generates a personalized English listening and speaking practice analysis report for students through a generator, thereby reducing the workload of teachers, assisting teachers to focus on personalized guidance of students, and then optimizing the teaching process, improving the overall teaching quality, and cultivating students' more comprehensive literacy.

[0006] According to one aspect of the present application, a teaching and research management system is provided, which includes: English teaching and research practice data acquisition module, used to obtain audio files of English listening and speaking practice submitted by students and video files of English listening and speaking practice submitted by students; An English teaching and research practice data processing module is used to extract audio feature vectors of the English listening and speaking practice audio and feature vectors of the English listening and speaking practice mouth target attention of the students from the audio files of the English listening and speaking practice submitted by the students and the video files of the English listening and speaking practice submitted by the students; The practice analysis report generating module is used to generate a personalized English listening and speaking practice analysis report for students based on the audio feature vector of the student's English listening and speaking practice audio and the feature vector of the student's English listening and speaking practice mouth target attention.

[0007] According to another aspect of the present application, a teaching and research management method is provided, which includes: Obtain audio files of English listening and speaking exercises submitted by students and video files of English listening and speaking exercises submitted by students; Extracting the audio feature vector of the English listening and speaking practice audio and the mouth target attention feature vector of the English listening and speaking practice of the student from the audio file of the English listening and speaking practice submitted by the student and the video file of the English listening and speaking practice submitted by the student; Based on the audio feature vector of the student's English listening and speaking practice audio and the feature vector of the student's English listening and speaking practice mouth target attention, a personalized English listening and speaking practice analysis report for the student is generated.

[0008] Compared with the prior art, the present application provides a teaching and research management system and method, which first obtains audio files of English listening and speaking exercises submitted by students and video files of English listening and speaking exercises submitted by students, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally generates a personalized English listening and speaking practice analysis report for students through a generator, thereby reducing the workload of teachers, assisting teachers to focus on personalized guidance of students, and then optimizing the teaching process, improving the overall teaching quality, and cultivating students' more comprehensive literacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 is a block diagram of a teaching and research management system according to an embodiment of the present application.

[0011] Figure 2 It is a block diagram of the English teaching and research exercise data processing module in the teaching and research management system according to an embodiment of the present application.

[0012] Figure 3 It is a block diagram of a student practice target area feature encoding unit in a teaching and research management system according to an embodiment of the present application.

[0013] Figure 4 It is a block diagram of an exercise analysis report generation module in a teaching and research management system according to an embodiment of the present application.

[0014] Figure 5 Flowchart of a teaching and research management method according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0016] Figure 1 : is a block diagram of the teaching and research management system of the present application embodiment. Figure 1 As shown, according to the teaching and research management system 100 of the embodiment of the present application, it includes: an English teaching and research practice data acquisition module 110, which is used to acquire audio files of English listening and speaking practice submitted by students and video files of English listening and speaking practice submitted by students; an English teaching and research practice data processing module 120, which is used to extract audio feature vectors of students' English listening and speaking practice audio and video feature vectors of students' English listening and speaking practice mouth target attention from the audio files of English listening and speaking practice submitted by the students and the video files of English listening and speaking practice submitted by the students; an exercise analysis report generation module 130, which is used to generate a personalized English listening and speaking practice analysis report for students based on the audio feature vectors of the students' English listening and speaking practice audio and the video feature vectors of the students' English listening and speaking practice mouth target attention.

[0017] In the above-mentioned teaching and research management system 100, the English teaching and research exercise data acquisition module 110 is used to obtain the audio files of English listening and speaking exercises submitted by students and the video files of English listening and speaking exercises submitted by students. It should be understood that in today's English teaching, the four basic skills of listening, speaking, reading and writing are interdependent, but traditional teaching methods often focus on grammar and writing, and teachers and students spend a lot of time memorizing grammatical rules and analyzing sentence structures. This approach leads to students' obvious deficiencies in listening, speaking and reading abilities. With the rapid advancement of educational technology, English listening and speaking abilities have begun to receive more and more attention. When students complete after-school exercises, they usually upload their audio and video materials to teachers for review. However, due to the huge number of practice materials submitted, it is difficult for teachers to conduct detailed reviews of each work, which not only increases the workload of teachers, but also may lead to inconsistent and missed scores, thereby affecting students' learning effects and enthusiasm. Therefore, in the technical solution of this application, the audio files of English listening and speaking exercises submitted by students and the video files of English listening and speaking exercises submitted by students are obtained, and combined with deep learning technology, a personalized English listening and speaking exercise analysis report is generated to reduce the workload of teachers and help teachers focus more on personalized guidance for students. This method helps to optimize the teaching process and improve the overall teaching quality, thereby cultivating students' more comprehensive abilities and qualities.

[0018] Specifically, audio files provide teachers with an intuitive assessment tool. By listening to students' pronunciation, intonation, and fluency, teachers can more accurately judge students' speaking ability. This assessment method is not limited to the scores of written tests, but reflects students' language application ability through actual performance. Teachers can find the strengths and weaknesses of students' pronunciation, grammar, and vocabulary usage based on their expressions in audio files, so as to provide targeted feedback to help students improve more effectively. Compared with pure audio, video files can show students' facial language, which is particularly important for evaluating non-verbal communication skills. Obtaining audio and video files of English listening and speaking exercises submitted by students is not only an effective means of evaluating students' language ability, but also an important way to promote students' independent learning and teachers to improve teaching. This method provides a more comprehensive and in-depth perspective on language learning, helping students to continue to grow and improve in real communication.

[0019] In the above-mentioned teaching and research management system 100, the English teaching and research practice data processing module 120 is used to extract the audio feature vector of the student's English listening and speaking practice audio and the feature vector of the student's English listening and speaking practice mouth target attention from the audio file of the English listening and speaking practice submitted by the student and the video file of the English listening and speaking practice submitted by the student. It should be understood that by extracting the feature vector from the audio and video files of the student, the teacher can adjust the teaching strategy in time, formulate corresponding exercises and activities for the weaknesses of the student, thereby improving the overall teaching effect. In addition, this data-based evaluation method can also motivate students to actively learn, and encourage them to pay more attention to their pronunciation and non-verbal expression ability in practice. Such an evaluation method will help to cultivate students' comprehensive language ability and make them more confident and calm in actual communication.

[0020] Figure 2 FIG. 1 is a block diagram of an English teaching and research exercise data processing module in a teaching and research management system according to an embodiment of the present application. Figure 2 As shown, in a specific embodiment of the present application, the English teaching and research practice data processing module 120 includes: a student practice audio file feature extraction unit 121, which is used to perform feature extraction on the audio file of the English listening and speaking practice submitted by the student to obtain the audio feature vector of the student's English listening and speaking practice audio; a student practice video file feature extraction unit 122, which is used to perform feature extraction on the video file of the English listening and speaking practice submitted by the student to obtain the student's English listening and speaking practice mouth target interest area; a student practice target area feature encoding unit 123, which is used to perform feature encoding on the student's English listening and speaking practice mouth target interest area to obtain the student's English listening and speaking practice mouth target attention feature vector.

[0021] It is understandable that in language learning, students may encounter many challenges when speaking, such as inaccurate pronunciation, speaking too fast or too slow, and insufficient emotional expression. By extracting features from audio files, teachers can analyze students' pronunciation patterns and voice characteristics to identify potential problems. For example, if a student's pitch range is too small, it may cause their expression to lack emotion and appeal. After identifying these problems, teachers can provide targeted feedback and guidance to help students improve their speaking skills.

[0022] Furthermore, through feature extraction, teachers can not only evaluate students' language ability, but also help them improve the fluency and naturalness of oral expression in practice. Furthermore, the movement of the mouth is crucial to the accuracy of pronunciation. By analyzing the dynamic changes of the mouth, teachers can understand the mouth shape, tongue position and lip movement of students during pronunciation. This information can help teachers identify students' pronunciation problems on certain specific syllables and provide targeted guidance to improve students' pronunciation accuracy. Through feature coding, teachers can evaluate students' emotional expression and non-verbal feedback in oral practice. This analysis will help students enhance their communication effectiveness when expressing their thoughts and emotions, making them more natural and contagious.

[0023] In a specific embodiment of the present application, the student practice audio file feature extraction unit 121 includes: performing audio vectorization on the audio file of English listening and speaking practice submitted by the student to obtain the student English listening and speaking practice audio input vector; passing the student English listening and speaking practice audio input vector through the student English listening and speaking practice audio deep neural network model to obtain the student English listening and speaking practice audio audio feature vector.

[0024] It should be understood that traditional audio analysis often requires direct processing of complex waveform data, which is not only time-consuming but also computationally expensive. Through vectorization, key features of audio signals, such as frequency, pitch, duration, and volume, can be extracted, reducing the complexity and computational burden of data processing. This method can help teachers quickly evaluate students' pronunciation, intonation, and rhythm, so as to provide feedback more effectively. Specifically, feature vectors can include multiple aspects such as the timbre, speed, and clarity of speech, which are important factors affecting language expression. Through audio vectorization, students' pronunciation and expression abilities in oral practice can be quantified. Among them, audio vectorization is the process of converting audio signals into digital feature vectors, aiming to extract and represent key information of sound. This process usually involves segmenting the audio signal into small time periods and using various feature extraction techniques, such as Mel-frequency cepstral coefficients (MFCC), spectrograms, and spectral features, to convert the sound features of each time period into numerical form.

[0025] Furthermore, traditional audio feature extraction methods usually rely on manually selecting and designing features, such as pitch, frequency, and duration. However, this method is often affected by subjective factors and may not fully capture the complex information in the audio signal. Deep neural networks (DNNs) have the ability to automatically learn features and can automatically extract richer and more meaningful features from the original audio input vector. This self-learning ability allows the model to effectively identify key factors that affect oral expression, such as pronunciation clarity, intonation changes, and fluency of speech speed. Among them, the deep learning model can extract features at different levels through a multi-layer structure to capture subtle changes in audio signals. This multi-level feature extraction method makes the model more robust and flexible when facing audio data from different students, and can effectively deal with different situations such as accents and changes in speech speed, thereby providing more accurate analysis results. This accuracy is crucial for teachers to assess students' language proficiency. Specifically, each layer of the student English listening and speaking practice audio deep neural network model is used to perform convolution processing, mean pooling processing based on the local feature matrix, and nonlinear activation processing on the input data in the forward pass of the layer, so that the last layer of the student English listening and speaking practice audio deep neural network model outputs the student English listening and speaking practice audio audio feature vector, wherein the input of the student English listening and speaking practice audio deep neural network model is the student English listening and speaking practice audio input vector.

[0026] In a specific embodiment of the present application, the student practice video file feature extraction unit 122 includes: passing the video file of the English listening and speaking practice submitted by the student through the student English listening and speaking practice video convolutional neural network to obtain an initial student English listening and speaking practice feature map; passing the initial student English listening and speaking practice feature map through the student English listening and speaking practice target detection network to obtain the student English listening and speaking practice mouth target interest area.

[0027] It should be understood that video data contains rich information. Convolutional neural networks can effectively extract spatial features in video frames, such as facial expressions, which are important elements that affect language communication. Through multi-level convolution and pooling operations, convolutional neural networks can extract higher-level abstract features. The initial feature map is used as input data to provide rich feature information for subsequent deep learning models. This process can not only improve the learning effect of the model, but also improve its robustness in different situations, so that the model can still maintain a high level of analysis accuracy when facing different students, different accents or different speaking styles. Specifically, each layer of the student English listening and speaking practice video convolutional neural network is used to perform the following operations on the input data in the forward pass of the layer: convolution processing is performed on the input data based on the convolution kernel to generate a convolution feature map; global mean pooling processing is performed on the convolution feature map based on the feature matrix to generate a pooled feature map; and non-linear activation is performed on the eigenvalues ​​at each position in the pooled feature map to generate an activated feature addition map; wherein the output of the last layer of the student English listening and speaking practice video convolutional neural network is the initial student English listening and speaking practice feature map, the input of the second layer to the last layer of the student English listening and speaking practice video convolutional neural network is the output of the previous layer, and the input of the student English listening and speaking practice video convolutional neural network is the video file of English listening and speaking practice submitted by the student.

[0028] Furthermore, in oral English practice, the movement of the mouth is crucial to the accuracy of pronunciation. Through the target detection network, the dynamic information of the mouth can be accurately located and extracted. The movements of the mouth, such as opening and closing the mouth, and changes in lip shape, directly affect the clarity of pronunciation and the quality of speech. Therefore, paying attention to the mouth area can help teachers and students deeply analyze the problems in the pronunciation process and provide targeted guidance and improvement measures. The target detection network can extract the specific area of ​​the mouth from the initial feature map for more detailed analysis. By monitoring the area of ​​interest in the mouth, subtle changes in students' pronunciation can be observed, such as the position of the tongue, changes in lip shape, and adjustments to the mouth shape. This refined analysis can not only reveal the specific problems of students in pronunciation, but also help students understand which oral movements will affect their pronunciation, so as to practice better. In this way, by analyzing the dynamic information of the mouth, teachers can more clearly identify the shortcomings of each student in pronunciation and provide targeted suggestions and practice plans. Specifically, the student English listening and speaking practice target detection network is used to process the initial student English listening and speaking practice feature map with the following target detection formula to obtain the student English listening and speaking practice mouth target interest region; Among them, the target detection formula is: ;in, Characteristic diagram for English listening and speaking practice for initial students, is the anchor frame, Mouth target area of ​​interest for students to practice English listening and speaking, Indicates classification, Indicates regression.

[0029] Figure 3 FIG. 1 is a block diagram of a student practice target area feature encoding unit in a teaching and research management system according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the student practice target area feature encoding unit 123 includes: a mouth target attention convolution subunit 1231, which is used to pass the student English listening and speaking practice mouth target interest area through the student English listening and speaking practice mouth target attention mechanism module to obtain the student English listening and speaking practice mouth target attention feature map; the student English listening and speaking practice information feature pooling subunit 1232, which is used to pool the student English listening and speaking practice mouth target attention feature map to obtain the student English listening and speaking practice mouth target attention feature vector.

[0030] It should be understood that the core of the attention mechanism is to be able to effectively focus on the key features in the input data. In English oral practice, the movement and morphological changes of the mouth are important factors in evaluating pronunciation accuracy and fluency. By inputting the mouth target region of interest into the attention mechanism module, the model's attention to mouth movements can be strengthened, thereby extracting more representative features. This attention can significantly improve the effectiveness of feature extraction and help the model more accurately understand and analyze subtle changes in the pronunciation process. The mechanism can dynamically adjust the weights of different regions in the feature map, so that the network pays more attention to the key frames and key postures of mouth movements during training and inference. In this way, the generated mouth target attention feature map can not only more comprehensively reflect the dynamic changes of the mouth, but also reduce the interference of irrelevant information, thereby improving the accuracy of the final evaluation. Specifically, the convolutional coding part of the student English listening and speaking practice mouth target attention mechanism module is used to perform deep convolutional coding on the student English listening and speaking practice mouth target interest area to obtain an initial convolutional feature map; the initial convolutional feature map is input into the spatial attention part of the student English listening and speaking practice mouth target attention mechanism module to obtain a spatial attention map; the spatial attention map is activated by a Softmax function to obtain a spatial attention feature map; the spatial attention feature map and the initial convolutional feature map are multiplied by the position point to obtain the student English listening and speaking practice mouth target attention feature map. More specifically, the initial convolutional feature map is input into the spatial attention part of the student English listening and speaking practice mouth target attention mechanism module to obtain the spatial attention map, including: performing average pooling and maximum pooling along the channel dimension on the initial convolutional feature map to obtain an average feature matrix and a maximum feature matrix; cascading and channel-adjusting the average feature matrix and the maximum feature matrix to obtain a channel feature matrix; using the convolutional layer of the spatial attention feature map to convolutionally encode the channel feature matrix to obtain a spatial attention map.

[0031] Furthermore, by compressing the information in the feature map into a feature vector, the model can significantly reduce the resources required for calculation while maintaining key information. Among them, since pooling is achieved by taking the maximum or average value of the local area of ​​the feature map, this aggregation method can effectively filter out subtle changes and unnecessary information and retain important features. In this way, even in some non-ideal environments, the model can still stably extract useful mouth movement information, thereby improving the accuracy of spoken language evaluation.

[0032] In a specific embodiment of the present application, the student English listening and speaking practice information feature pooling subunit includes: obtaining a student English listening and speaking practice mouth target attention feature map in binary format; using BufferedInputStream to read the student English listening and speaking practice mouth target attention feature map from a file; parsing the read binary data into a student English listening and speaking practice mouth target attention feature map data structure, wherein the student English listening and speaking practice mouth target attention feature map data structure is a multidimensional array representing the attention distribution of the mouth target; performing a maximum pooling operation on the parsed student English listening and speaking practice mouth target attention feature map, wherein the maximum pooling operation is performed by sliding a fixed-size window on the student English listening and speaking practice mouth target attention feature map and taking the maximum value of the elements in the window; converting the pooled result into a student English listening and speaking practice mouth target attention feature vector and saving it to a file.

[0033] It should be understood that BufferedInputStream is a class in the Java standard library that provides a buffering mechanism when reading data. It reduces the number of accesses to the underlying input stream by maintaining a byte array buffer internally, thereby improving reading efficiency. When an application reads data from BufferedInputStream, the data is first read from the underlying input stream to the buffer, and then read from the buffer. This method reduces frequent I / O operations, especially when processing large files or network data, significantly improving performance.

[0034] Part of the deployment code is shown below.

[0035] import java.io.*; import java.nio.ByteBuffer; import java.util.Arrays; public class MaxPoolingDeployment { public static void main(String[] args) { / / Get feature map data String filePath = "path / to / feature_map_data.bin"; try (BufferedInputStream bis = new BufferedInputStream(newFileInputStream(filePath))) { / / Read feature map data byte[] buffer = new byte[4 * 64 * 64]; / / Assume the feature map size is 64x64 and each element is 4 bytes (float) int bytesRead = bis.read(buffer); if (bytesRead != buffer.length) { throw new IOException("Failed to read the entire feature map data."); } / / Parse the read binary data into a float array float[] featureMap = new float[64 * 64]; ByteBuffer.wrap(buffer).asFloatBuffer().get(featureMap); / / Convert a one-dimensional array to a two-dimensional array float[][] featureMap2D = new float

[64]

[64] ; for (int i = 0; i < 64; i++) { for (int j = 0; j < 64; j++) { featureMap2D[i][j] = featureMap[i * 64 + j]; } } / / Perform maximum pooling on the feature map float[][] pooledFeatureMap = maxPooling(featureMap2D,2); / / Convert the pooled result into a feature vector float[] featureVector = flatten(pooledFeatureMap); / / Output feature vector System.out.println("Pooled Feature Vector: " + Arrays.toString(featureVector)); } catch (IOException e) { e.printStackTrace(); } } / / Max pooling function private static float[][] maxPooling(float[][] featureMap, int poolSize) { int height = featureMap.length; int width = featureMap[0].length; int pooledHeight = height / poolSize; int pooledWidth = width / poolSize; float[][] pooledFeatureMap = new float[pooledHeight][pooledWidth]; for (int i = 0; i < pooledHeight; i++) { for (int j = 0; j < pooledWidth; j++) { float maxVal = Float.NEGATIVE_INFINITY; for (int pi = 0; pi < poolSize; pi++) { for (int pj = 0; pj < poolSize; pj++) { maxVal = Math.max(maxVal, featureMap[i * poolSize + pi][j * poolSize + pj]); } } pooledFeatureMap[i][j] = maxVal; } } return pooledFeatureMap; } / / Flatten the two-dimensional array into a one-dimensional array private static float[] flatten(float[][] array) { int rows = array.length; int cols = array[0].length; float[] flattenedArray = new float[rows * cols]; for (int i = 0; i < rows; i++) { for (int j = 0; j < cols; j++) { flattenedArray[i * cols + j] = array[i][j]; } } return flattenedArray; } } In this way, the maximum value pooling method of the mouth target attention feature map of students' English listening and speaking practice based on the BufferedInputStream mechanism is implemented. This method generates feature vectors for deep learning model training or inference through efficient data reading and parsing, combined with the maximum value pooling operation. Among them, firstly, the mouth target attention feature map of students' English listening and speaking practice is obtained in a binary format in the file, and the path is path / to / feature_map_data.bin. Then, the binary data mouth target attention feature map of students' English listening and speaking practice is read from the file using BufferedInputStream. Specifically, BufferedInputStream reduces the number of accesses to the underlying input stream through the internal buffer, thereby improving the reading efficiency. Then, the read binary data mouth target attention feature map of students' English listening and speaking practice is parsed into a float array, where the mouth target attention feature map of students' English listening and speaking practice is 64x64 in size, and each element is 4 bytes (float). Next, a simple 2x2 maximum value pooling operation is implemented. Specifically, the pooling operation reduces the size of the mouth target attention feature map of students' English listening and speaking practice and retains the most important features by sliding a 2x2 window on the feature map and taking the maximum value of the elements in the window. Furthermore, the pooled two-dimensional array is flattened into a student English listening and speaking practice mouth target attention feature map. Finally, the generated student English listening and speaking practice mouth target attention feature map is printed to the console for easy observation and verification.

[0036] In this way, the BufferedInputStream mechanism significantly improves the data reading efficiency, especially when processing large files or network data, reducing frequent I / O operations, thereby improving overall performance. The maximum pooling operation further reduces the size of the feature map, retains the most important features, and makes the generated feature vector more compact and effective. This method is suitable for large-scale data processing scenarios, such as the processing of mouth target attention feature maps for students' English listening and speaking exercises, and can efficiently generate feature vectors for deep learning model training or reasoning. In this way, not only the efficiency of data processing is improved, but also the quality of the feature vector is ensured, providing a reliable data foundation for subsequent deep learning tasks.

[0037] In the above-mentioned teaching and research management system 100, the exercise analysis report generation module 130 is used to generate a personalized English listening and speaking exercise analysis report for students based on the audio feature vector of the student's English listening and speaking exercise audio and the feature vector of the mouth target attention of the student's English listening and speaking exercise. It should be understood that the audio feature vector and the mouth target attention feature vector evaluate the student's English oral performance from different dimensions. The audio feature vector mainly reflects information on the pitch, rhythm, speaking speed, pronunciation accuracy, etc. of the voice, while the mouth target attention feature vector focuses on the details of the mouth movement and pronunciation process. The combination of the two can provide a comprehensive analysis of student performance, ensure that the evaluation covers multiple key factors of oral expression, and help teachers better understand the overall ability of students. The generated analysis report is not only an evaluation tool for current performance, but also an effective means to monitor students' learning progress. By regularly analyzing the audio features and mouth target features of students, it is possible to compare the performance changes of students in different time periods and identify the effectiveness and progress of learning. This dynamic monitoring can help teachers adjust teaching strategies in a timely manner to ensure that students always maintain a good development momentum during the learning process.

[0038] Figure 4 FIG. 1 is a block diagram of an exercise analysis report generation module in a teaching and research management system according to an embodiment of the present application. Figure 4 As shown, in a specific embodiment of the present application, the practice analysis report generating module 130 includes: a student English listening and speaking practice feature association unit 131, which is used to associate the student English listening and speaking practice audio audio feature vector and the student English listening and speaking practice mouth target attention feature vector to obtain a student practice report generation associated feature vector; a student English listening and speaking practice feature optimization unit 132, which is used to perform information confusion suppression based on maximizing entropy degradation on the student practice report generation associated feature vector to obtain a sparse optimized student practice report generation associated feature vector; a student personalized English listening and speaking practice analysis report generating unit 133, which is used to pass the sparse optimized student practice report generation associated feature vector through a generator to generate a student personalized English listening and speaking practice analysis report.

[0039] It should be understood that by associating these two feature vectors, the system can identify the intrinsic relationship between speech features and mouth movements. For example, if the audio features show that a syllable is not clearly pronounced when the student is pronouncing it, and the mouth target features also show that the mouth shape is incorrect, the system can analyze the root cause of the problem more directly. Such in-depth analysis not only improves the accuracy of the assessment, but also provides students with more targeted feedback.

[0040] In particular, in the technical solution of the present application, the student practice report generation associated feature vector is extracted from multiple data sources and may contain different contexts and semantics. However, the structure of the learning parameter matrix is ​​usually preset, and it is designed based on a certain training set or a specific task. If these designs do not fully take into account the diversity and complexity of the student practice report generation associated feature vector, it will cause structural redundancy. For example, the learning parameter matrix may contain some feature dimensions that are not important in the task, which are not directly related to the student's actual performance, but increase the complexity and noise of the information. This redundant structure makes it difficult for the generator to effectively extract useful information when processing the input, resulting in unnecessary features and interference in the generated analysis report. Further, if the learning parameter matrix cannot effectively distinguish the diversity information of the student practice report generation associated feature vector, it may mix information from different sources together, resulting in the model being unable to clearly associate features with actual student performance when inferring. In this way, feature redundancy and noise will appear, thereby causing the overfitting phenomenon of the model. Therefore, in the technical solution of the present application, the student practice report generation associated feature vector is subjected to information confusion suppression based on entropy degradation maximization to obtain a sparse optimized student practice report generation associated feature vector.

[0041] Among them, the information confusion suppression based on entropy degradation maximization is performed on the student practice report generation associated feature vector to obtain a sparse optimized student practice report generation associated feature vector, including: extracting each row vector of the learning parameter matrix in the generator as a set of student practice learning parameter row vectors; calculating the student practice entropy degradation maximization simulation value between the student practice report generation associated feature vector and each student practice learning parameter row vector in the set of student practice learning parameter row vectors to obtain a set of student practice entropy degradation maximization simulation values; based on the comparison between each student practice entropy degradation maximization simulation value in the set of student practice entropy degradation maximization simulation values ​​and a preset threshold, the learning parameter matrix is ​​gated and pruned to obtain a student practice learning parameter entropy degradation optimization matrix; the student practice report generation associated feature vector is matrix multiplied with the student practice learning parameter entropy degradation optimization matrix to obtain the sparse optimized student practice report generation associated feature vector.

[0042] The optimization steps are specifically expressed as follows: ; ; ; ; ; ;in, Exercise for students to learn the set of parameter row vectors, is a set operation, , , , are the first, second, and third row vectors of the student's practice learning parameters. , Each student exercises to learn the eigenvalues ​​of the parameter row vector, For students to practice entropy degradation maximization simulation value set Students practice entropy degradation maximization simulation value, is the maximum value function, Represents the calculation vector and The Euclidean distance between Generate associated feature vectors for student exercise reports, For the Each student exercises to learn the transpose vector of the parameter row vector, is the length of the feature vector, is matrix multiplication, , , ,, The first, second, and , learning parameter mask values, For mask operation, Indicates the preset threshold value, Exercises for students to learn parameter entropy degradation optimization matrices, Generate associated eigenvectors for sparse optimization student exercise report.

[0043] In the technical solution of the present application, it is taken into consideration that in the process of inputting the associated feature vector generated by the student practice report into the learning parameter matrix of the generator to obtain the student's personalized English listening and speaking practice analysis report, there is a mismatch between the structure of the learning parameter matrix itself and the associated feature vector generated by the student practice report, that is, the learning parameter matrix has structural redundancy and information confusion relative to the associated feature vector generated by the student practice report, which may cause feature redundancy or noise in the student's personalized English listening and speaking practice analysis report, resulting in model overfitting.

[0044] Based on this, in the technical solution of the present application, the information confusion suppression based on the maximization of entropy degradation is performed on the associated feature vector generated by the student exercise report, which uses the associated feature vector generated by the student exercise report as a wandering node, and uses the characteristic distribution between the associated feature vector generated by the student exercise report and each row vector in the learning parameter matrix to simulate the wandering pattern, and uses the distance topology between the associated feature vector generated by the student exercise report and each row vector in the learning parameter matrix as a wandering map. In this way, the wandering result of the wandering node in a specific wandering pattern on the wandering map is used to obtain the student exercise entropy degradation maximization simulation value. Furthermore, based on the comparison between each student exercise entropy degradation maximization simulation value in the set of the student exercise entropy degradation maximization simulation value and a preset threshold, the part of the learning parameter matrix that does not contribute significantly to the input data representation and may even introduce noise or redundant information is screened out. Specifically, based on the comparison result, the learning parameter matrix is ​​gated and pruned to obtain the student exercise learning parameter entropy degradation optimization matrix. Finally, the structurally optimized learning parameter matrix is ​​used as a feature modulation unit to linearly modulate the student practice report generation-related feature vector to obtain the sparse optimized student practice report generation-related feature vector.

[0045] Furthermore, by processing the sparse optimization of student practice reports to generate associated feature vectors, the system can generate personalized analysis reports. These reports not only include the specific performance of students during practice, but also provide tailored improvement suggestions to help students improve their oral English skills in a targeted manner. This dynamic feedback mechanism can adjust the learning plan in a timely manner to ensure that students can maintain continuous progress in the learning process. Teachers can also flexibly adjust teaching strategies based on the analysis results to meet the changing needs of students.

[0046] In summary, the embodiment of the present application first obtains the audio files of English listening and speaking exercises submitted by students and the video files of English listening and speaking exercises submitted by students, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally uses a generator to generate a personalized English listening and speaking practice analysis report for students, thereby reducing the workload of teachers, assisting teachers to focus on personalized guidance of students, and then optimizing the teaching process, improving the overall teaching quality, and cultivating students' more comprehensive literacy.

[0047] As described above, the teaching and research management system 100 according to the embodiment of the present application can be implemented in various terminal devices. In one example, the teaching and research management system 100 can be integrated into the terminal device as a software module and / or a hardware module. For example, the teaching and research management system 100 can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the teaching and research management system 100 can also be one of the many hardware modules of the terminal device.

[0048] Alternatively, in another example, the teaching and research management system 100 and the terminal device may also be separate devices, and the teaching and research management system 100 may be connected to the terminal device via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0049] Figure 5 Flowchart of the teaching and research management method according to the embodiment of the present application. Figure 5 As shown, according to the teaching and research management method of the embodiment of the present application, it includes: S110, obtaining an audio file of English listening and speaking practice submitted by a student and a video file of English listening and speaking practice submitted by a student; S120, extracting an audio feature vector of the student's English listening and speaking practice audio and a feature vector of the student's English listening and speaking practice mouth target attention from the audio file of the English listening and speaking practice submitted by the student and the video file of the English listening and speaking practice submitted by the student; S130, generating a personalized English listening and speaking practice analysis report for the student based on the audio feature vector of the student's English listening and speaking practice audio and the feature vector of the student's English listening and speaking practice mouth target attention.

[0050] Here, those skilled in the art can understand that the specific operations of each step in the above teaching and research management method have been referred to above. Figures 1 to 4 The teaching and research management system has been introduced in detail, and therefore, its repeated description will be omitted.

[0051] The phrase "based on" used in this specification does not mean "only based on" unless otherwise specified in other paragraphs. In other words, the phrase "based on" means both "only based on" and "at least based on".

[0052] Any reference to a unit using the names "first", "second", etc. used in this specification does not fully limit the number or order of these units. These names may be used in this specification as a convenient method to distinguish between two or more units. Therefore, a reference to a first unit and a second unit does not mean that only two units can be used or that the first unit must precede the second unit in some form.

[0053] The term "determining" as used in this specification sometimes includes a variety of actions. For example, with respect to "determining", calculating, computing, processing, deriving, investigating, looking up (e.g., searching in a table, database, or other data structure), ascertaining, etc. may be considered as performing "determining". In addition, with respect to "determining", receiving (e.g., receiving information), transmitting (e.g., transmitting a message), etc. may also be considered as performing "determining". Sending information), input, output, accessing (for example, accessing data in a memory), etc. are considered to be "judgment (determination)". In addition, regarding "judgment (determination)", resolving, selecting, choosing, establishing, comparing, etc. can also be considered to be "judgment (determination)". In other words, regarding "judgment (determination)", several actions can be considered to be "judgment (determination)".

[0054] The terms "connected", "coupled" or any variation thereof used in this specification refer to any direct or indirect connection or combination between two or more units, and may include the following situations: there are one or more intermediate units between two units that are "connected" or "coupled" to each other. The combination or connection between units can be physical, logical, or a combination of the two. For example, "connect" can also be replaced by "access". When used in this specification, two units can be considered to be "connected" or "coupled" to each other by using one or more wires, cables, and / or printed electrical connections, and as several non-limiting and non-exhaustive examples, by using electromagnetic energy with wavelengths in the radio frequency region, microwave region, and / or light (both visible and invisible light) region, etc.

[0055] The present invention has been described in detail above, but it is obvious to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented as a modification and variation without departing from the purpose and scope of the present invention as determined by the description of the claims. Therefore, the description of this specification is for the purpose of illustration and does not have any restrictive meaning for the present invention.

Claims

1. A teaching and research management system, characterized in that: include: English teaching and research practice data acquisition module, used to obtain audio files of English listening and speaking practice submitted by students and video files of English listening and speaking practice submitted by students; An English teaching and research practice data processing module is used to extract audio feature vectors of the English listening and speaking practice audio and feature vectors of the English listening and speaking practice mouth target attention of the students from the audio files of the English listening and speaking practice submitted by the students and the video files of the English listening and speaking practice submitted by the students; The practice analysis report generating module is used to generate a personalized English listening and speaking practice analysis report for students based on the audio feature vector of the student's English listening and speaking practice audio and the feature vector of the student's English listening and speaking practice mouth target attention.

2. The teaching and research management system according to claim 1, characterized in that: The English teaching and research practice data processing module includes: A student practice audio file feature extraction unit is used to extract features from the audio file of English listening and speaking practice submitted by the student to obtain an audio feature vector of the student's English listening and speaking practice audio; A student practice video file feature extraction unit is used to extract features from the video file of English listening and speaking practice submitted by the student to obtain a target region of interest of the mouth of the student's English listening and speaking practice; The student practice target area feature encoding unit is used to perform feature encoding on the student's English listening and speaking practice mouth target interest area to obtain the student's English listening and speaking practice mouth target attention feature vector.

3. The teaching and research management system according to claim 2, characterized in that: The student practices the audio file feature extraction unit, including: Performing audio vectorization on the audio file of the English listening and speaking practice submitted by the student to obtain the student's English listening and speaking practice audio input vector; The student English listening and speaking practice audio input vector is passed through the student English listening and speaking practice audio deep neural network model to obtain the student English listening and speaking practice audio audio feature vector.

4. The teaching and research management system according to claim 3, characterized in that: The student exercises a video file feature extraction unit, including: Passing the video file of the English listening and speaking practice submitted by the student through the student English listening and speaking practice video convolutional neural network to obtain an initial student English listening and speaking practice feature map; The initial student English listening and speaking practice feature map is passed through the student English listening and speaking practice target detection network to obtain the student English listening and speaking practice mouth target interest region.

5. The teaching and research management system according to claim 4, characterized in that: The student exercises a target area feature coding unit, including: The mouth target attention convolution subunit is used to pass the student English listening and speaking practice mouth target interest area through the student English listening and speaking practice mouth target attention mechanism module to obtain the student English listening and speaking practice mouth target attention feature map; The student English listening and speaking practice information feature pooling subunit is used to pool the student English listening and speaking practice mouth target attention feature map to obtain the student English listening and speaking practice mouth target attention feature vector.

6. The teaching and research management system according to claim 5, characterized in that: The student English listening and speaking practice information feature pooling subunit includes: Obtain the characteristic map of students' mouth target attention in English listening and speaking practice in binary format; Use BufferedInputStream to read the student's English listening and speaking practice mouth target attention feature map from the file; Parsing the read binary data into a data structure of a mouth target attention feature map for a student's English listening and speaking practice, wherein the data structure of the mouth target attention feature map for a student's English listening and speaking practice is a multidimensional array representing the attention distribution of the mouth target; Performing a maximum pooling operation on the parsed student English listening and speaking practice mouth target attention feature map, wherein the maximum pooling operation slides a fixed-size window on the student English listening and speaking practice mouth target attention feature map and takes the maximum value of the elements in the window; The pooled results are converted into the mouth target attention feature vector of students' English listening and speaking practice and saved in a file.

7. The teaching and research management system according to claim 6, characterized in that: The exercise analysis report generation module includes: A student English listening and speaking practice feature association unit, used for associating the audio feature vector of the student English listening and speaking practice audio with the mouth target attention feature vector of the student English listening and speaking practice to obtain a student practice report generation association feature vector; A student English listening and speaking practice feature optimization unit is used to suppress information confusion based on maximizing entropy degradation on the student practice report generation-related feature vector to obtain a sparse optimized student practice report generation-related feature vector; The student personalized English listening and speaking practice analysis report generating unit is used to generate the associated feature vector of the sparse optimized student practice report through a generator to generate a student personalized English listening and speaking practice analysis report.

8. The teaching and research management system according to claim 7, characterized in that: The student English listening and speaking practice feature optimization unit includes: Extracting each row vector of the learning parameter matrix in the generator as a set of row vectors of the student's practice learning parameters; Calculating the student practice entropy degradation maximization simulation value between the student practice report generation associated feature vector and each student practice learning parameter row vector in the set of student practice learning parameter row vectors to obtain a set of student practice entropy degradation maximization simulation values; Based on the comparison between each student practice entropy degradation maximization simulation value in the set of student practice entropy degradation maximization simulation values ​​and a preset threshold, the learning parameter matrix is ​​gated and pruned to obtain a student practice learning parameter entropy degradation optimization matrix; The student practice report generation associated feature vector is matrix multiplied with the student practice learning parameter entropy degradation optimization matrix to obtain the sparse optimized student practice report generation associated feature vector.

9. A teaching and research management method, characterized in that: include: Obtain audio files of English listening and speaking exercises submitted by students and video files of English listening and speaking exercises submitted by students; Extracting the audio feature vector of the English listening and speaking practice audio and the mouth target attention feature vector of the English listening and speaking practice of the student from the audio file of the English listening and speaking practice submitted by the student and the video file of the English listening and speaking practice submitted by the student; Based on the audio feature vector of the student's English listening and speaking practice audio and the feature vector of the student's English listening and speaking practice mouth target attention, a personalized English listening and speaking practice analysis report for the student is generated.

10. The teaching and research management method according to claim 9, characterized in that: Based on the audio feature vector of the student's English listening and speaking practice audio and the feature vector of the student's English listening and speaking practice mouth target attention, a personalized English listening and speaking practice analysis report for the student is generated, including: Associating the audio feature vector of the student's English listening and speaking practice audio with the mouth target attention feature vector of the student's English listening and speaking practice to obtain a student practice report generation associated feature vector; Performing information confusion suppression based on maximizing entropy degradation on the associated feature vector generated by the student exercise report to obtain a sparse optimized associated feature vector generated by the student exercise report; The sparse optimized student practice report generates associated feature vectors and passes them through a generator to generate a personalized English listening and speaking practice analysis report for the student.

Images