Automatic wrong question collection method based on video image acquisition

Abstract

The invention discloses an automatic wrong question collection method based on video image collection, and belongs to the technical field of image processing, and the method specifically comprises the steps: obtaining continuous video frames of homework or test paper through video collection equipment, and extracting a question region image through preprocessing; executing text detection and recognition on the subject area image, extracting subject content and answer content, and constructing a data pair; matching the data pair with a standard answer library, and identifying a wrong question area in combination with correction trace features; performing knowledge point labeling and difficulty grading on the regional questions, performing classification and collection based on labeling results, and generating and outputting a structured wrong question set; according to the method, automatic identification, classification and collection of the wrong questions are realized, the wrong question sorting efficiency is improved, and the method is suitable for wrong question management scenes of various homework and test papers.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and in particular relates to an automatic error question collection method based on video image collection. BACKGROUND

[0002] In the process of education and teaching, error question collection is an important means to help students check for omissions and improve learning efficiency. The traditional error question collection method mainly relies on manual sorting, and students need to manually copy error questions and record error reasons, which not only consumes time and effort, but also is prone to omissions and non-standard sorting.

[0003] In the prior art, although there are some automatic error question collection tools, most of them only determine error questions by scanning the question text and comparing it with the standard answer, which has obvious limitations. On the one hand, these tools cannot perform deep semantic analysis on the question text, and cannot accurately grasp the subject knowledge points and logical structure examined by the question; on the other hand, they ignore the time sequence characteristics in the student's answering process, such as erasing traces, pause duration, and answering order, which can reflect the student's thinking process and error source to a great extent. In addition, the error question clustering method of the existing tools is relatively rough, cannot accurately cluster error questions according to their specific characteristics, and cannot effectively associate error questions with subject knowledge point maps and dynamically update label weights, making it difficult to meet the needs of personalized teaching and precise tutoring.

[0004] A homework error question collection system based on image recognition is disclosed in Chinese Patent No. CN115841670B, which includes a database module, a data acquisition module, a data processing module, and a data proofreading module. The system acquires picture information uploaded by a user terminal, identifies the text outline corresponding to the answer in the picture based on the picture information, compares the text outline with the outline pattern stored in the database, determines the text corresponding to the text outline, considers the associated text of adjacent text outlines when the coincidence degree of the text outline and the outline pattern is low, and after adjusting the coincidence degree comparison threshold, reacquires the coincidence degree comparison result of the outline pattern corresponding to the text outline and the associated text, thereby improving the recognition rate of the text corresponding to the text outline.

[0005] The main shortcomings of the prior art are that it relies on static image input and cannot extract error questions from continuous video in real time; the recognition accuracy of correction traces is insufficient and is easily affected by handwriting or typesetting; and there is a lack of automatic knowledge point classification and difficulty grading mechanism, resulting in low reuse value of the error question set. Therefore, the present application proposes an automatic error question collection method based on video image collection to solve the above problems. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides an automatic wrong question collection method based on video image acquisition, which obtains continuous video frames of homework or test papers through a video acquisition device, extracts question region images through preprocessing, performs text detection and recognition on the question region images, extracts question content and answer content and constructs data pairs, matches the data pairs with a standard answer library, identifies wrong question regions in combination with correction trace features, annotates knowledge points and classifies difficulties for the question in the region, classifies and collects based on the annotation results, generates a structured wrong question set and outputs, and realizes automatic identification, classification and collection of wrong questions, improves wrong question collation efficiency and is suitable for wrong question management scenarios of various homework and test papers.

[0007] To achieve the above object, the present application provides the following technical scheme.

[0008] An automatic wrong question collection method based on video image acquisition, comprising:

[0009] A video acquisition device is used to obtain continuous video frames containing homework or test papers, and the video frames are preprocessed to obtain question region images;

[0010] Text detection and recognition are performed on the question region images, question content and corresponding answer content are extracted, and question-answer data pairs are constructed;

[0011] The question-answer data pairs are matched with a preset standard answer library, and wrong question regions are identified in combination with correction trace features in the question region images;

[0012] Knowledge points and difficulty levels are annotated for the question in the wrong question region, classification and collection are performed based on the annotation results, a structured wrong question set is generated and outputted.

[0013] Specifically, the video frames are preprocessed to obtain question region images, comprising:

[0014] Interframe anti-shake processing is performed on the continuous video frames, feature point matching is used to eliminate image offset caused by shooting jitter, and a stable video frame sequence is obtained;

[0015] An adaptive threshold segmentation algorithm is used to extract paper regions in the video frames based on the stable video frame sequence, and an initial paper region image is obtained;

[0016] Perspective transformation correction is performed on the initial paper region image, and a rectangular page image of orthographic projection is obtained;

[0017] Based on the rectangular page image of orthographic projection, a question segmentation line is located through edge detection and Hough transformation, and an independent question region image is segmented according to the question segmentation line.

[0018] Specifically, perspective transformation correction is performed on the initial paper region image to obtain a rectangular page image of orthographic projection, including:

[0019] Edge detection is performed on the initial paper region image to identify four vertex coordinates of the paper region, and spatial distortion parameters of the vertices are calculated according to the spatial positional relationship of the four vertices;

[0020] A perspective transformation matrix is constructed based on the spatial distortion parameters, and the perspective transformation matrix is used to map the inclined initial paper region image to a two-dimensional orthographic plane to obtain a transformed image;

[0021] The transformed image is subjected to gray scale equalization processing to obtain a rectangular page image of orthographic projection.

[0022] Specifically, text detection and recognition are performed on the question region image to extract question content and corresponding answer content, including:

[0023] For the question region image, an EAST text detection model is used to locate text lines in the question region, and the stem text and the answer area text are distinguished by the position features and color features of the text lines to obtain stem text lines and answer area text lines;

[0024] The CRNN model is used to recognize characters in the above-mentioned stem text lines and answer area text lines respectively, and the image text corresponding to the stem text lines is converted into editable stem text data, and the image text corresponding to the answer area text lines is converted into editable answer content text data;

[0025] The recognized stem text data and answer content text data are subjected to semantic error correction, and a preset subject terminology dictionary is combined to correct the recognized incorrect characters or formula symbols, so as to finally obtain accurate question content and corresponding answer content.

[0026] Specifically, the combination of the correction trace features in the question region image identifies the wrong question region, including:

[0027] Red or blue correction traces in the question region image are extracted by color space threshold segmentation; the red or blue correction traces include cross, wavy line, and annotation text;

[0028] The spatial overlap degree of the correction trace and the answer area is calculated, and when the overlap degree exceeds a preset threshold, the region is marked as a suspected wrong question;

[0029] Semantic similarity calculation is performed on the answer content and the standard answer of the suspected wrong question, and when the similarity is lower than a qualified threshold, the question is confirmed as a wrong question.

[0030] Specifically, the semantic similarity calculation of the answer content and the standard answer of the suspected wrong question includes:

[0031] determine the type of the suspected wrong question, and divide it into objective questions, subjective questions or formula questions;

[0032] For the suspected wrong question determined as an objective question, the answer result in the answer content is extracted, the answer result is compared with the corresponding standard answer at the character level, the character consistency rate is calculated, and the objective question semantic similarity is taken as the character consistency rate;

[0033] For the suspected wrong question determined as a subjective question, the answer content and the corresponding standard answer are obtained, and are respectively input into the pre-trained BERT model. The answer content is converted into an answer semantic vector by the BERT model, the standard answer is converted into a standard semantic vector, the cosine similarity of the answer semantic vector and the standard semantic vector is calculated, and the subjective question semantic similarity is taken as the cosine similarity;

[0034] For the suspected wrong question determined as a formula question, the formula text in the recognized answer content is converted into an answer formula in LaTeX format, and the formula text in the corresponding standard answer is converted into a standard formula in LaTeX format. Syntax trees are constructed based on the answer formula in LaTeX format and the standard formula in LaTeX format respectively, the structural similarity is calculated by comparing the structural differences of the two syntax trees, and the formula question semantic similarity is taken as the structural similarity.

[0035] Specifically, the question corresponding to the wrong question region is annotated with knowledge points and classified by difficulty, including:

[0036] The question text corresponding to the wrong question region is extracted, and the question text corresponding to the wrong question region is input into a pre-set knowledge point classification model to output a knowledge point label corresponding to the question text; the classification model is obtained by training a subject knowledge point corpus;

[0037] Based on the error type of the wrong question, combined with the error rate data of the question in previous years, the difficulty coefficient of the question is calculated according to a pre-set calculation rule; the error type of the wrong question includes concept error, calculation error and logic error;

[0038] Integrate the knowledge point label and the difficulty coefficient to generate a wrong question multi-dimensional attribute label containing knowledge point information and difficulty information.

[0039] Specifically, the knowledge point classification model includes a text embedding layer, an attention mechanism layer and a classification output layer, wherein:

[0040] The text embedding layer receives the input question text, converts each word in the question text into a fixed-dimensional word vector through a pre-set word embedding algorithm, and forms a word vector sequence corresponding to the question text;

[0041] The attention mechanism layer receives the word vector sequence output by the text embedding layer, and performs weight calculation and distribution on each word vector in the sequence to obtain a weight-adjusted word vector sequence;

[0042] The classification output layer receives the weight-adjusted word vector sequence output by the attention mechanism layer, performs feature extraction and classification calculation on the weight-adjusted word vector sequence through a fully connected network, outputs a probability distribution corresponding to the knowledge point label, and selects the label with the highest probability as the knowledge point classification result of the question text.

[0043] Specifically, the preset calculation rule refers to a specific method for calculating the difficulty coefficient by combining the error type weight of the wrong question and the historical error rate data, including:

[0044] According to the difference in knowledge mastery difficulty reflected by different error types, a fixed weight value is set for each error type in advance;

[0045] Convert the error rate of the question in previous years into a standardized value between 0 and 1;

[0046] After calculating the weight value and the standardized value of the error rate of the question in previous years according to the preset proportion respectively, add the two results to obtain the difficulty coefficient;

[0047] Map the calculated difficulty coefficient to a difficulty level.

[0048] Specifically, the classification and collection based on the annotation result generates a structured wrong question set, including:

[0049] Construct a tree-shaped classification directory according to the knowledge point label of the wrong question to form a preliminary classification system divided by knowledge points; the tree-shaped classification directory takes a subject category as the root node and each subdivided knowledge point as a child node, and the wrong questions with the same knowledge point label are classified under the corresponding child node directory;

[0050] In each of the above knowledge point child node directories, extract the difficulty coefficient of the wrong question, arrange the wrong questions in ascending order of difficulty coefficient, and form a gradient exercise sequence;

[0051] For each wrong question in the gradient exercise sequence, extract the key error step and the corresponding correct analysis of the wrong question, associate the key error step and the correct analysis to the corresponding question in the wrong question set, and generate a wrong question document with analysis;

[0052] The key error step refers to the error answering part located based on the correction trace;

[0053] The corresponding correct analysis refers to the analysis content matched from the standard answer library;

[0054] The wrong question document with analysis contains question content, answering situation, key error step and correct analysis.

[0055] Compared with the prior art, the present application has the following advantages:

[0056] 1. The present application provides an automatic wrong question collection method based on video image collection, which realizes the full-process automation of wrong question collection through video image collection and intelligent processing technology, accurately extracts the question area from continuous video frames, efficiently identifies wrong questions by combining text recognition and correction trace analysis, and greatly reduces the time cost of wrong question sorting, thereby improving the efficiency and convenience of wrong question management.

[0057] 2. The present application provides an automatic wrong question collection method based on video image collection, which supports real-time video stream collection, adapts to dynamic shooting needs in various scenes such as classrooms and homes through inter-frame processing and perspective correction, integrates text semantic comparison and correction trace recognition to improve the accuracy of wrong question judgment, especially suitable for handwritten answering and teacher correction scenes, and realizes automatic knowledge point labeling and difficulty grading to generate structured wrong question sets. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 Fig. 1 is a schematic diagram of the automatic wrong question collection method based on video image collection of the present application;

[0059] Figure 2 Fig. 4 is a principle flowchart of the automatic wrong question collection method based on video image collection of the present application. DETAILED DESCRIPTION

[0060] Embodiment 1

[0061] Please refer to Figure 1 and Figure 2 , the present application provides an embodiment: an automatic wrong question collection method based on video image collection, the method comprises steps S1~S4, including the following steps:

[0062] S1: acquiring continuous video frames containing homework or test papers through a video acquisition device, pre-processing the video frames to obtain question area images;

[0063] S2: performing text detection and recognition on the question area images, extracting question content and corresponding answering content, and constructing question-answering data pairs;

[0064] S3: Matching the question-answer data pair with a preset standard answer library, combining the correction trace features in the question area image, and identifying the wrong question area;

[0065] S4: Knowledge point annotation and difficulty classification are performed on the question corresponding to the wrong question area, classification is performed based on the annotation result, a structured wrong question set is generated, and output is performed.

[0066] Further, the method further comprises:

[0067] Periodically updating the structured wrong question set, and updating the knowledge point classification model through incremental learning;

[0068] Based on the wrong question collection record of the user, an individual weak knowledge point analysis report is generated;

[0069] Supporting export and sharing of the wrong question set, formats including PDF, Word or JSON format adapted to the education platform.

[0070] In summary, the present application is suitable for automatic collection of wrong questions of paper carriers such as student homework, test papers, exercise books, and is especially suitable for the following scenarios:

[0071] In class exercises, the teacher takes pictures of student homework by mobile device when patrolling, and real-time identification of wrong questions and statistics of class error rate are performed;

[0072] In homework, students take pictures of completed homework through tablets or mobile phones, and automatically generate personal wrong question sets;

[0073] After correcting the test paper, the test paper video is processed in batches through a scanner, and a wrong question analysis report classified by knowledge points is quickly generated.

[0074] Embodiment 2

[0075] In this embodiment, the video frame is preprocessed to obtain a question area image, including:

[0076] S1.1: Interframe anti-shake processing is performed on the continuous video frames, and the image offset caused by the shooting jitter is eliminated through feature point matching to obtain a stable video frame sequence;

[0077] Further, the specific steps of S1.1 include:

[0078] (1) read the input of the continuous video frame, arrange it in time sequence as frame sequence, and select the initial reference frame from the frame sequence, usually select the first frame or the visually most stable frame, in the application, by calculating the motion amplitude of the first few frames, select the frame with the minimum motion as the first frame, that is, as the alignment reference of all subsequent frames, the core of this step in the application is to establish a stable anchor point to ensure that the offset of the subsequent frames is corrected based on the reference frame;

[0079] (2) perform feature point detection on the reference frame and each subsequent frame, the specific process is:

[0080] Through the gray value of the contrast neighborhood pixels around each pixel by fast corner detection algorithm, if there are continuous multiple pixels whose gray value difference exceeds the threshold value, it is determined as a key point, these key points are usually the edge, corner and other gray value change areas in the image, which have strong anti-interference ability, wherein the fast corner detection algorithm is the prior art content in the art, and is not the inventive scheme of the present application, and will not be described here;

[0081] Based on the gray distribution around the key point, a binary descriptor with rotation invariance is calculated, wherein the binary descriptor with rotation invariance is determined by the gray centroid to determine the direction, and then the gray contrast coding of the neighborhood pixels is performed, so that the key points at the same physical position in different frames have similar descriptors, which is convenient for subsequent matching;

[0082] (3) match the feature points of the reference frame and the current frame;

[0083] (4) based on the effective matching feature point pairs, calculate the transformation matrix which can describe the offset of the current frame relative to the reference frame, wherein since the shooting jitter mainly shows as translation, rotation, scaling and other rigid motion, an affine transformation matrix is usually used; the affine transformation matrix is a 2x3 matrix, which includes translation, rotation angle and scaling factor, and its principle is to minimize the error between the coordinates of the transformed current frame feature points and the corresponding point coordinates of the reference frame by least square method, that is, to make the transformed current frame feature points as close as possible to the positions in the reference frame.

[0084] Specifically, an equation group is established for the coordinate difference of each matching pair, the matrix parameters that minimize the overall error are solved, and finally the transformation matrix that can completely describe the offset of the current frame is obtained.

[0085] (5) apply the calculated transformation matrix to all pixels of the current frame, and perform geometric transformation on the current frame by inverse mapping, so that the feature points of the current frame are aligned with the reference frame, and the offset caused by jitter is eliminated.

[0086] In the transformation process, for the pixels beyond the range of the original image after transformation, an edge filling strategy is adopted, i.e. copying the edge pixel value or performing a clipping process, to ensure the integrity of the output image.

[0087] (6) For each subsequent frame in the video sequence, repeat (3) to (5): based on the reference frame, detect feature points, match, calculate the transformation matrix, and correct the offset. If the video is too long or the shaking pattern changes in the later period, the reference frame can be dynamically updated. For example, after processing a certain number of frames, the current stable frame is set as the new reference frame to avoid excessive cumulative error;

[0088] (7) After the above processing, all frames are aligned with the reference frame, and the irregular offset caused by shaking is eliminated, finally forming a continuous and stable video frame sequence.

[0089] Further, the feature points of the reference frame and the current frame are matched, specifically including:

[0090] (1) The Hamming distance is used to measure the similarity between the descriptor of each feature point in the current frame and the descriptors of all feature points in the reference frame. The smaller the Hamming distance, the higher the similarity. The Hamming distance calculation formula is a prior art in the art and is not the inventive scheme of the present application, and will not be described here.

[0091] (2) The highest similarity candidate matching pair is preliminarily selected, wherein each current frame feature point corresponds to the nearest feature point in the reference frame.

[0092] (3) At the same time, due to shaking or noise, an error matching may occur, and a random sample consensus algorithm is used for further screening, including: a small number of matching pairs are randomly selected to fit an initial transformation model, the inliers, i.e. the matching pairs with errors within a threshold, are counted, and after repeating multiple times, the matching pairs corresponding to the model with the most inliers are retained as the final effective feature point correspondence. The random sample consensus algorithm is a prior art in the art and is not the inventive scheme of the present application, and will not be described here.

[0093] In summary, in the present application, the difference between two feature point descriptors is compared, such as the Hamming distance of the binary descriptor, i.e. the number of different bits in the two descriptors. The smaller the difference, the higher the possibility of being the same physical point. The matrix parameters are determined by minimizing the position difference between the current frame feature points after transformation and the corresponding points in the reference frame. The essence is to make the transformed current frame as consistent as possible with the structure of the reference frame, so as to offset the offset caused by shaking. The spatial correspondence is found through cross-frame matching, and the position deviation is corrected by mathematical transformation, finally realizing the stable alignment between frames.

[0094] S1.2: For the stable video frame sequence, an adaptive threshold segmentation algorithm is used to extract the paper area in the video frame, and an initial paper area image is obtained, wherein the adaptive threshold segmentation algorithm is prior art content in the field and is not the creative scheme of the present application, and will not be described here;

[0095] S1.3: Perspective transformation correction is performed on the initial paper area image to obtain a rectangular page image of orthographic projection;

[0096] S1.4: Based on the rectangular page image of orthographic projection, the question division line is located by edge detection and Hough transformation, and the independent question area image is segmented according to the question division line.

[0097] Further, the specific steps of S1.4 include:

[0098] (1) The orthographic projection rectangular page image is preprocessed to reduce noise interference and enhance the accuracy of subsequent edge detection, including:

[0099] The colored rectangular page image is converted into a gray image by retaining the brightness information of the pixels and removing the color channels, simplifying the image data and making the subsequent processing more efficient;

[0100] A Gaussian blur algorithm is used to smooth the gray image, and a weighted average of the gray value of each pixel is performed through a preset size Gaussian kernel or a weighted sliding window, wherein the center pixel weight is high and the edge pixel weight is low, so as to weaken the random noise in the image, such as paper texture and light spots during shooting, while trying to preserve the true edge information, wherein the Gaussian blur algorithm is prior art content in the field and is not the creative scheme of the present application, and will not be described here;

[0101] (2) Edge detection is performed on the preprocessed gray image, the core of which is to identify the regions with sharp changes in gray value in the image, which usually correspond to lines, character edges or question division lines on the paper in the present application, and the specific process includes:

[0102] The Sobel operator is used to calculate the gray gradient of the image in the horizontal and vertical directions respectively, wherein the size of the gradient reflects the strength of the edge, and the direction of the gradient indicates the direction of the edge, and the Sobel operator is prior art content in the field and is not the creative scheme of the present application, and will not be described here;

[0103] The gradient direction of each pixel is judged, only the pixel with the largest local gray change in the gradient direction is retained, and the pixel without the largest local gray change is eliminated, so that the edge changes from a wide band to a thin line;

[0104] Two thresholds, high threshold and low threshold, are set, pixels with gradient values exceeding the high threshold are determined as strong edges, pixels with gradient values lower than the low threshold are determined as non-edges and are directly removed, and pixels with gradient values between the two thresholds are further determined, including: if adjacent to a strong edge pixel, the pixel is retained as a weak edge, otherwise, the pixel is removed, and finally a binary image containing only clear edges is obtained, in which edge pixels are white and the background is black;

[0105] (3) The edge image is mapped to a Hough space, in the image space, a straight line can be described by a polar equation, i.e., the coordinates of each point are related to the angle and distance of the straight line, wherein the core of the Hough transform is to convert each edge pixel in the image space into a curve in the Hough space, which represents all possible straight lines passing through the pixel;

[0106] (4) An accumulator, i.e., a counter, is set in the Hough space, and the points passing through each curve are counted, if multiple edge pixels correspond to the same straight line, the curves in the Hough space will intersect at the same point, and the accumulator count of the point will be significantly higher than that of the non-intersecting point, and the higher the count, the more edge pixels the corresponding straight line contains, and the more likely it is a real straight line;

[0107] (5) A count threshold is set, and the straight lines corresponding to the points with counts exceeding the threshold in the accumulator are extracted, and then the horizontal straight lines and the vertical straight lines are screened out according to the angles of the straight lines, and finally the coordinates of all possible question segmentation lines, such as the Y-axis coordinates of the horizontal straight lines and the X-axis coordinates of the vertical straight lines, are obtained;

[0108] (6) The boundaries of each question are determined and the independent regions are segmented according to the located horizontal and vertical separation lines, including:

[0109] The horizontal separation lines are sorted in ascending order of Y-axis coordinates, and the vertical separation lines are sorted in ascending order of X-axis coordinates, to clarify the spatial distribution of the lines in the page;

[0110] For the horizontal separation lines, the region between any two adjacent lines is usually the up-down range of a question, and for the vertical separation lines, the region between any two adjacent lines is usually the left-right range, and in combination with the horizontal and vertical separation lines, each question region is defined as a rectangular region below the last horizontal separation line, above the next horizontal separation line, right of the left vertical separation line, and left of the right vertical separation line;

[0111] If any region has only horizontal separation lines and no vertical separation lines, the left and right edges of the page are used as the left and right boundaries, and if there are multiple dense horizontal or vertical separation lines, the line spacing is calculated to remove redundant lines with too small spacing;

[0112] According to the determined rectangular boundary of each question, a corresponding region is cropped from the orthographic projection rectangular page image to obtain an independent question region image.

[0113] In the present application, the whole process starts from the orthographic projection rectangular page image, and then the clear edge contour is extracted through pre-processing and edge detection, and then the straight line in the edge is converted from the image space to the parameter space through the Hough transform for identification and screening, and finally the boundary of each question is determined according to the coordinates of the separation line to complete the region segmentation, and finally the target of accurately extracting the independent question region from the complete page is realized.

[0114] The perspective transformation correction is performed on the initial paper region image to obtain an orthographic projection rectangular page image, comprising:

[0115] S1.3.1: edge detection is performed on the initial paper region image to identify the four vertex coordinates of the paper region, and the spatial distortion parameters of the vertex are calculated according to the spatial positional relationship of the four vertices, wherein the Canny edge detection algorithm is used for edge detection, and the Canny edge detection algorithm is prior art content in the field and is not the creative scheme of the present application, and will not be described here;

[0116] S1.3.2: a perspective transformation matrix is constructed based on the spatial distortion parameters, and the perspective transformation matrix is used to map the inclined initial paper region image to a two-dimensional orthographic plane to obtain a transformed image;

[0117] Further, the specific steps of S1.3.2 include:

[0118] (1) determining the vertex coordinates; the vertex coordinates include initial vertex and target vertex;

[0119] Initial vertex: from the extracted initial paper region image, the pixel coordinates of the four identified vertices are obtained, which are set as A, B, C, and D, corresponding to the four corners in the inclined state of the paper, and the coordinates take the upper left corner of the image as the origin, with the horizontal direction as the X-axis and the vertical direction as the Y-axis, wherein the pixel coordinates reflect the distortion of the paper due to the inclination, for example, the originally rectangular paper may present a trapezoidal or irregular quadrilateral in the image, and the relative positions of the four vertices deviate from the right angle relationship of the rectangle;

[0120] Target vertex: set the ideal two-dimensional orthographic plane, i.e. the four vertex coordinates of the rectangle that the corrected paper should present, which are set as A', B', C', and D', and are usually set according to the approximate proportion of the initial paper, for example, A' is set as (0, 0), B' is set as (W, 0), C' is set as (W, H), and D' is set as (0, H), forming a standard rectangle, i.e. the perspective transformation matrix, wherein W represents the target width and H represents the target height;

[0121] (2) For each pair of corresponding vertices, such as A and A', assume the initial coordinates are The target coordinates are According to the principle of perspective transformation, the coordinates before and after the transformation satisfy the condition that the target coordinates are the normalized result of the initial coordinates after transformation by the perspective transformation matrix. Simply put, it involves using the parameters of the perspective transformation matrix to... Convert to Furthermore, this transformation requires a one-to-one correspondence between four pairs of vertices, where x and y represent the initial X-axis and Y-axis coordinates, respectively. and These represent the transformed X-axis and Y-axis coordinate values, respectively.

[0122] (3) By establishing such a correspondence for each of the four pairs of vertices, we can obtain 8 equations, of which each pair of vertices contributes 2 equations. For example, for vertices A and A', correspond At that time, the equation will contain 8 parameters of the perspective transformation matrix and , , , The relationship, and so on, includes, and These represent the X-axis and Y-axis coordinates of vertex A, respectively. and These represent the X-axis and Y-axis coordinates of vertex A', respectively.

[0123] (4) By solving the system of equations consisting of the above 8 equations, the 8 parameters of the perspective transformation matrix are obtained. Since the vertex coordinates in the actual image may contain noise, the direct solution may have deviation. Therefore, it is necessary to optimize by the least squares method. The least squares method is the prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0124] (5) Using the solved perspective transformation matrix, perform coordinate transformation on each pixel of the initial paper area image. The specific process is as follows:

[0125] Traverse each pixel of the target frontal plane image, i.e. the rectangular image to be generated, and calculate the original coordinates of the pixel in the initial tilted image by using the inverse of the perspective transformation matrix;

[0126] Repeat the above operation for all pixels of the target frontal image, rearrange the pixels in the initial tilted image according to the rules of the perspective transformation matrix, fill them into the target rectangular area, and finally obtain the transformed image with tilt distortion eliminated.

[0127] Where, since the calculated original coordinates can be decimals, not integer pixel positions, the pixel value at this position needs to be determined by a bilinear interpolation method: by looking up the gray scale values of the four integer pixels around the original coordinates, the final value is calculated according to the distance weight, ensuring the smoothness of the target image.

[0128] In the present application, the whole process starts with the difference between the initial inclined vertex and the target orthographic plane vertex, solves the perspective transformation matrix by establishing a corresponding relationship, and then repositions the pixels of the inclined image to the orthographic plane using the matrix, finally realizes the mapping from the distorted state to the regular state, with the output of each step as the input of the next step, realizing vertex coordinates → matrix parameters → mapped pixels → complete image, forming a coherent distortion correction logic, and ensuring that the inclined paper image is accurately converted into a flat rectangular image.

[0129] S1.3.3: Perform gray scale equalization processing on the transformed image to obtain a rectangular page image of orthographic projection, wherein the gray scale equalization processing is a prior art content in the field and is not the inventive scheme of the present application, and will not be described here.

[0130] Perform text detection and recognition on the question area image to extract the question content and the corresponding answer content, including:

[0131] S2.1: For the question area image, use the EAST text detection model to locate the text lines in the question area, distinguish the stem text from the answer area text by the position features and color features of the text lines, and obtain the stem text lines and the answer area text lines;

[0132] Wherein, EAST (Efficient and Accurate Scene Text Detector) is a scene text detection model, and its core advantage is end-to-end detection: directly outputting the boundary box of the text line from the input image without intermediate steps, so it is fast and accurate.

[0133] Further, the specific steps of S2.1 include:

[0134] (1) Preprocess the question area image to ensure that it meets the input requirements of the EAST model, including: first convert the image to RGB format, because the EAST model usually receives three-channel color images, then perform size normalization, and finally standardize the pixel values to make the input data distribution consistent with the model training data, improving the detection stability.

[0135] (2) Input the preprocessed question area image into the EAST model to calculate the boundary box of all text lines, including:

[0136] The backbone network in the EAST model performs multiple convolutions and down-sampling on the input image to generate feature maps of different scales. Among these feature maps, the low-level features retain the edges and stroke details of the text, which are suitable for detecting small-sized text. The high-level features contain more abstract semantic information, which are suitable for detecting large-sized text or text in complex backgrounds.

[0137] The feature pyramid network is used to fuse feature maps of different scales. The fused feature maps contain both detailed information and semantic information, which can accurately locate text lines of different sizes and directions. The feature pyramid network is prior art in the field and is not part of the inventive concept of the present application, so it will not be described here.

[0138] The fused feature maps are input into two branches of the output layer:

[0139] Score map branch: The probability of each pixel being text is predicted through convolution operation to generate a text score map. The probability value is between 0 and 1, and the closer it is to 1, the more likely it is to be text.

[0140] Geometric shape branch: The boundary box parameters corresponding to each text pixel are predicted through convolution operation to generate a geometric shape map.

[0141] Extracting text line boundary boxes: Threshold filtering is performed on the score map, and then the boundary box parameters corresponding to these pixels in the geometric shape map are used to merge adjacent text pixels into complete text line boundary boxes. For example, the boundary boxes corresponding to pixels belonging to the same text line are merged into a single rectangular box. Finally, the boundary boxes of all text lines in the question area image are obtained, where each boundary box is represented by four vertex coordinates and encloses a text line.

[0142] (3) Using the boundary box information and pixel color features of the text lines to distinguish the stem and the answer area, including:

[0143] Statistical all text line boundary box coordinates, where the origin is the top left corner of the question area image, X is the horizontal direction, and Y is the vertical direction. The center coordinates of each text line are calculated. In general, the stem text is located more to the top or left in the question area, and the answer area text is located more to the bottom or right. By comparing the relative positions of the center coordinates of the text lines, the two types of text lines are preliminarily divided, where the stem text includes the question description and the problem statement, and the answer area text includes the student's handwritten answer.

[0144] Extracting the color features of the text lines, where the stem text is mostly printed and the color is usually pure black, and the answer area text is mostly handwritten and the color is usually blue or black. The specific operation is as follows:

[0145] Convert the pixels within each text line boundary box from RGB color space to HSV space.

[0146] statistical distribution of hue H, saturation S, and lightness V values of pixels in the bounding box;

[0147] According to the color distribution characteristics, the preliminarily divided text lines are subjected to secondary verification: if the pixels in the text line bounding box are mainly black printing characteristics, the text line is confirmed as a stem text line; if the pixels are mainly blue or handwritten black characteristics, the text line is confirmed as an answer area text line;

[0148] In combination with the preliminary division based on the position characteristics and the secondary verification based on the color characteristics, misjudgments are eliminated, and finally all stem text lines and answer area text lines are determined.

[0149] In the present application, the entire process starts from the image of the question area, first adapts the input of the EAST model through preprocessing, and then uses the feature extraction, fusion and prediction of the EAST model to locate the bounding boxes of all text lines; finally, the position coordinates and color distribution of the text lines are analyzed to distinguish the stem text lines and the answer area text lines, forming a complete logical chain from the image to the text line type.

[0150] S2.2: The above stem text lines and answer area text lines are subjected to character recognition through a CRNN model, respectively, to convert the image text corresponding to the stem text lines into editable stem text data and convert the image text corresponding to the answer area text lines into editable answer content text data;

[0151] The CRNN (Convolutional Recurrent Neural Network) includes a convolution layer, a recurrent layer and a transcription layer, is suitable for character recognition tasks, and can directly recognize an ordered character sequence from continuous image pixels without manually segmenting individual characters.

[0152] Further, the specific steps of S2.2 include:

[0153] (1) The images of the stem text lines and the answer area text lines are respectively preprocessed to ensure that they meet the input requirements of the CRNN, wherein the CRNN is a prior art content in the field and is not the inventive scheme of the present application, and thus is not described here;

[0154] (2) The preprocessed text line images are input into the convolution layer of the CRNN to extract feature sequences;

[0155] (3) The one-dimensional feature sequences output by the convolution layer are input into the recurrent layer of the CRNN to process the order dependence between characters;

[0156] (4) Through the CTC (Connectionist Temporal Classification) connection timing classification loss function, the probability distribution output by the cycle layer is converted into the final character sequence to obtain editable text, and the problem of mismatch between the feature sequence length and the character sequence length is solved;

[0157] (5) The above steps (1)-(4) are repeated for the stem text line image and the answer area text line image;

[0158] After the stem text line is recognized by the CRNN, the editable stem text data is output;

[0159] After the answer area text line is recognized by the CRNN, the editable answer content text data is output.

[0160] In the present application, the whole process starts with the pre-processed text line image, first converts the image into a one-dimensional sequence containing visual features through the convolution layer, then captures the sequential dependence between characters through the bidirectional LSTM, outputs the character probability at each position, and finally transcribes the final character sequence through the CTC. This process is performed on the text lines of the stem and the answer area, which can convert the text in the image into editable text data.

[0161] S2.3: Perform semantic error correction on the recognized stem text data and answer content text data, correct the recognized incorrect characters or formula symbols in combination with the preset subject terminology dictionary, and finally obtain the accurate question content and corresponding answer content.

[0162] Further, the specific steps of S2.3 include:

[0163] (1) Pre-process the stem text data and answer content text data recognized by the CRNN to remove interference information;

[0164] (2) Construct a subject terminology dictionary to establish a reference standard for error correction;

[0165] (3) Compare the pre-processed text unit with the subject terminology dictionary to mark the units that may have errors, specifically including: for each text unit after splitting, first compare it with the standard writing in the basic terminology library, if it is completely matched, such as acceleration exists in the terminology library, it is determined as a reliable unit, if it is not matched, such as acceleration is not in the terminology library, it is marked as a suspicious unit;

[0166] (4) For the marked suspicious units, generate multiple candidate correction words in combination with the subject terminology dictionary and the context, specifically including: query the historical errors similar to the suspicious unit in the error case library, and directly generate the corresponding correct terminology as the candidate;

[0167] (5) Select the most suitable option from the candidate correction words according to the context and subject logic to complete the correction.

[0168] Further, the process of constructing the subject terminology dictionary includes:

[0169] (1) Store core concepts, commonly used symbols, and formula templates according to subject classification, and mark each term with standard writing, common error forms, and occurrence scenarios;

[0170] (2) Record common collocation relationships between terms and the syntax rules of formula symbols;

[0171] (3) Collect historical recognition error data, mark error types and correction schemes, and use them as references for targeted correction.

[0172] In the present application, the entire process starts with the recognized original text data. First, the noise is removed and the units are split through preprocessing. Then, relying on the subject terminology dictionary, the suspicious error units are located, and the targeted candidate correction words are generated. Finally, the optimal result is selected through semantic coherence and subject rules, forming a complete correction logic from the recognition result to the accurate text, providing a reliable data foundation for subsequent error analysis.

[0173] The combination of the correction trace features in the question area image identifies the error question area, comprising:

[0174] S3.1: Extract red or blue correction traces in the question area image through color space threshold segmentation; the red or blue correction traces include cross, wavy line, and annotation text;

[0175] S3.2: Calculate the spatial overlap between the correction trace and the answer area. When the spatial overlap exceeds a preset threshold, mark the area as a suspected error question;

[0176] Further, the specific steps of S3.2 include:

[0177] (1) Obtain the correction trace binary image obtained by color space threshold segmentation and the coordinate range of the area in the question image, wherein in the binary image, the pixel value of the correction trace is 1, indicating that there is a trace, and the background pixel value is 0, indicating that there is no trace;

[0178] (2) Obtain the answer area binary image located by text detection and the coordinate range of the area in the question image, wherein in the binary image, the pixel value of the answer area is 1, indicating that there is answer content, and the background pixel value is 0, indicating that there is no answer content;

[0179] Wherein, it is necessary to ensure that the coordinates of the two areas are based on the same coordinate system to avoid calculation errors caused by different coordinate references.

[0180] (3) Locking the position of the overlapping part of the correction trace area and the answer area in the image through coordinate operation, including:

[0181] Calculate the intersection of coordinates: compare the coordinates of the circumscribed rectangles of the two areas to determine the boundary of the overlapping area. Specifically, the X-axis coordinate of the upper left corner of the overlapping area is the larger value of the X-axis coordinates of the upper left corners of the two areas, and the Y-axis coordinate of the upper left corner is the larger value of the Y-axis coordinates of the upper left corners of the two areas. The X-axis coordinate of the lower right corner of the overlapping area is the smaller value of the X-axis coordinates of the lower right corners of the two areas, and the Y-axis coordinate of the lower right corner is the smaller value of the Y-axis coordinates of the lower right corners of the two areas.

[0182] Determine whether there is an intersection: if the calculated X-axis coordinate of the upper left corner of the overlapping area is greater than the X-axis coordinate of the lower right corner, or the Y-axis coordinate of the upper left corner is greater than the Y-axis coordinate of the lower right corner, it means that the two areas have no spatial overlap, and the overlap degree is directly determined to be 0. Otherwise, the coordinate range is the spatial range where the two areas may overlap.

[0183] (4) Based on the binary image and the intersection range, count the number of pixels in the overlapping part of the two areas and the number of pixels covered by the two areas as a whole. The number of pixels in the overlapping part of the two areas refers to the intersection area, and the number of pixels covered by the two areas as a whole refers to the union area.

[0184] Calculate the intersection area: within the determined overlapping coordinate range, check the corresponding positions of the correction trace binary image and the answer area binary image pixel by pixel. If the pixel values of the two images at that pixel position are both 1, i.e. both the correction trace and the answer content, then that pixel belongs to the intersection area. Count the total number of all pixels that meet this condition, which is the area of the intersection area.

[0185] Calculate the union area: the union area is the total number of all pixels belonging to the correction trace area or the answer area. Its calculation method is as follows: first, count the total number of pixels in the correction trace area, i.e. the total number of pixels with a value of 1 in the correction trace binary image. Then count the total number of pixels in the answer area, i.e. the total number of pixels with a value of 1 in the answer area binary image. Then subtract the number of pixels in the intersection area to get the area of the union area.

[0186] (5) Quantify the spatial overlap degree of the two areas through the ratio of the intersection area to the union area. The higher the ratio, the more obvious the overlap between the correction trace and the answer area. The specific calculation logic is as follows: divide the area of the intersection area by the area of the union area to get the spatial overlap degree.

[0187] (6) determining whether to mark the region as a suspected wrong question according to the overlap degree, including: comparing the calculated spatial overlap degree with a preset threshold value, if the overlap degree is greater than or equal to the threshold value, it means that the overlap between the correction traces and the answer region is significant, which meets the common characteristics of wrong questions, so the answer region is marked as a suspected wrong question region; if the overlap degree is less than the threshold value, it is determined as a non-suspected wrong question region.

[0188] S3.3: performing semantic similarity calculation on the answer content of the suspected wrong question and the standard answer, when the similarity is lower than the qualified threshold, confirming that the question is a wrong question, wherein the similarity calculation uses cosine similarity, and the cosine similarity is a prior art content in the art, not the inventive scheme of the present application, and will not be described here.

[0189] The semantic similarity calculation on the answer content of the suspected wrong question and the standard answer includes:

[0190] S3.3.1: determining the type of the suspected wrong question, and dividing it into objective questions, subjective questions or formula questions;

[0191] S3.3.2: for the suspected wrong question determined as an objective question, extracting the answer result in the answer content, comparing the answer result with the corresponding standard answer at the character level, calculating the character consistency rate, and taking it as the semantic similarity of the objective question;

[0192] S3.3.3: for the suspected wrong question determined as a subjective question, obtaining the answer content and the corresponding standard answer, and inputting them into a pre-trained BERT model respectively, converting the answer content into an answer semantic vector and the standard answer into a standard semantic vector through the BERT model, calculating the cosine similarity of the answer semantic vector and the standard semantic vector, and taking it as the semantic similarity of the subjective question, wherein the BERT model is a prior art content in the art, not the inventive scheme of the present application, and will not be described here;

[0193] S3.3.4: for the suspected wrong question determined as a formula question, converting the formula text in the recognized answer content into LaTeX format answer formula, and converting the formula text in the corresponding standard answer into LaTeX format standard formula, constructing syntax trees based on the LaTeX format answer formula and the LaTeX format standard formula respectively, calculating the structural similarity by comparing the structural differences of the two syntax trees, and taking it as the semantic similarity of the formula question.

[0194] annotating the knowledge points and classifying the difficulty of the question corresponding to the wrong question region, including:

[0195] S4.1: Extract the question text corresponding to the wrong question area, input the question text corresponding to the wrong question area into a preset knowledge point classification model, and output the knowledge point label corresponding to the question text; the classification model is obtained by training the subject knowledge point corpus;

[0196] S4.2: Based on the error type of the wrong question, combined with the error rate data of the question in previous years, the difficulty coefficient of the question is calculated according to the preset calculation rule; the error type of the wrong question includes concept error, calculation error and logic error;

[0197] S4.3: Integrate the knowledge point label and the difficulty coefficient to generate a wrong question multi-dimensional attribute label containing knowledge point information and difficulty information.

[0198] The knowledge point classification model in S4.1 includes a text embedding layer, an attention mechanism layer and a classification output layer, wherein:

[0199] The text embedding layer receives the input question text, converts each word in the question text into a fixed-dimensional word vector through a preset word embedding algorithm, and forms a word vector sequence corresponding to the question text;

[0200] The attention mechanism layer receives the word vector sequence output by the text embedding layer, and performs weight calculation and distribution on each word vector in the sequence to obtain a weight-adjusted word vector sequence;

[0201] The classification output layer receives the weight-adjusted word vector sequence output by the attention mechanism layer, extracts and classifies the weight-adjusted word vector sequence through a fully connected network, outputs the probability distribution corresponding to the knowledge point label, and selects the label with the highest probability as the knowledge point classification result of the question text.

[0202] The preset calculation rule in S4.2 refers to the specific method of calculating the difficulty coefficient by combining the error type weight of the wrong question and the error rate data in previous years, including:

[0203] According to the difference in knowledge mastery reflected by different error types, a fixed weight value is preset for each error type;

[0204] Convert the error rate of the question in previous years to a standardized value between 0 and 1;

[0205] After calculating the weight value and the standardized value of the error rate of the question in previous years according to the preset proportion respectively, add the two results to obtain the difficulty coefficient;

[0206] Map the calculated difficulty coefficient to a difficulty level.

[0207] The classification and collection based on the labeling result to generate a structured wrong question set, comprising:

[0208] A1: Construct a tree-shaped classification directory according to the knowledge point tags of the wrong questions, form a preliminary classification system divided by knowledge points; the tree-shaped classification directory takes a subject category as a root node, takes each subdivided knowledge point as a subnode, and classifies wrong questions with the same knowledge point tags into the corresponding subnode directory;

[0209] A2: In each of the above knowledge point subnode directories, the difficulty coefficient of the wrong question is extracted, the wrong questions are arranged in ascending order of the difficulty coefficient, and a gradient exercise sequence is formed;

[0210] A3: For each wrong question in the gradient exercise sequence, the key error step and the corresponding correct analysis of the wrong question are extracted, the key error step and the correct analysis are associated with the corresponding question in the wrong question set, and a wrong question document with analysis is generated;

[0211] The key error step refers to the error answering part based on the correction trace positioning;

[0212] The corresponding correct analysis refers to the analysis content matched from the standard answer library;

[0213] The wrong question document with analysis contains question content, answering situation, key error step and correct analysis.

[0214] The embodiments of the application are described above in combination with the drawings, but the application is not limited to the above specific embodiments, the above specific embodiments are only illustrative and not restrictive, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments without departing from the purpose of the application and the scope of protection, which are all within the protection of the application.

[0215] If the technical solutions of the present disclosure involve personal information, the product applying the technical solutions of the present disclosure has explicitly informed the personal information processing rules before processing the personal information and has obtained the personal independent consent. If the technical solutions of the present disclosure involve sensitive personal information, the product applying the technical solutions of the present disclosure has obtained the personal independent consent before processing the sensitive personal information and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent sign is set to inform that the personal information collection range has been entered and the personal information will be collected, and if the individual voluntarily enters the collection range, it is considered to agree to collect the personal information; or on the device for processing personal information, the personal information processing rules are informed through obvious signs / information, and the personal authorization is obtained through pop-up information or by asking the individual to upload his / her personal information; wherein, the personal information processing rules can include personal information processor, personal information processing purpose, processing method and personal information type, etc.

Claims

1. An automatic mistake collection method based on video image acquisition, characterized in that, The method comprises the following steps: acquiring continuous video frames containing a test or a paper by a video acquisition device, and preprocessing the video frames to obtain a question region image; performing text detection and recognition on the question region image to extract question content and corresponding answer content, and constructing a question-answer data pair; matching the question-answer data pair with a preset standard answer library, and identifying an error question region in combination with correction trace features in the question region image; annotating the question corresponding to the error question region with knowledge points and classifying the question according to difficulty, collecting the questions based on the annotation results, generating a structured error question set, and outputting the error question set.

2. The automatic mistake collection method based on video image collection according to claim 1, wherein, The preprocessing of the video frames to obtain the question region image comprises the following steps: performing inter-frame anti-shake processing on the continuous video frames to eliminate image offset caused by shooting jitter through feature point matching, and obtaining a stable video frame sequence; extracting a paper region in the video frame from the stable video frame sequence by using an adaptive threshold segmentation algorithm to obtain an initial paper region image; performing perspective transformation correction on the initial paper region image to obtain a rectangular page image of orthographic projection; locating a question segmentation line by edge detection and Hough transformation based on the rectangular page image of orthographic projection, and segmenting an independent question region image according to the question segmentation line.

3. The method of claim 2, wherein the method further comprises: The perspective transformation correction on the initial paper region image to obtain a rectangular page image of orthographic projection comprises the following steps: performing edge detection on the initial paper region image to identify four vertex coordinates of the paper region, and calculating spatial distortion parameters of the four vertices according to the spatial positional relationship of the four vertices; constructing a perspective transformation matrix based on the spatial distortion parameters, and mapping the tilted initial paper region image to a two-dimensional orthographic plane by using the perspective transformation matrix to obtain a transformed image; performing gray scale equalization processing on the transformed image to obtain a rectangular page image of orthographic projection.

4. The method of claim 1, wherein the method further comprises: The text detection and recognition on the question region image to extract question content and corresponding answer content comprises the following steps: locating a text line in the question region by using an EAST text detection model for the question region image, and distinguishing a stem text from an answer area text by the position feature and color feature of the text line to obtain a stem text line and an answer area text line; performing character recognition on the stem text line and the answer area text line by using a CRNN model respectively, converting the image text corresponding to the stem text line into editable stem text data, and converting the image text corresponding to the answer area text line into editable answer content text data; performing semantic error correction on the recognized stem text data and answer content text data, and correcting the recognized error characters or formula symbols in combination with a preset subject terminology dictionary to finally obtain accurate question content and corresponding answer content.

5. The method of claim 1, wherein the method further comprises: The identification of the error question region in combination with the correction trace features in the question region image comprises the following steps: extracting red or blue correction traces in the question region image by color space threshold segmentation; the red or blue correction traces comprise a cross, a wavy line, and a comment word; calculating the spatial overlap degree of the correction trace and the answer area, and marking the region as a suspected error question when the overlap degree exceeds a preset threshold. The semantic similarity between the answers to suspected incorrect questions and the standard answers is calculated. If the similarity is lower than the acceptable threshold, the question is confirmed as incorrect.

6. The automatic mistake collection method based on video image collection according to claim 5, wherein, The semantic similarity calculation between the answers to suspected incorrect questions and the standard answers includes: Identify the types of questions suspected of being incorrect, and categorize them as objective questions, subjective questions, or formula-based questions; For suspected incorrect questions that are judged to be objective questions, the answer content is extracted, the answer is compared with the corresponding standard answer at the character level, the character consistency rate is calculated, and it is used as the semantic similarity of the objective question; For questions suspected of being wrong and identified as subjective questions, their answers and corresponding standard answers are obtained and input into a pre-trained BERT model. The BERT model converts the answers into semantic vectors and the standard answers into standard semantic vectors. The cosine similarity between the semantic vectors and the standard semantic vectors is calculated and used as the semantic similarity of the subjective questions. For suspected incorrect questions identified as formula-related questions, the formula text in the identified answer is converted into a LaTeX format answer formula. At the same time, the formula text in the corresponding standard answer is converted into a LaTeX format standard formula. Syntax trees are constructed based on the LaTeX format answer formula and the LaTeX format standard formula, respectively. The structural similarity is calculated by comparing the structural differences between the two syntax trees, and this is used as the semantic similarity of the formula-related questions.

7. The automatic error collection method based on video image acquisition as described in claim 1, characterized in that, The questions corresponding to the aforementioned incorrect question areas are marked with knowledge points and graded by difficulty, including: Extract the question text corresponding to the incorrect question area, input the question text corresponding to the incorrect question area into a preset knowledge point classification model, and output the knowledge point label corresponding to the question text; the classification model is obtained by training through subject knowledge point corpus. Based on the error types of incorrect questions and combined with the error rate data of the same questions over the years, the difficulty coefficient of the question is calculated according to the preset calculation rules; the error types of the incorrect questions include conceptual errors, calculation errors, and logical errors; Integrate knowledge point tags and difficulty levels to generate multi-dimensional attribute tags for incorrect questions that include both knowledge point and difficulty information.

8. The automatic mistake collection method based on video image collection according to claim 7, characterized in that, The knowledge point classification model includes a text embedding layer, an attention mechanism layer, and a classification output layer, wherein: The text embedding layer receives the input question text and converts each word in the question text into a fixed-dimensional word vector through a preset word embedding algorithm, forming a word vector sequence corresponding to the question text. The attention mechanism layer receives the word vector sequence output by the text embedding layer, calculates and assigns weights to each word vector in the sequence, and obtains a word vector sequence after weight adjustment. The classification output layer receives the weighted word vector sequence output by the attention mechanism layer, performs feature extraction and classification calculation on the weighted word vector sequence through a fully connected network, outputs the probability distribution corresponding to the knowledge point labels, and selects the label with the highest probability as the knowledge point classification result of the question text.

9. The automatic mistake collection method based on video image collection according to claim 8, wherein, The preset calculation rule refers to a specific method for calculating the difficulty coefficient using a quantitative formula, which combines the error type weights of incorrect questions with historical error rate data, including: Based on the differences in knowledge mastery difficulty reflected by different error types, a fixed weight value is pre-set for each error type; Convert the error rate of this question over the years into a standardized value between 0 and 1; The weight values ​​and the standardized values ​​of the error rates of this question over the years are calculated according to a preset ratio, and then the two results are added together to obtain the difficulty coefficient. The calculated difficulty coefficient is mapped to a difficulty level.

10. The automatic error collection method based on video image acquisition as described in claim 1, characterized in that, The process of classifying and grouping questions based on annotation results to generate a structured set of incorrect answers includes: A tree-shaped classification directory is constructed based on the knowledge point tags of the incorrect questions, forming a preliminary classification system divided by knowledge points; the tree-shaped classification directory takes the major subject category as the root node and each sub-knowledge point as the child node, and incorrect questions with the same knowledge point tags are assigned to the corresponding child node directory; Within each of the above knowledge point sub-node directories, extract the difficulty coefficient of the incorrect questions, arrange the incorrect questions in ascending order of difficulty coefficient, and form a graded practice sequence; For each incorrect question in the graded practice sequence, extract the key error steps and the corresponding correct explanations, associate the key error steps and correct explanations with the corresponding questions in the incorrect question set, and generate a document of incorrect questions with explanations. The critical error step refers to the erroneous answer section located based on the correction traces; The corresponding correct analysis refers to the analysis content matched from the standard answer database; The document containing the analyzed incorrect questions includes the question content, the answer details, key incorrect steps, and the correct explanation.

Images