Image processing method applied to test question search
By detecting images and text in test question search, using embedding models and local feature reordering, combined with key information extraction from formula text, the problem of insufficient accuracy of image processing methods in existing technologies is solved, and higher test question recall accuracy is achieved.
Patent Information
- Application Number
- CN202510778148.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-11
- Publication Date
- 2025-09-23
AI Technical Summary
In existing test question search methods, the accuracy of image processing is insufficient, especially when considering the type of test question information to be recalled, resulting in low recall accuracy.
A method combining image detection and text detection is adopted. The images and text in the test question images are detected and identified, and recalled separately. The semantic vectors are extracted through the embedding model to recall the first test question, and re-ranking is performed based on local features. At the same time, key information is extracted from the formula text and low-level information is filtered out. Finally, a joint sorting is performed to confirm the final recall result.
The accuracy of test question search has been improved, especially when dealing with test questions containing multiple illustrations and formula texts. By merging the illustrations for retrieval, the accuracy of retrieval and the precision of recall are enhanced.
Smart Images
Figure CN120687626A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and more particularly to an image processing method applied to test question search. Background Art
[0002] Image search technology is capable of returning images to users based on their intent. Mainstream image search can be categorized as image-by-image and image-by-image. Optical Character Recognition (OCR) technology processes and recognizes images or videos containing text, extracting the text and layout information contained within them.
[0003] Currently, image search is far more convenient than text input. However, current image processing methods don't adequately consider the type of information in the test questions and still rely on simple feature matching to recall test questions, leaving room for improvement in recall accuracy.
[0004] Therefore, how to improve the accuracy of test question search is an urgent problem that those skilled in the art need to solve. Summary of the Invention
[0005] In view of this, the present invention provides an image processing method applied to test question search, which takes into account the division of formula text and question stem text, thereby improving the accuracy of the search.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] An image processing method for test question search comprises the following steps:
[0008] Obtain the image of the test question to be searched;
[0009] Performing picture detection based on the test question image to be searched, and recalling the first test question according to the picture, to obtain a picture-recall test question sequence;
[0010] Performing text recognition based on the image of the test question to be searched to filter out the question stem text and the formula text; and performing a second test question recall based on the question stem text and the formula text to obtain a text recall test question sequence;
[0011] The text recall test question sequence and the picture recall test question sequence are jointly sorted, and the final recall result is confirmed based on the sorting result.
[0012] Preferably, the joint sorting specifically includes weighted sorting of text recall results and image recall results.
[0013] Preferably, the step of detecting the image includes:
[0014] The embedding model is used to extract the semantic vector of the accompanying image, and the first test question is recalled from the vector database based on the semantic vector;
[0015] Extract local features and re-rank test questions based on recall.
[0016] Preferably, before extracting the semantic vector, the step further includes: merging multiple images through image processing;
[0017] In the subsequent processing, semantic vectors are extracted from each illustration and the merged illustrations to perform question recall, and local features are extracted and first re-ranked.
[0018] Preferably, the image detection step further includes:
[0019] The multiple picture recall results are jointly sorted, and the picture recall sequence is confirmed according to the sorting results.
[0020] Preferably, the text recognition step includes:
[0021] Obtaining the image of the test question to be searched and performing preliminary text recognition to obtain a first text;
[0022] Eliminating formula text and illustration text from the first text to obtain a second text;
[0023] Extracting key information from the formula text to obtain a third text;
[0024] A second test question recall is performed based on the second text and the third text.
[0025] Preferably, key information is extracted from the formula text, including removing the style and attributes of the formula.
[0026] Preferably, the formula text is in latex format and / or mathml format;
[0027] For formulas in LaTeX format, remove special symbols therein, including one or more of curly brackets, superscript and subscript identifiers, and fraction identifiers;
[0028] For formulas in MathML format, remove the HTML tags.
[0029] Preferably, the text recognition step further includes: performing a second reordering on the results of the second test question recall.
[0030] A test question search system, comprising:
[0031] An image acquisition module, used to obtain images of test questions to be searched;
[0032] A text detection module is used to perform text recognition based on the image of the test question to be searched, extract the question stem text and the formula text, and obtain a text recall test question sequence based on the question stem text and the formula text;
[0033] An image detection module is used to extract an image region according to the test question image to be searched and to confirm an image recall test question sequence according to the image region;
[0034] The sorting module is used to perform joint sorting based on the image and text recall results and output the final test question recall results.
[0035] Preferably, the storage medium stores a computer program, and when the computer program is executed, the above-mentioned image processing method is implemented.
[0036] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses an image processing method for test question search, which takes into account the division of formula text and question stem text, thereby improving the accuracy of the search. In addition, the present invention also takes into account the information of illustrations. Illustration recall is divided into two stages. The first stage uses global features to recall M test questions, and the second stage uses local features to reorder the results of the first round of recall; for test questions containing multiple illustrations, each illustration is recalled separately, and finally comprehensively sorted; for test questions containing multiple illustrations, multiple illustrations are merged and stored. When searching online, for test questions containing multiple illustrations, in addition to searching a single illustration, multiple illustrations are also merged and searched, thereby improving the accuracy of the search. The present invention filters out low-level information of the formula in text recall, such as structural information in the formula, which can effectively improve the accuracy of the search. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0038] Figure 1 A schematic flow chart of an image processing method for test question search provided by the present invention.
[0039] Figure 2 Schematic diagram of the offline storage process in an embodiment of the present invention. DETAILED DESCRIPTION
[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0041] Example 1
[0042] like Figure 1 and Figure 2 The embodiment of the present invention discloses an image processing method for test question search, comprising the following steps:
[0043] S1: Obtain the image of the test question to be searched.
[0044] S2: Perform picture detection based on the image of the test question to be searched, recall the first test question based on the picture of the picture, and obtain a picture-recall test question sequence.
[0045] S3: Perform text recognition based on the image of the test question to be searched, and filter out the question stem text and formula text; and recall the second test question based on the question stem text and formula text to obtain a text recall test question sequence.
[0046] S4: Jointly sort the text recall question sequence and the picture recall question sequence, and determine the final recall result based on the sorting results. During the joint sorting, the recall results in the text recall question sequence and the picture recall question sequence are weighted by a pre-set weight coefficient, and then re-sorted to obtain a final unique sequence, i.e., the final recall result.
[0047] Next, each step in this embodiment is described in detail.
[0048] For S1, in one feasible implementation, online capture can be performed using a camera, or existing images can be uploaded or temporary screenshots can be used. Furthermore, to enhance flexibility, the test image can be cropped or adaptively adjusted before the search begins, such as by zooming in, zooming out, or partially filling in. These operations enhance image quality and cover irrelevant information.
[0049] For S2, in a feasible implementation, the image area in the test question can be detected by the target detection algorithm. Among them, a model can be constructed according to the selected target detection algorithm, such as Fast R-CNN, YOLO, SSD, etc. Taking FastR-CNN as an example, the input test question image is first extracted with features through the ResNet50 backbone network, and the spatial information of the image is converted into a high-level feature representation to obtain a feature map; then the candidate region is generated in the feature map through RPN; then the candidate region features are converted into a fixed-size feature vector through RoIPooling or RoIAlign; finally, these feature vectors are classified and bounding box regressed through the fully connected layer. The targets in each bounding box are extracted to obtain multiple images.
[0050] Furthermore, the multiple detected pictures are merged, and the merged pictures and the multiple individual pictures before the merge are respectively recalled through global features. Each picture (including the merged pictures) is embedded to obtain the semantic vector of the picture, and m test questions are recalled from the vector database according to the vector similarity. The embedding model can be a model based on a convolutional neural network, such as resnet, or a model based on a Transformer architecture, such as ViT, Clip, etc. The embodiment of the present invention is to use the Dino model based on an unsupervised method, which is a model trained on millions of pictures. Each picture will generate an m-dimensional vector global feature vector.
[0051] Specifically, the Dino model preprocesses the input image (either a single image or a combined image) into equal-sized blocks. Each block is then projected into a low-dimensional vector using a linear layer. This results in multiple low-dimensional vectors corresponding to the blocks, which serve as embedding vectors. Positional encodings are then added to the embeddings to represent their spatial location within the image. The positional encodings and embeddings have the same dimensions, and the two are concatenated to fuse the positional information.
[0052] The position-encoded embedding vector enters the multi-head self-attention module. In the multi-head self-attention mechanism (MSA), the input vector is first projected into the query (query, Q), key (key, K) and value (value, V) vectors through linear transformation. Assuming the number of heads is h, the dimension of each head is d (d = D / h) and D is the dimension of the embedding vector, then for each head i, there is Where x is the input vector, is a learnable weight matrix. Then calculate the attention score Finally, we get the output of each head O i =A i V iThe outputs of the heads are concatenated and passed through a linear layer to obtain the final output of MSA.
[0053] The output of the MSA is then passed through a multi-layer perceptron (MLP) (typically consisting of two fully connected layers and a GELU activation function) for further feature transformation. Let the input of the MLP be z. The first fully connected layer transforms its dimension D to 4D, then passes through the GELU activation function, and then passes through the second fully connected layer to convert the dimension back to D.
[0054] Residual Connections and Layer Normalization: Both the MSA and MLP modules use residual connections (adding the module input to the module output) and layer normalization to help the model train and learn better. For example, after the output of the MSA module, it undergoes a residual connection and layer normalization to obtain the final output for the next layer. Similarly, after the output of the MLP module, it undergoes a residual connection and layer normalization to obtain the output of the entire Transformer encoder layer.
[0055] The output of the Transformer encoder is a characteristic feature vector corresponding to each image block. To obtain the semantic vector for the entire image, these feature vectors can be subjected to global average pooling or other aggregation methods to obtain the final multidimensional vector, namely the semantic vector. By calculating the cosine similarity of the semantic vectors, the m images closest to the semantic vector corresponding to the current image are extracted from the database as the recall result, which is the preliminary retrieval result. Images (illustrations) are associated with test questions. Recall refers to retrieving the image features in the vector library through the images, but the vector library associates the images with the test question ID. Therefore, recalling the images is equivalent to recalling the test questions.
[0056] Furthermore, after obtaining the semantic vector, re-ranking is performed based on local features.
[0057] As a feasible embodiment, SIFT can be used as a local feature extraction algorithm. The SIFT algorithm generates multiple m-dimensional local feature vectors for each image. Alternatively, shallow neural network features can be used as local features, such as the features output by the 3rd and 4th blocks of RestNet or the features output by the 3rd, 6th, and 9th blocks of the Transformer architecture.
[0058] Extract the local feature vectors of the current image and each image in the preliminary recall result, reorder them within the result according to the similarity of the local feature vectors, and obtain the final recall result of the current image.
[0059] Furthermore, the recall results of multiple matching pictures are jointly sorted, and higher coefficients are given to the pictures recalled by merged matching pictures and the pictures recalled repeatedly (the same picture is recalled by different matching pictures). The recall results of each matching picture are aggregated, weighted with the corresponding coefficients respectively, and then re-sorted. After joint sorting, the top k questions are selected as the recall results of the picture branch. Note: k << m, where m is the number recalled by global features.
[0060] For S3, as a feasible embodiment, optical character recognition (OCR) can be used to achieve preliminary text extraction and obtain the first text.
[0061] As a preferred embodiment, the stem text, formula text, and drawing text can be extracted through preliminary text extraction, and various text types are mixed together; in order to extract key information, it is necessary to remove the formula text and drawing text based on the first text to obtain the clean stem text, that is, the second text.
[0062] Specifically, remove the text information in the picture: Since during the process of storing questions in the database (offline storage module), text and pictures are stored separately, and the text does not contain the text information in the picture, in the online retrieval - text retrieval branch, the text to be retrieved cannot contain the text information in the picture. The OCR recognition result contains all the text information of the entire question picture, that is, it contains both the text information of the question itself and the text information in the picture. Given the positions of the known text and the picture (picture detection module), the text information of the question itself (stem, options, etc.) can be obtained. The OCR model will output all the text and the positions of the text in the question. The picture detection model will output the position of the picture. If the position of the text is within the position of the picture, it is determined that this text belongs to the picture and needs to be removed.
[0063] Specifically, the picture detection model can be trained based on self - collected data using YOLOv8. For text detection in OCR, PAN (Pixel - Attention Network) is used, and for recognition, CRNN (Convolutional Recurrent Neural Network) is used.
[0064] Furthermore, when generating the first text, the formula text is marked with special markers, such as two "$" symbols at the beginning and end of the formula text, to facilitate formula processing. Specifically, during training, the formula format can be expressed in LaTeX (enclosed in two "$" symbols). Through training with a large amount of data, the recognition model is equipped with formula recognition capabilities. The model algorithm can adopt the CRNN (Convolutional Recurrent Neural Network) algorithm.
[0065] As a feasible embodiment, the processing of the formula text is achieved by extracting the key information in the formula and eliminating the information of low importance. After eliminating this information, accurate search results can be obtained more effectively. Commonly used formula texts are latex format and mathml format. Whether it is latex formula or mathml, in order to express the structure of the formula, a large proportion of characters will be used, while the characters that truly reflect the formula are relatively few. Since it contains more special characters or labels, when recalling, most of the focus will be on the structure, thereby ignoring the main information, and often recalling more English test questions. By removing special characters and labels, completing the extraction of key information and filtering out low-level information, the accuracy of recall can be greatly improved. For example, for the formula y=x 2 After processing "+bx+c" and removing special symbols, the result is "y=x²+bx+c". That is, for LaText, special symbols (curly braces, superscript and subscript identifiers, fraction indicators, etc.) are removed, and for MathML, HTML tags are removed.
[0066] like Figure 2 In this embodiment, in order to support recall during online retrieval, a corresponding question bank needs to be constructed. When constructing the question bank, the main task is to complete the separate storage of text and illustrations.
[0067] The storage of text information mainly includes the following steps:
[0068] Obtain the text information of the test questions; process the formula text according to the process in the online retrieval step and merge it with the ordinary text information (the relative order remains unchanged) for ES storage.
[0069] The storage of image information mainly includes the following steps:
[0070] Obtain the test question's accompanying images and create an image list. Merge the images and perform feature extraction on each image, extracting both global and local features. These global and local features are stored. For example, a Milvus vector database is used to store global features, while a MySQL database is used to extract local features.
[0071] Example 2
[0072] Based on the same inventive concept, an embodiment of the present invention discloses a test question search system, which adopts the image detection method in Example 1 and includes an image acquisition module, a text detection module, a picture detection module and a sorting module.
[0073] An image acquisition module is used to obtain the image of the test question to be searched; a text detection module is used to perform text recognition based on the image of the test question to be searched, extract the question stem text and the formula text, and obtain a text recall test question sequence based on the question stem text and the formula text; a picture detection module is used to extract the picture area based on the image of the test question to be searched and confirm the picture recall test question sequence based on the picture area; a sorting module is used to perform joint sorting based on the picture and text recall results, and output the final test question recall result.
[0074] In this embodiment, an offline storage module is also included, which is used to obtain test question data and generate a test question bank.
[0075] Example 3
[0076] Based on the same inventive concept, an embodiment of the present invention discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed, the image processing method in embodiment 1 is implemented.
[0077] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.
[0078] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An image processing method for test question search, characterized in that: The following steps are involved: Obtain the image of the test question to be searched; Performing picture detection based on the test question image to be searched, and recalling the first test question according to the picture, to obtain a picture-recall test question sequence; Performing text recognition based on the image of the test question to be searched to filter out the question stem text and the formula text; and performing a second test question recall based on the question stem text and the formula text to obtain a text recall test question sequence; The text recall test question sequence and the picture recall test question sequence are jointly sorted, and the final recall result is confirmed based on the sorting result.
2. The image processing method for test question search according to claim 1, characterized in that: The step of image detection includes: The embedding model is used to extract the semantic vector of the accompanying image, and the first test question is recalled from the vector database based on the semantic vector; Based on the recalled test questions, local features of the corresponding images are extracted and re-ranked.
3. The image processing method for test question search according to claim 2, characterized in that: Before extracting the semantic vector, the steps further include: merging multiple images through image processing; In the subsequent processing, semantic vectors are extracted from each illustration and the merged illustrations to perform question recall, and local features are extracted and first re-ranked.
4. The image processing method for test question search according to claim 2 or 3, characterized in that: The step of image detection also includes: The multiple picture recall results are jointly sorted, and the picture recall sequence is confirmed according to the sorting results.
5. The image processing method for test question search according to claim 1, characterized in that: The text recognition step includes: Obtaining the image of the test question to be searched and performing preliminary text recognition to obtain a first text; the first text includes the formula text and the illustrated text; Eliminating the illustrated text based on the first text to obtain a second text; Recall the second test question based on the second text line.
6. The image processing method for test question search according to claim 5, characterized in that: The text recognition step also includes: Key information is extracted from the formula text, including removing the style and attributes of the formula.
7. An image processing method for test question search according to claim 5 or 6, characterized in that: Formula text is in Latex and / or MathML format; For formulas in LaTeX format, remove special symbols therein, including one or more of curly brackets, superscript and subscript identifiers, and fraction identifiers; For formulas in MathML format, remove the HTML tags.
8. The image processing method for test question search according to claim 5, characterized in that: The text recognition step further includes: performing a second reordering on the results of the second test question recall.
9. A test question search system, characterized in that: The image processing method according to any one of claims 1 to 8 is adopted, comprising: An image acquisition module, used to obtain images of test questions to be searched; A text detection module is used to perform text recognition based on the image of the test question to be searched, extract the question stem text and the formula text, and obtain a text recall test question sequence based on the question stem text and the formula text; An image detection module is used to extract an image region according to the test question image to be searched and to confirm an image recall test question sequence according to the image region; The sorting module is used to perform joint sorting based on the image and text recall results and output the final test question recall results.
10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed, the image processing method according to any one of claims 1 to 8 is implemented.