Answer image recognition method and device, computing equipment, medium and program product

Abstract

The embodiment of the invention provides an answer image recognition method and device, computing equipment, a medium and a program product. The answer image recognition method comprises the steps that subject information of a test paper image is acquired; obtaining question stem information of the test paper image; splicing the subject information and the question stem information to obtain context features; acquiring image information of handwritten answer content corresponding to the question stem information; encoding the image information to obtain an image feature sequence; and decoding the image feature sequence based on the context features to obtain an identification result for the image information.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, computing device, medium, and program product for recognizing response images. Background Technology

[0002] With the continuous development of science and technology, more and more business processes can be automated, bringing great convenience to users. For example, customer service, exam grading, and Q&A services have largely achieved automated processing, reducing human intervention. For exam grading, it's necessary to perform character recognition on the user's handwritten answers, determine the correctness of the answer based on the recognition results, and then assign a score. However, handwritten answers may have illegible or blurry handwriting, making the recognition results inaccurate, and consequently, the scoring results inaccurate.

[0003] Therefore, there is an urgent need for a recognition method for answer images to improve the recognition accuracy of handwritten answer images. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a method for recognizing response images. One or more embodiments of this specification also relate to a device for recognizing response images, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a method for recognizing a response image is provided, comprising: Obtain subject information from the exam paper image; Obtain the question stem information from the test paper image; The subject information and question stem information are concatenated to obtain contextual features; Obtain image information of the handwritten answer content corresponding to the question stem information; The image information is encoded to obtain an image feature sequence; Based on the contextual features, the image feature sequence is decoded to obtain the recognition result for the image information. According to a second aspect of the embodiments of this specification, a device for recognizing a response image is provided, comprising: The subject acquisition module is configured to acquire subject information from exam paper images; The question stem acquisition module is configured to acquire the question stem information of the test paper image; The splicing module is configured to splice the subject information and question stem information to obtain context features; The answer image acquisition module is configured to acquire image information of the handwritten answer content corresponding to the question stem information; An encoding module is configured to encode the image information to obtain an image feature sequence; The decoding module is configured to decode the image feature sequence based on the context features to obtain the recognition result for the image information.

[0006] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described method for recognizing response images.

[0007] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described method for recognizing a response image.

[0008] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described method for recognizing response images.

[0009] At least one embodiment in this specification achieves the following beneficial effects: by concatenating the subject information and question stem information of the acquired test paper image to obtain contextual features, obtaining image information of the handwritten answer content corresponding to the question stem information, encoding the image information to obtain an image feature sequence, and decoding the image feature sequence based on the contextual features to obtain the recognition result for the image information. Thus, the subject information and question stem information can be used as contextual information to assist in recognizing the image information of the handwritten answer content, avoiding the problem of recognition errors caused by multiple similarly shaped characters in the handwritten answer, and improving the accuracy of the recognition result. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating a method for recognizing response images according to one embodiment of this specification; Figure 2 This is a schematic diagram of image information of handwritten answer content provided in one embodiment of this specification; Figure 3 This is a schematic diagram of the overall process of a method for recognizing response images provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the structure of a response image recognition device provided in one embodiment of this specification; Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0011] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0012] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0013] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0014] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0015] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0016] Optical Character Recognition (OCR): It is a technology that converts printed or handwritten text from text images such as pictures and scanned documents into a machine-readable format. OCR technology can capture the characters printed on paper through electronic devices such as scanners or digital cameras, and then use pattern recognition technology to translate the shapes of these characters into computer text. It can be understood that OCR technology can convert the text in paper documents into an editable and searchable text format.

[0017] Artificial Intelligence (AI): It refers to using a computer system to simulate the intelligent performance of humans, enabling the system to complete tasks that usually require human intelligence, such as perceiving the environment, learning, reasoning, solving problems, language understanding and generation, etc.

[0018] K12: It refers to the basic education stage from kindergarten to grade 12, covering preschool education, primary school, junior high school and senior high school, and is a common abbreviation for the basic education market in the online education and educational technology industries at home and abroad.

[0019] Convolutional Neural Network (CNN): A feed-forward neural network used for image feature extraction, often used in tasks such as object detection and image classification.

[0020] Transformer: A neural network architecture based on self-attention mechanism, widely used in the fields of natural language processing and computer vision.

[0021] In general technology, the OCR recognition method only performs character recognition based on the features of the image of the student's answer itself, such as models based on the CNN or Transformer architecture. The model usually only receives the picture of the student's answer as input and outputs the recognized text content. Since the input is only a small part of the area of the student's answer, when facing moving characters or scribbled writing, such as '日' and '曰', '1' and 'l', etc., it is easy to cause confusion and the recognition accuracy is low, making it difficult to meet the requirements of subsequent automatic marking. In general technology, the student's answer is recognized through a historical multi-modal large model, but the deployment cost of the large model is high and the inference speed is slow, making it difficult to support the need for real-time large-scale marking in educational scenarios.

[0022] In this specification, a recognition method for answer images is provided. This specification also relates to a recognition device for answer images, a computing device, and a computer-readable storage medium, which will be described in detail one by one in the following embodiments.

[0023] See Figure 1 , Figure 1A flowchart illustrating a method for recognizing answer images according to an embodiment of this specification is shown. From a programming perspective, the entity executing the process can be a program mounted on a server or terminal. From a hardware perspective, the entity executing the process can be a server or a marking platform capable of recognizing answer images. Figure 1 As shown, the specific steps include:

[0024] Step 102: Obtain subject information from the exam paper image.

[0025] In this embodiment of the specification, the test paper image can be an image of the entire test paper, or an image containing a portion of the test paper content, etc. The test paper image can be obtained by taking a picture or by scanning. Subject information can be obtained from the test paper image; specifically, an OCR model can be used to identify subject information from the test paper image. Subject information can also be obtained from a database, which can store the mapping relationship between test paper information corresponding to test paper images and subjects, thereby enabling the retrieval of corresponding subject information from the database based on the test paper information represented by the test paper image. Test paper information can include at least one of test paper name information and test paper identification information. Subject information can be the subjects that users need to learn from kindergarten to high school, such as Chinese, chemistry, history, mathematics, physics, politics, etc.

[0026] Step 104: Obtain the question stem information of the test paper image.

[0027] In the embodiments of this specification, the question stem information can be obtained from the test paper image using an OCR recognition model. Alternatively, the question stem information can be based on the question stem numbers contained in the test paper image, and retrieved from the database the question stem information that has a mapping relationship with the test paper information corresponding to the test paper image. The question stem information can be the question information and problem information that the user needs to answer in the test paper image.

[0028] Step 106: Combine the subject information and the question stem information to obtain the context features.

[0029] In the embodiments of this specification, the subject information can be appended to the question stem information to obtain contextual features; alternatively, the question stem information can be appended to the subject information to obtain contextual features.

[0030] Step 108: Obtain image information of the handwritten answer corresponding to the question stem information.

[0031] In the embodiments of this specification, answer box information containing question stem information and handwritten answer content can be obtained from the test paper image using object detection. The answer box can be cropped according to the font shape in the answer box information to obtain image information representing the handwritten answer content. Alternatively, the answer box information can be obtained from the test paper image based on the question number in the question stem information. For example, the question number in the test paper image can be identified, and based on two question numbers, the answer box information of any question stem information and the corresponding handwritten answer content can be determined. Then, the answer box can be cropped according to the font shape in the answer box information to obtain image information representing the handwritten answer content.

[0032] In practical applications, image information of handwritten answers can also be extracted from exam paper images according to preset rules. Specifically, the question type information corresponding to the question stem can be identified, and image information of handwritten answers can be obtained based on the image acquisition rules corresponding to the question type information. For example, if the question type information indicates that the question stem is a fill-in-the-blank question, the position of the downward-sloping line in the answer box can be determined, and the area where the downward-sloping line is located in the exam paper image can be used as the image information of the handwritten answer; if the question type information indicates that the question stem is a multiple-choice question, the position of the parentheses in the answer box can be determined, and the area where the parentheses are located in the exam paper image can be used as the image information of the handwritten answer; if the question type information indicates that the question stem is a problem-solving question, the area in the exam paper image containing the handwritten word "solution" can be determined as the image information of the handwritten answer, and so on. The rules for obtaining image information of handwritten answers can be set according to actual needs and are not specifically limited here.

[0033] Step 110: Encode the image information to obtain an image feature sequence.

[0034] In the embodiments of this specification, a preset encoder can be used to encode image information. The preset encoder can be at least one of a VIT encoder, a DeiT encoder, a PVT encoder, and a Swing Transformer encoder.

[0035] Step 112: Decode the image feature sequence based on the context features to obtain the recognition result for the image information.

[0036] In the embodiments of this specification, contextual features can serve as constraints and prompts when decoding and recognizing image feature sequences. Specifically, when using a decoder to decode and recognize an image feature sequence to predict a certain character, and that character has multiple similar predicted characters, the corresponding contextual information can be obtained from the question stem information and subject information contained in the contextual features. This allows for the acquisition of the target predicted character that matches the corresponding context, thus obtaining the recognition result. For example, in mathematics, the probability of fractions appearing is relatively high; in physics, Greek letters may appear, etc.

[0037] It should be understood that the order of some steps in the methods described in one or more embodiments of this specification may be interchanged according to actual needs, or some steps may be omitted or deleted.

[0038] Figure 1 The method described above involves concatenating the subject information and question stem information of the acquired exam paper image to obtain contextual features. Then, it acquires the image information of the handwritten answers corresponding to the question stem information, encodes the image information to obtain an image feature sequence, and decodes the image feature sequence based on the contextual features to obtain the recognition result for the image information. This allows the subject information and question stem information to serve as contextual information, assisting in the recognition of handwritten answers and avoiding recognition errors caused by multiple similar-shaped characters in the handwritten answers, thus improving the accuracy of the recognition results.

[0039] based on Figure 1 In addition to the method described herein, this specification also provides some specific implementation methods of this method, which will be described below.

[0040] In one or more embodiments of this specification, positional encoding can be added to the concatenated information, thereby enabling the decoder to perceive the sequential relationship of characters during decoding and avoiding confusion in the recognition logic. Optionally, concatenating the subject information and question stem information to obtain contextual features may specifically include: concatenating the subject information and question stem information according to a preset concatenation template to obtain concatenated information; and adding positional encoding to the concatenated information to obtain the contextual features.

[0041] In the embodiments of this specification, the preset splicing template can be determined based on expert experience, or it can be determined based on actual needs. For example, the splicing template can be " <subject>{Subject}<\subject> <context>{The preceding text} <blank>{Text following the question stem}<\context>; If the answer is located in the middle of the question stem, the information can be divided into the text preceding and following the question stem, centered on the handwritten answer area; if the answer precedes the question stem, the text preceding the question stem can be left blank; if the answer follows the question stem, the text following the question stem can be left blank. If some results have already been identified from the image information of the handwritten answer, a splicing template can be used. <subject>{Subject}<\subject> <context>{The preceding text} <blank>The phrase "{the rest of the question stem}<\context>{the identified partial results}" is used for concatenation. If a question stem contains multiple handwritten answers, a concatenation template can be used. <subject>{Subject}<\subject> <context>{The preceding text} <blank>{Question text}<other blank> {The following text of the question}<\context>{The identified partial results}. <subject>The `<subject>` symbol can be used to indicate the start and end characters of the filling subject. <context>The `<\context>` symbol can be used to indicate the start and end characters of the text. <blank>It can be used as a placeholder for an answer.

[0042] In practical applications, the text type corresponding to the question stem information can be obtained, and the text format that maps to the text type can be determined. The question stem information is then converted into the text format to obtain the target question stem information. The subject information and the target question stem information are then concatenated according to a preset splicing template to obtain the spliced ​​information. Specifically, if the text type of the question stem information is formula text with subscripts and superscripts, then the text type corresponds to LaTeX format. The question stem information can then be converted into a LaTeX format question stem, and the LaTeX format question stem and subject information are then concatenated according to the splicing template. For example, please provide a chemical equation... After modification, the correct equation is obtained: ______. After splicing, it can be... <subject>{Chemistry}<\subject> <context>Please modify the chemical equation \stackrel{+2}{\text{Mg}} + \stackrel{0}{\text{O}}_2 \xlongequal{\text{ignition}}\stackrel{+2}{\text{Mg}}\stackrel{-2}{\text{O}} to obtain the correct equation. <blank><context>. Here, \stackrel{+2}{\text{Mg}} can represent "+2" as the oxidation state of Mg; \xlongequal{\text{ignition}} can represent an equal sign with the "ignition" reaction condition; \stackrel{-2}{\text{O} can represent "-2" as the oxidation state of O. If the text type of the question stem is regular text, the plain text format of the question stem can be used without conversion. The question stem and subject information are then concatenated to obtain the concatenated information.

[0043] In one or more embodiments of this specification, optionally, the question stem information includes a preceding text and a following text; adding positional encoding to the spliced ​​information may specifically include: obtaining a preset positional encoding for the answer area corresponding to the image information; performing a decremental encoding process on the preceding text based on the preset positional encoding to obtain a preceding text carrying a positional encoding; and performing an incremental encoding process on the following text based on the preset positional encoding to obtain a preceding text carrying a positional encoding.

[0044] In this embodiment, the position code is used to represent the position of a character within the overall question stem. The answer area can represent the area where the user writes their answer to the question stem information. The preset position code can be a base code. The position code of each character in the preceding text of the question stem can be the base code minus the distance from the character to the answer area, resulting in the preceding text of the question stem carrying the position code. For example, if the entire question stem is "Given a=___, find the value of b", where the base position code of the answer area is 10, and "=" is in the preceding text of the question stem and its distance from the answer area is 1, then the position code of "=" can be 9. The position code of each character in the following text of the question stem can be the base code plus the distance from the character to the answer area, resulting in the following text of the question stem carrying the position code. Continuing the example above, if "b" is in the following text of the question stem and its distance from the answer area is 3, then the position code of "b" can be 13. If the concatenated information contains identified partial results, then the position codes of each character in the identified partial results can be obtained by sequentially adding the first self-reading from the identified partial results.

[0045] In the embodiments of this specification, by adding positional encoding to the spliced ​​information, the model can accurately understand the logical position of the text when processing contextual features, thereby understanding the textual logic and semantics expressed by the question text.

[0046] In practical applications, after obtaining the concatenated information according to the concatenation template, each character in the concatenated information can be converted into its corresponding identifier information, resulting in an identifier sequence. Each character uniquely corresponds to one identifier. The identifier sequence is then converted into a fixed-dimensional identifier vector sequence. The positional encoding of each character is converted into a position vector. A vector-by-vector fusion process is then performed on the representation vector sequence and the position vectors to obtain contextual features containing positional information. This allows the model to identify the position of each character in the question stem and the semantic content of the question stem by examining the contextual features.

[0047] In one or more embodiments of this specification, optionally, the step of encoding the image information to obtain an image feature sequence may specifically include: segmenting the image information to obtain several image blocks; converting the several image blocks into a vector sequence containing positional information; and extracting features from the vector sequence using a VIT encoder to obtain the image feature sequence.

[0048] In the embodiments of this specification, the server can segment the image information according to a preset size to obtain several image blocks of a preset size; or, it can segment the image information according to a vertical projection method to obtain several image blocks of a preset size; thereby avoiding problems such as the model being unable to process the image information due to its large size, or having low processing efficiency.

[0049] In this embodiment, the plurality of image blocks are converted into a vector sequence containing positional information. Specifically, linear projection can be used to convert each image block into an image block vector represented by a string of numbers. The image block vector can be used to characterize features such as color, pixels, and background of the image block. The positional information of each image block in the image information is determined. Positional information is added to the image block vector to obtain a vector sequence containing positional information. Thus, when the model recognizes the vector sequence, the position of each image block can be determined.

[0050] In the embodiments of this specification, the VIT encoder can be a Transformer-structured visual encoder capable of processing image-based visual information based on a self-attention mechanism. It can encode the input vector sequence into a feature sequence containing global correlation information. The VIT encoder can extract image feature sequences from the vector sequence that can identify features such as handwriting shape and layout. Specifically, the VIT encoder can determine the correlation weight between each image block vector and other image block vectors in the vector sequence through a self-attention mechanism. The correlation weight represents the degree of correlation between each image block vector and other image block vectors. It identifies one or more image block vectors that can be integrated into a single character, obtaining multiple vectors representing a single character, thus achieving character-level visual semantic feature recognition. For example, the upper arc image block vector A of "6" will have a high correlation weight with the lower vertical line image block vector B of "6", determining that image block vector A and image block vector B belong to the same character "6". Therefore, based on the correlation weights between image block vectors, scattered image block features can be integrated into complete character features.

[0051] In practical applications, after determining multiple vectors representing individual characters, basic visual features can be captured, such as stroke edges, note thickness, and whether there have been alterations. Advanced semantic features can also be captured, such as the positional order between characters and the semantics of words composed of related characters. Furthermore, non-linear transformations can be applied to each vector to strengthen the connections between strokes, resulting in enhanced vectors and improving the accuracy of subsequent character recognition. The enhanced vectors are then concatenated according to character order to obtain an image feature sequence.

[0052] In one or more embodiments of this specification, optionally, the step of decoding the image feature sequence based on the context features to obtain a recognition result for the image information may specifically include: using a cross-attention mechanism to fuse the image feature sequence with the context features to obtain a first fused feature; using a decoder to decode the first fused feature to obtain a first predicted character; and determining the recognition result of the image information based on the first predicted character.

[0053] In the embodiments described in this specification, the cross-attention mechanism is a feature fusion technique in deep learning that establishes a correlation between two different types of features, allowing the decoder to focus on features more important to the recognition task during decoding. Contextual features can include already recognized results from the image feature sequence. For example, if the image feature sequence contains four characters and two have been recognized, these two recognized characters can be added to the context features. The server can calculate the similarity between each feature vector in the image feature sequence and the context features, assign different attention weights, and then weight-fuse the image feature sequence and context features based on these weights to form a first fused feature with image details and contextual information. The cross-attention mechanism can dynamically capture the correlation between the image feature sequence and the context features, improving the effectiveness of the fused features. For example, the context text "change speed" can guide the model to focus on features related to "change speed" in the image feature sequence.

[0054] In the embodiments of this specification, the image feature sequence and context features can be processed to unify their dimensions so that the two features have the same dimensions; the similarity matrix of the two features is calculated through a cross-attention mechanism to determine the context attention weight corresponding to each image feature vector; the image feature sequence is multiplied by the corresponding context attention weight, and then weighted and summed with the context features to obtain the first fused feature.

[0055] In the embodiments of this specification, the decoder can be at least one of a recurrent neural network decoder, a Transformer decoder, a long short-term memory network decoder, etc. The decoder can decode the first fused features represented by high-dimensional vectors into human-understandable characters, such as Chinese characters, numbers, symbols, and letters. If the image feature sequence is being decoded and recognized by the decoder for the first time, the first predicted character can be the first character of the sequence; if the image feature sequence is not being decoded and recognized by the decoder for the first time, the first predicted character can be the character following the last recognized character in sequence. For example, if a handwritten answer contains four characters and two characters have been recognized, the first predicted character can be the third character. This allows for character recognition according to their order, improving the semantic accuracy of the recognition results.

[0056] In one or more embodiments of this specification, optionally, the step of decoding the fused features using a decoder to obtain a first predicted character may specifically include: acquiring a plurality of preset characters; performing a linear mapping on the first fused features to obtain a mapping feature; the dimension of the mapping feature is the same as the dimension of the preset characters; calculating the matching degree between the mapping feature and each of the preset characters; and determining the first predicted character from the preset characters based on the matching degree.

[0057] In the embodiments of this specification, the preset characters can be determined based on expert experience, or they can be a set of all possible output characters predefined according to the requirements of the image recognition task. Each preset character is stored in the form of a fixed-dimensional vector, such as a 512-dimensional vector, a 256-dimensional vector, etc. The vector of each preset character can be generated by character embedding to ensure that subsequent matching calculations can be performed with the mapped features in dimension alignment.

[0058] In the embodiments of this specification, the first fusion feature can be converted into the dimension of a preset character vector through linear mapping, thereby changing the dimension of the feature without destroying the image visual and contextual semantic information contained in the first fusion feature, and ensuring the effectiveness of subsequent matching calculations.

[0059] In the embodiments of this specification, a preset matching algorithm can be used to calculate the matching degree between the mapping feature and each preset character vector to obtain a matching value. The preset matching algorithm can be semantic feature matching, such as calculating the cosine of the angle between the mapping feature and each preset character vector, and using the cosine as the matching value; where the larger the cosine, the higher the matching degree. The preset matching algorithm can also be numerical feature matching, calculating the straight-line distance between the mapping feature and the preset character vector in the vector space, and using the reciprocal of the straight-line distance as the matching value; where the greater the straight-line distance, the higher the matching degree. The preset character with the largest matching value can be used as the first predicted character; or, if the largest matching value is less than a preset threshold, it can be determined that the matching degree between the mapping feature and all preset characters is low, and a special label "$UNK$" or an empty character can be output as the recognition result, thereby avoiding misjudgment caused by forced prediction and avoiding inaccurate prediction results. The special label "$UNK$" can represent an unknown character. If the model determines that a character exists but fails to recognize it in a given attempt, the image features and contextual features can be fused together again for recognition. If the character cannot be recognized after a preset number of prediction attempts, it can be marked in the test paper image, and a prompt message can be generated after the marking is completed. This prompt message will then guide a human to review and mark the question, thus improving the accuracy of the marking results.

[0060] In one or more embodiments of this specification, optionally, determining the recognition result of the image information based on the first predicted character may specifically include: determining whether the first predicted character is a terminator; if the first predicted character is not a terminator, concatenating the subject information, the question stem information, and the first predicted character to obtain a context feature to be processed; using a cross-attention mechanism, fusing the image feature sequence with the context feature to be processed to obtain a second fused feature; using a decoder to decode the second fused feature to obtain a second predicted character; if the second predicted character is a terminator, using the first predicted character as the recognition result.

[0061] In the embodiments of this specification, the terminator can be a predefined special character used to mark the end of character sequence decoding, without actual semantics. The model can determine whether the image feature sequence has been fully recognized based on the recognized results contained in the context features. If recognition is incomplete, a terminator is not added; if recognition is complete, a terminator is added. This allows the server to determine whether to continue processing the handwritten answer information using the decoder based on the terminator. Alternatively, the model can determine whether to add a terminator based on the context features and the image feature sequence, such as whether the predicted character for the image feature sequence is empty. If the first predicted character is a terminator, it indicates that decoding is complete, and the recognition result can be output. If the context features contain recognized results, these recognized results can be used as the recognition result for the handwritten answer. If the context features do not contain any recognition results, a recognition result indicating no character can be output, indicating that the user has not answered the question. If the first predicted character is not a terminator, it indicates that decoding is incomplete, and further decoding is needed to generate subsequent characters, entering the context update and image feature sequence re-fusion stage.

[0062] In the embodiments of this specification, the context features to be processed contain the identified results. On the basis of retaining the initial context information, the semantics of the identified characters are added, which provides guidance for subsequent accurate prediction and ensures that semantically coherent characters are generated in the future.

[0063] In the embodiments described in this specification, if the second predicted character is still not a terminator, the prediction can continue according to the above steps until a predicted character representing a terminator is obtained; due to space limitations, the description will not be elaborated here, but can be referred to the above description.

[0064] In practical applications, if no predicted character representing the terminator is obtained, and the number of characters in the recognized results has reached a preset number, the recognition of the image information can be terminated, and the recognized results containing the preset number of characters can be output as the recognition result of the image information. Specifically, the context features may include the recognized results of some characters in the handwritten answer content in the image information; if the number of characters in the recognized results and the first predicted character reaches a preset number of characters, such as 2000 or 1800, the recognized results and the first predicted character can be used as the recognition result of the handwritten answer content, thereby avoiding excessive data volume and ensuring that the marking system can accurately mark the test information.

[0065] In one or more embodiments of this specification, optionally, obtaining the question stem information of the test paper image may specifically include: obtaining the test paper identifier corresponding to the test paper image; obtaining the question stem information identified by the test paper identifier from the preset database; or, identifying the question stem information from the test paper image using optical character recognition.

[0066] In this embodiment of the specification, the test paper identifier can be an identifier used to uniquely identify a specific test paper, and the meaning of the test paper identifier can be the same as the aforementioned test paper identifier information. The test paper identifier can be flexibly set according to actual needs, and can be a structured identifier, such as a test paper number, QR code, barcode, etc.; or it can be a semantic identifier, such as test paper name + version, school + grade + exam type, etc. The test paper identifier can be extracted from the test paper image.

[0067] In the embodiments of this specification, the question stem information stored in the preset database can be pre-processed feature information. The question stem information obtained using optical character recognition (OCR) requires further mapping and positional encoding to obtain feature information specific to the question stem. An OCR model for recognizing printed text can be selected based on the question stem information.

[0068] In practical applications, after determining the recognition result, the similarity between the semantic information represented by the recognition result and the preset semantic information of the preset result can be calculated. If the similarity is greater than or equal to a preset threshold, the handwritten answer can be determined to be correct, and the user's answer can be confirmed as correct. If the similarity is less than the preset threshold, the handwritten answer can be determined to be incorrect, and the user's answer can be confirmed as incorrect. The preset result can be the correct answer corresponding to the question stem information stored in the database.

[0069] To facilitate understanding of the identification of handwritten answers in the embodiments of this specification. Figure 2 This specification provides a schematic diagram of image information related to handwritten answers in an embodiment. For example... Figure 2 As shown, the answer is based on the question stem "Because the specific heat capacity of sand is ____ of that of water, the temperature rises faster when absorbing the same amount of heat." This answer contains several similar-looking characters, such as "N," and "small." Furthermore, the answer in the image is more similar to "N." However, combining the question stem information and the subject information "Physics," it can be helpful to determine that the user's intended written result should be an answer related to "large" or "small." This allows "N" to be excluded, confirming the handwritten content as "small," thus improving the accuracy of the recognition result. Figure 2 This is an example provided to more clearly describe how the information in the question stem and subject information helps to identify handwritten answers, and is not intended as a specific limitation.

[0070] Figure 3 The diagram illustrates the overall flow of a method for recognizing a response image provided in an embodiment of this specification, which specifically includes the following steps.

[0071] Step 302: Obtain subject information from the exam paper image.

[0072] Step 304: Obtain the question stem information from the test paper image.

[0073] Step 306: Combine the subject information and question information according to the preset splicing template to obtain the spliced ​​information.

[0074] In this embodiment, since the image information is predicted character by character, some recognized results may already exist when predicting a certain character. The preset splicing template may also include an area for splicing the recognized partial results. Thus, the preset splicing template can be used to splice subject information, question stem information, and the recognized partial results to obtain spliced ​​information. If no characters have been recognized from the image information, the area in the preset splicing template for splicing the recognized partial results can be left empty during splicing. In practical applications, the question type information corresponding to the question stem information can also be spliced. For example, if the question stem is "The small steel ball moves in a straight line on the inclined plane at a constant speed (choose from 'uniform' or 'variable speed')," and the subject is physics, then the spliced ​​result can be " <subject> physics< / subject> <type> Multiple choice fill-in-the-blank questions< / type> <context>The small steel ball on the inclined plane is doing... <blank> (Choose either 'uniform' or 'variable') Linear motion.< / blank> < / context> ".

[0075] Step 308: Add position encoding to the spliced ​​information to obtain context features.

[0076] Step 310: Obtain the image information of the handwritten answer corresponding to the information in the question stem.

[0077] In the embodiments of this specification, the order of acquiring the question stem information and the image information of the handwritten answer content is not specifically limited. The question stem information can be acquired first, followed by the image information; or, the image information can be acquired first, followed by the question stem information; or, both the question stem information and the image information can be acquired simultaneously. For example, an answer box image containing both question stem information and image information can be obtained through object detection. While cropping the image information from the answer box image, the question stem information can also be recognized from the answer box image using a printed text OCR recognition model. There is a correspondence between the question stem information and the image information, which facilitates the determination of the contextual features corresponding to the image information during OCR model recognition. The question stem information can be obtained from a preset database based on the test paper identifier and question stem number.

[0078] Step 312: Segment the image information to obtain several image blocks.

[0079] Step 314: Convert several image information into a vector sequence containing position information.

[0080] Step 316: Use the VIT encoder to extract features from the vector sequence to obtain the image feature sequence.

[0081] Step 318: Use the cross-attention mechanism to fuse the image feature sequence with the context features to obtain the fused features.

[0082] Step 320: Use the decoder to decode the fused features to obtain the predicted characters.

[0083] Step 322: Determine if the predicted character is a terminator. If not, proceed to step 306; if yes, proceed to step 324: Obtain the recognition result for the image information.

[0084] In the embodiments of this specification, if the predicted character is not a terminator, it can be determined that the image information has not been fully recognized. The predicted character can then be added to the already recognized partial results, concatenated with the subject information and question stem information to obtain concatenated information, and the prediction of the next character can continue until the predicted character becomes a terminator and recognition stops; or until the number of characters in the already recognized partial results reaches a preset number, at which point the recognition of the image information stops. The terminator does not have actual character semantics; it only serves as an identifier indicating whether character recognition has ended. If the predicted character is a terminator, it can be determined that the recognition of the image information has ended, and the partially recognized results before the terminator will be used as the recognition result; if the partially recognized results are empty, a recognition result without characters can be output. The partially recognized results can represent all the characters already recognized by the OCR model. Both the VIT encoder and decoder can be components of the OCR model. This allows the OCR model to recognize image information based on contextual features and obtain accurate recognition results.

[0085] The above one or more steps have the same or similar technical features as the aforementioned method for recognizing response images. Please refer to the detailed description of the foregoing embodiments, which will not be repeated here.

[0086] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they have not been described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0087] Through the methods described above, the embodiments of this specification can introduce subject information and question stem information as contextual features to assist in the recognition of handwritten answers, significantly improving the accuracy of the answers. This allows for accurate recognition based on contextual features even when the answer contains similar-looking characters or illegible handwriting that affects recognition. Simultaneously, it enables the recognition of handwritten answers using encoders and decoders with fewer parameters without significantly increasing computational load, achieving low cost and high efficiency. Furthermore, positional encoding preserves the relative position of the answer within the question stem information, which helps the model understand the semantic context of the answer and improves recognition accuracy. Visual hierarchical labels and subject semantic labels can also be used to improve the semantic accuracy of the recognition results and enhance grading accuracy.

[0088] Corresponding to the above method embodiments, this specification also provides an embodiment of a device for recognizing response images. Figure 4 A schematic diagram of the structure of a response image recognition device according to one embodiment of this specification is shown. Figure 4 As shown, the device includes: Subject acquisition module 402 is configured to acquire subject information from the exam paper image; The question stem acquisition module 404 is configured to acquire the question stem information of the test paper image; The splicing module 406 is configured to splice the subject information and the question stem information to obtain context features; The answer image acquisition module 408 is configured to acquire image information of the handwritten answer content corresponding to the question stem information; Encoding module 410 is configured to encode the image information to obtain an image feature sequence; The decoding module 412 is configured to decode the image feature sequence based on the context features to obtain the recognition result for the image information.

[0089] based on Figure 4 The embodiments of this specification also provide some specific implementation schemes of the method, which are described below.

[0090] Optionally, the splicing module can be specifically configured to: splice the subject information and question stem information according to a preset splicing template to obtain spliced ​​information; and add position encoding to the spliced ​​information to obtain the context features.

[0091] Optionally, the question stem information includes the preceding and following text of the question stem; the splicing module can be specifically configured to: obtain a preset position code for the answer area corresponding to the image information; perform decremental encoding processing on the preceding text of the question stem based on the preset position code to obtain the preceding text of the question stem carrying the position code; and perform incremental encoding processing on the following text of the question stem based on the preset position code to obtain the preceding text of the question stem carrying the position code.

[0092] Optionally, the encoding module can be specifically configured to: segment the image information to obtain several image blocks; convert the several image blocks into a vector sequence containing position information; and extract features from the vector sequence using a VIT encoder to obtain the image feature sequence.

[0093] Optionally, the decoding module can be specifically configured to: use a cross-attention mechanism to fuse the image feature sequence with the context features to obtain a first fused feature; use a decoder to decode the first fused feature to obtain a first predicted character; and determine the recognition result of the image information based on the first predicted character.

[0094] Optionally, the decoding module can be specifically configured to: acquire a plurality of preset characters; perform linear mapping on the first fusion feature to obtain a mapping feature; the dimension of the mapping feature is the same as the dimension of the preset characters; calculate the matching degree between the mapping feature and each preset character; and determine the first predicted character from the preset characters based on the matching degree.

[0095] Optionally, the decoding module can be specifically configured to: determine whether the first predicted character is a terminator; if the first predicted character is not a terminator, concatenate the subject information, the question stem information, and the first predicted character to obtain a context feature to be processed; use a cross-attention mechanism to fuse the image feature sequence with the context feature to be processed to obtain a second fused feature; use a decoder to decode the second fused feature to obtain a second predicted character; if the second predicted character is a terminator, use the first predicted character as the recognition result.

[0096] Optionally, the question stem acquisition module can be specifically configured to: acquire the test paper identifier corresponding to the test paper image; acquire the question stem information identified by the test paper identifier from the preset database; or, identify the question stem information from the test paper image using optical character recognition.

[0097] The above is a schematic scheme of a response image recognition device according to this embodiment. It should be noted that the technical solution of this response image recognition device and the technical solution of the response image recognition method described above belong to the same concept. For details not described in detail in the technical solution of the response image recognition device, please refer to the description of the technical solution of the response image recognition method described above.

[0098] Figure 5 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0099] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0100] In one embodiment of this specification, the above-described components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0101] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 500 can also be a mobile or stationary server.

[0102] The processor 520 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described method for recognizing the response image.

[0103] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described method for recognizing response images belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described method for recognizing response images.

[0104] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described method for recognizing response images.

[0105] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above-described method for recognizing response images belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above-described method for recognizing response images.

[0106] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described method for recognizing the response image.

[0107] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the above-described method for recognizing response images belong to the same concept. Details not described in detail in the computer program's technical solution can be found in the description of the technical solution of the above-described method for recognizing response images.

[0108] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0109] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0110] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0111] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0112] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.< / blank> < / context> < / subject> < / blank> < / context> < / subject> < / blank> < / context> < / subject> < / blank> < / context> < / subject> < / blank> < / context> < / subject>

Claims

1. A method for recognizing response images, characterized in that, include: Obtain subject information from the exam paper image; Obtain the question stem information from the test paper image; The subject information and question stem information are concatenated to obtain contextual features; Obtain image information of the handwritten answer content corresponding to the question stem information; The image information is encoded to obtain an image feature sequence; The image feature sequence is decoded based on the contextual features to obtain the recognition result for the image information.

2. The method according to claim 1, characterized in that, The step of concatenating the subject information and the question stem information to obtain contextual features specifically includes: The subject information and question stem information are spliced ​​together according to the preset splicing template to obtain the spliced ​​information; The concatenated information is encoded with positional codes to obtain the contextual features.

3. The method according to claim 2, characterized in that, The question stem information includes the preceding and following text of the question stem; adding positional encoding to the concatenated information specifically includes: Obtain the preset position code for the answer area corresponding to the image information; Based on the preset position code, the preceding text of the question stem is processed by decreasing encoding to obtain the preceding text of the question stem carrying the position code; Based on the preset position code, the text following the question stem is incremented and encoded to obtain the text preceding the question stem carrying the position code.

4. The method according to claim 1, characterized in that, The process of encoding the image information to obtain an image feature sequence specifically includes: The image information is segmented to obtain several image blocks; The aforementioned image blocks are converted into vector sequences containing location information; The image feature sequence is obtained by extracting features from the vector sequence using a VIT encoder.

5. The method according to claim 1, characterized in that, The step of decoding the image feature sequence based on the context features to obtain the recognition result for the image information specifically includes: By using a cross-attention mechanism, the image feature sequence is fused with the context features to obtain a first fused feature; The first fused feature is decoded using a decoder to obtain the first predicted character; Based on the first predicted character, the recognition result of the image information is determined.

6. The method according to claim 5, characterized in that, The step of decoding the fused features using a decoder to obtain the first predicted character specifically includes: Get a number of preset characters; The first fusion feature is linearly mapped to obtain a mapped feature; the dimension of the mapped feature is the same as the dimension of the preset character. Calculate the matching degree between the mapping feature and each of the preset characters; Based on the matching degree, the first predicted character is determined from the preset characters.

7. The method according to claim 5, characterized in that, The step of determining the recognition result of the image information based on the first predicted character specifically includes: Determine whether the first predicted character is a terminator; If the first predicted character is not a terminator, then the subject information, the question stem information, and the first predicted character are concatenated to obtain the context features to be processed. By using a cross-attention mechanism, the image feature sequence is fused with the context features to be processed to obtain a second fused feature; The second fused feature is decoded using a decoder to obtain the second predicted character; If the second predicted character is a terminator, then the first predicted character is taken as the recognition result.

8. The method according to claim 1, characterized in that, The acquisition of the question stem information from the test paper image specifically includes: Obtain the test paper identifier corresponding to the test paper image; Retrieve the question stem information identified by the test paper identifier from the preset database; or, The question stem information is identified from the test paper image using optical character recognition.

9. A device for recognizing a response image, characterized in that, include: The subject acquisition module is configured to acquire subject information from exam paper images; The question stem acquisition module is configured to acquire the question stem information of the test paper image; The splicing module is configured to splice the subject information and question stem information to obtain context features; The answer image acquisition module is configured to acquire image information of the handwritten answer content corresponding to the question stem information; An encoding module is configured to encode the image information to obtain an image feature sequence; The decoding module is configured to decode the image feature sequence based on the context features to obtain the recognition result for the image information.

10. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the response image recognition method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method for recognizing the response image as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the steps of the method for recognizing the response image as described in any one of claims 1 to 8.

Images