Method, system and computing device for identifying handwritten answer content based on question

Abstract

The present invention discloses a method, system and computing device for identifying handwritten answer content based on a question, the method comprising: receiving an image containing handwritten answer content and a question text sequence of a question corresponding to the handwritten answer content; and using a trained handwritten answer content recognition model to identify the handwritten answer content based on the received image and the question text sequence. According to the method, system and computing device for identifying handwritten answer content based on a question of the present invention, multimodal information is used as input, and question text information can be adaptively used for auxiliary recognition, thereby effectively eliminating ambiguity when identifying handwritten answer content and improving the recognition accuracy of ambiguous symbols.

Description

Technical Field

[0001] The present invention relates to the field of text recognition, and more specifically, to a method, system and computing device for recognizing handwritten answer content based on a question. Background Art

[0002] In production and life, people have to deal with a lot of words, reports and texts. In order to reduce people's labor and improve processing efficiency, people have developed many text recognition methods.

[0003] Common text recognition methods can be divided into printed text recognition and handwriting recognition. Among them, the difficulties of printed text recognition mainly lie in the rich scenes, text tilt and distortion, and multiple font types. In addition to facing various problems in printed text recognition, handwriting recognition is also affected by writing style. Because different people have different writing styles, on the one hand, the same symbol written by different people often has certain changes in appearance, and there are large differences within the class; on the other hand, different symbols written by different people have certain similarities, and there are small differences between classes. Therefore, compared with printed text recognition, handwriting recognition is more likely to have inaccurate recognition results.

[0004] Existing text recognition methods usually use a single image as input to predict text sequences. When encountering ambiguous symbols, there is no additional auxiliary information and it is easy to make recognition errors. In general scenarios, this problem can be partially solved by built-in or external language models, because the semantic association between contexts in general scenarios is more obvious. By using language models, the recognition accuracy of ambiguous symbols can be improved to a certain extent. However, in handwritten answer scenarios, there are many types of symbols, involving a combination of numbers, English letters, Chinese characters, Greek symbols, basic mathematical symbols, and other symbols. The handwriting of different types of symbols is easily confused. For example, the handwritten 2 and z, 6 and b, 9 and q, B and β, x and × are very easy to confuse. In addition, the language model used for answer content is weaker than the language model used for general scenarios. Especially when the answer content is short, it is more difficult to correct the recognition result through the language model.

[0005] Therefore, a new method, system and computing device for identifying handwritten answer content based on questions is needed to solve the above problems. Summary of the invention

[0006] A series of simplified concepts are introduced in the Summary of the Invention, which will be further described in detail in the Detailed Description of the Invention. The Summary of the Invention does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the scope of protection of the claimed technical solution.

[0007] According to one aspect of the present invention, a method for identifying handwritten answer content based on a question is provided, the method comprising: receiving an image containing handwritten answer content and a question text sequence of a question corresponding to the handwritten answer content; and identifying a recognition result of the handwritten answer content based on the received image and the question text sequence by a trained handwritten answer content recognition model.

[0008] In one embodiment, the handwritten answer content recognition model includes a handwritten content recognition network, a question auxiliary network and a fusion network, and the method also includes: the handwritten content recognition network obtains the hidden state of each symbol of the handwritten answer content in the image based on the received image; the question auxiliary network obtains the hidden state of each symbol of the possible answer corresponding to the question based on the question text sequence; the fusion network is used to obtain the recognition result of the handwritten answer content based on the hidden state of each symbol of the handwritten answer content and the hidden state of each symbol of the possible answer.

[0009] In one embodiment, the handwritten content recognition network includes a first encoder and a first decoder, and the method further includes: the first encoder encodes the received image into an image feature vector; and the first decoder uses an attention mechanism to perform a decoding operation based on the image feature vector to obtain the hidden state of each symbol of the handwritten answer content in the image.

[0010] In one embodiment, the question auxiliary network includes a word vector extractor, a second encoder and a second decoder, and the method further includes: the word vector extractor extracts word vectors from the question text sequence; the second encoder encodes the extracted word vectors into text feature vectors; and the second decoder uses an attention mechanism to perform a decoding operation based on the text feature vector to obtain the hidden state of each symbol of the possible answer.

[0011] In one embodiment, wherein the fusion network includes a predictor and a classifier, the method further includes: the predictor obtains prediction weights based on the hidden states of each symbol of the handwritten answer content in the image and the hidden states of each symbol of the possible answers, and the classifier obtains a recognition result of the handwritten answer content based on the hidden states of each symbol of the handwritten answer content in the image, the hidden states of each symbol of the possible answers and the prediction weights.

[0012] In one embodiment, training the handwritten answer content recognition model includes: pre-training the handwritten content recognition network, the question auxiliary network and the fusion network respectively.

[0013] In one embodiment, the training of the handwritten answer content recognition model further includes: after pre-training the handwritten content recognition network, the question auxiliary network and the fusion network respectively, jointly training the handwritten content recognition network, the question auxiliary network and the fusion network.

[0014] In one embodiment, the step of pre-training the handwritten answer content recognition model includes: obtaining a training data set including a question, an image containing the handwritten answer content and its corresponding text label; inputting the image in the training data set into the handwritten content recognition network to obtain a plurality of first prediction results and the hidden states of each symbol of the handwritten answer content in the image, and obtaining a first loss function based on the plurality of first prediction results and the text label; inputting the question text sequence of the question in the training data set into the question auxiliary network to obtain a plurality of second prediction results and the hidden states of each symbol of the possible answer corresponding to the question, and obtaining a second loss function based on the plurality of second prediction results and the text label; keeping the parameters of the handwritten content recognition network and the question auxiliary network unchanged, inputting the hidden states of each symbol of the handwritten answer content in the image and the hidden states of each symbol of the possible answer into the fusion network to obtain a third prediction result, and obtaining a third loss function based on the third prediction result and the text label; and adjusting the parameters of the handwritten content recognition network, the question auxiliary network and the fusion network based on the first loss function, the second loss function and the third loss function respectively.

[0015] In one embodiment, the step of jointly training the handwritten answer content recognition model includes: obtaining a joint loss function based on the first loss function, the second loss function and the third loss function; fine-tuning the parameters of the handwritten content recognition network, the question auxiliary network and the fusion network based on the joint loss function to obtain a trained handwritten answer content recognition model.

[0016] In one embodiment, the method further includes: the first decoder uses an attention mechanism to perform a decoding operation based on the hidden state of the symbols of the handwritten answer content at a previous moment and the recognition result.

[0017] In one embodiment, the method further includes: the second decoder uses an attention mechanism to perform a decoding operation based on the hidden state of the symbol of the possible answer at a previous moment and the recognition result.

[0018] According to another embodiment of the present invention, a system for identifying handwritten answer content based on a question is provided, the system comprising: a processor for using one or more neural networks to: receive an image containing handwritten answer content and a question text sequence of a question corresponding to the handwritten answer content; and identify a recognition result of the handwritten answer content based on the received image and the question text sequence, and a memory for storing network parameters of the one or more neural networks.

[0019] According to another embodiment of the present invention, a computing device is provided, comprising: a receiving module for receiving an image containing handwritten answer content and a question text sequence of a question corresponding to the handwritten answer content; and a recognition module for identifying a recognition result of the handwritten answer content based on the received image and the question text sequence.

[0020] According to another embodiment of the present invention, a computing device is provided. The computing device includes a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the method described above.

[0021] According to yet another embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed, the method described above is executed.

[0022] According to the method, system and computing device for identifying handwritten answer content based on questions according to the embodiments of the present invention, multimodal information is used as input, and question text information can be adaptively used for auxiliary recognition, thereby effectively eliminating ambiguity in identifying handwritten answer content and improving the recognition accuracy of ambiguous symbols. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The following drawings of the present invention are used to understand the present invention as part of the present invention. The embodiments of the present invention are shown in the drawings and the description thereof is used to explain the principle of the present invention.

[0024] In the attached figure:

[0025] Figure 1 The present invention is a schematic structural block diagram of an electronic device that implements a method, system, and computing device for identifying handwritten answer content based on a question according to an embodiment of the present invention.

[0026] Figure 2 The figure is a flowchart of exemplary steps of a method for identifying handwritten answer content based on a question according to an embodiment of the present invention.

[0027] Figure 3The figure is a schematic structural block diagram of a handwritten answer content recognition model according to an embodiment of the present invention.

[0028] Figure 4 is an image of an exemplary ambiguous symbol according to an embodiment of the present invention.

[0029] Figure 5 The figure is a schematic structural block diagram of a system for identifying handwritten answer content based on a question according to an embodiment of the present invention.

[0030] Figure 6 A schematic structural block diagram of a computing device according to an embodiment of the present invention is shown.

[0031] Figure 7 A schematic structural block diagram of a computing device according to another embodiment of the present invention is shown. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0033] As mentioned above, existing text recognition methods usually use a single image as input to predict text sequences. When encountering ambiguous symbols, there is no additional auxiliary information and it is easy to make recognition errors.

[0034] Therefore, in order to improve the accuracy of text recognition, the present invention provides a method for identifying handwritten answer content based on a question, the method comprising: receiving an image containing handwritten answer content and a question text sequence of a question corresponding to the handwritten answer content; and using a trained handwritten answer content recognition model to identify a recognition result of the handwritten answer content based on the received image and the question text sequence.

[0035] According to the method for identifying handwritten answer content based on questions of the present invention, multimodal information is used as input, and question text information can be adaptively used for auxiliary recognition, thereby effectively eliminating ambiguity in identifying handwritten answer content and improving the recognition accuracy of ambiguous symbols.

[0036] The method, system and computing device for identifying handwritten answer content based on a question according to the present invention are described in detail below in conjunction with specific embodiments.

[0037] First, refer to Figure 1The electronic device 100 for implementing the method, system and computing device for recognizing handwritten answer content based on a question according to an embodiment of the present invention is described.

[0038] In one embodiment, the electronic device 100 can be, for example, a laptop computer, a desktop computer, a tablet computer, a learning machine, a mobile device (such as a smart phone, a phone watch, etc.), an embedded computer, a tower server, a rack server, a blade server, or any other suitable electronic device.

[0039] In one embodiment, the electronic device 100 may include at least one processor 102 and at least one memory 104 .

[0040] Among them, the memory 104 can be a volatile memory, such as a random access memory (RAM), a cache memory (cache), a dynamic random access memory (DRAM) (including stacked DRAM) or a high bandwidth memory (HBM), etc., or it can be a non-volatile memory, such as a read-only memory (ROM), a flash memory, 3D Xpoint, etc. In one embodiment, some parts of the memory 104 can be volatile memory, while another part can be a non-volatile memory (for example, using a two-level memory hierarchy). The memory 104 is used to store a computer program, which, when executed, can implement the client functions (implemented by the processor) and / or other desired functions in the embodiments of the present invention described below.

[0041] Processor 102 may be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, or other processing unit having data processing capabilities and / or instruction execution capabilities. Processor 102 may be communicatively coupled to any suitable number or type of components, peripherals, modules, or devices via a communication bus. In one embodiment, the communication bus may be implemented using any suitable protocol, such as Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Express (PCIe), Accelerated Graphics Port (AGP), HyperTransport, or any other bus or one or more point-to-point communication protocols.

[0042] The electronic device 100 may further include an input device 106 and an output device 108. The input device 106 is a device for receiving user input, which may include a keyboard, a mouse, a touch pad, a microphone, etc. In addition, the input device 106 may also be any interface for receiving information. The output device 108 may output various information (such as images or sounds) to the outside (such as a user), which may include one or more of a display, a speaker, etc. In addition, the output device 108 may also be any other device with an output function, such as a printer, etc.

[0043] Reference below Figure 2 A flowchart of exemplary steps of a method 200 for identifying handwritten answer content based on a question according to an embodiment of the present invention is described.

[0044] like Figure 2 As shown, the method 200 for identifying handwritten answer content based on a question may include the following steps:

[0045] In step S210: an image including handwritten answer content and a question text sequence of a question corresponding to the handwritten answer content are received.

[0046] In one embodiment, the image containing the handwritten answer content can be captured by any image capture device, such as a CMOS sensor, a CCD sensor, etc., or can be generated by a computer, which is not limited in the present invention.

[0047] In one embodiment, the question corresponding to the handwritten answer content can be retrieved from a question bank or obtained by recognizing from an image containing the question content using a print recognition model.

[0048] In step S220: the trained handwritten answer content recognition model identifies a recognition result of the handwritten answer content based on the received image and the question text sequence of the question.

[0049] refer to Figure 3 , Figure 3 FIG. 3 shows a schematic structural block diagram of a handwritten answer content recognition model 300 according to an embodiment of the present invention. Figure 3 As shown, in one embodiment, the handwriting answer content recognition model 300 may include a handwriting content recognition network 310 , a question assistance network 320 , and a fusion network 330 .

[0050] In one embodiment, the method 200 may further include: the handwritten content recognition network 310 obtains the hidden state of each symbol of the handwritten answer content in the image based on the received image containing the handwritten answer content. In one embodiment, the handwritten content recognition network 310 recognizes each symbol of the handwritten answer content on the image one by one. For example, if the image to be recognized is an image containing the handwritten answer content "120°", it can be recognized one by one in order from front to back, first recognizing the symbol "1", and after obtaining the recognition result of the symbol "1", then recognizing the symbol "2", and after obtaining the recognition result of the symbol "2", then recognizing the symbol "0", and then recognizing the symbol "°". It should be understood that it is also possible to recognize one by one in order from back to front, and the present invention is not limited to this.

[0051] In one embodiment, handwriting recognition network 310 may include a first encoder and a first decoder.

[0052] In one embodiment, the method 200 may further include: encoding, by the first encoder, the received image containing the handwritten answer content into an image feature vector Feat I , the process can be expressed by the formula as follows:

[0053] Feat I =Encoder(I)

[0054] Among them, the characteristic Feat I Size in is the height in pixels, is the width in pixels.

[0055] In one embodiment, the first encoder may be a convolutional neural network (CNN), such as VGG, ResNet, DenseNet, etc., or a recurrent neural network (RNN), a generative adversarial network (GAN), etc., which is not limited in the present invention.

[0056] In one embodiment, the method 200 may further include: receiving the above-mentioned image feature vector by the first decoder, and performing a decoding operation based on the image feature vector using an attention mechanism to obtain the hidden state of each symbol of the handwritten answer content in the image. In one embodiment, when the first decoder performs the decoding operation, it is also based on the hidden state of the symbol of the handwritten answer content in the image at the previous moment and the recognition result at the previous moment. At this time, the hidden state of the symbol output by the first decoder The following formula can be used for calculation:

[0057]

[0058] in, is the hidden state of the symbol at the previous moment, P t-1 is the recognition result of the previous moment, and [;] represents the concatenation operation.

[0059] in The following formula can be used for calculation:

[0060]

[0061]

[0062]

[0063] in, is a trainable parameter, N = H Feat ×W Feat .

[0064] In one embodiment, the first decoder can be a gated recurrent unit (GRU), or other recurrent neural networks (RNN) (for example, long short-term memory networks (LSTM)), convolutional neural networks (CNN), generative adversarial networks (GAN), etc., but the present invention is not limited to this.

[0065] In one embodiment, a classification layer may be provided after the first decoder to obtain a first prediction result for pre-training the first decoder. Exemplarily, the classification layer may be a softmax layer.

[0066] In one embodiment, the method 200 may further include: the question auxiliary network 320 obtains the hidden state of each symbol of the possible answer corresponding to the question based on the received question text sequence. In one embodiment, the question auxiliary network 320 outputs the hidden state of each symbol of the possible answer of the question one by one, wherein the possible answer is a reasonable answer to the question, but not necessarily a correct answer. For example, if the input question is "How many degrees is ∠BCD?", the possible answer of the question may be 120°, 150°, 160°, etc., but may not be 1z0°, 1200, 1500, 1b0°, etc. Taking the possible answer of 120° and outputting each symbol of the possible answer from front to back as an example, the question auxiliary network 320 first outputs the symbol "1", and after obtaining the result of the symbol "1", then outputs the symbol "2", and after obtaining the result of the symbol "2", outputs the symbol "0", and then outputs the symbol "°".

[0067] In one embodiment, the topic assistance network 320 may include a word vector extractor, a first encoder, and a first decoder.

[0068] In order to allow the question auxiliary network 320 to learn richer semantic representations, in one embodiment, the method 200 may further include: extracting word vectors from the question text by a word vector extractor. In one embodiment, any word vector model known in the art may be used to extract word vectors, such as LSA, PLSA, LDA, Word2Vector, skip-gram model, etc., which is not limited in the present invention.

[0069] In one embodiment, method 200 may further include: encoding the extracted word vector into a text feature vector by a second encoder, which may also be represented by a formula similar to that of the first encoder and will not be repeated here.

[0070] In one embodiment, the second encoder may be a convolutional neural network (CNN), such as VGG, ResNet, DenseNet, etc., and may also be a recurrent neural network (RNN), a generative adversarial network (GAN), etc., which is not limited in the present invention. In one embodiment, if the text sequence of the question is one-dimensional, and the question auxiliary network as an auxiliary network should minimize the time consumption, the second encoder may be a one-dimensional convolutional neural network.

[0071] In one embodiment, method 200 may further include: the second decoder uses an attention mechanism to perform a decoding operation based on the extracted text feature vector to obtain the hidden state of each symbol of the possible answer. In one embodiment, when the second decoder performs a decoding operation, it is also based on the hidden state of the symbol of the possible answer at the previous moment and the recognition result at the previous moment. At this time, the hidden state of each symbol of the possible answer output by the second decoder The following formula can be used for calculation:

[0072]

[0073] in, is the hidden state of the title text at the previous moment, P t-1 is the recognition result of the previous moment, and [;] represents the concatenation operation.

[0074] in The following formula can be used for calculation:

[0075]

[0076]

[0077]

[0078] in, are trainable parameters, is the length of the title text sequence after being encoded by the second encoder.

[0079] In one embodiment, the second decoder can be a gated recurrent unit (GRU), or other recurrent neural networks (RNN) (for example, long short-term memory networks (LSTM)), convolutional neural networks (CNN), generative adversarial networks (GAN), etc., but the present invention is not limited to this.

[0080] In one embodiment, a classification layer may be provided after the second decoder to obtain a second prediction result for pre-training the second decoder. Exemplarily, the classification layer may be a softmax layer.

[0081] Compared with the handwriting content recognition network 310, the question auxiliary network 320 has fewer parameters and input data, and its running time is much lower than that of the handwriting content recognition network 310, so the multimodal recognition model of the present invention is close to the efficiency of the existing single-modal recognition model. Since one question corresponds to multiple possible answers, and the text sequence of the answers is not necessarily the same, the question auxiliary network 320 learns a one-to-many mapping, and ultimately learns the fuzzy semantic relationship between the question and all possible answers, so as to assist the fusion network 330 in selecting the prediction result related to the question when there is ambiguity, which enables the question auxiliary network 320 to simultaneously solve the recognition ambiguity problem of the handwritten answer content when the handwritten answer content is correct or incorrect.

[0082] In one embodiment, method 200 may further include: receiving, by the fusion network 330, the hidden states of each symbol of the handwritten answer content and the hidden states of each symbol of the possible answers, and obtaining a recognition result of the handwritten answer content based on the hidden states of each symbol of the handwritten answer content and the hidden states of each symbol of the possible answers.

[0083] In one embodiment, the fusion network 330 may include a predictor and a classifier. The predictor is used to obtain a prediction weight γ based on the hidden state of each symbol of the handwritten answer content and the hidden state of each symbol of the possible answer. In one embodiment, the prediction weight γ can be obtained by the following formula:

[0084]

[0085] Among them, W F , b F are learnable parameters.

[0086] The classifier is used to obtain the predicted probability based on the hidden state of each symbol of the handwritten answer content and the hidden state of each symbol of the possible answer and the prediction weight γ. Specifically, the predicted probability It can be obtained by the following formula:

[0087]

[0088] in, are learnable parameters.

[0089] Among them, the predicted probability can be the same size as the dictionary, and the final recognition result is The symbol with the highest probability in the equation can be expressed as follows:

[0090]

[0091] In one embodiment, the step of obtaining the trained handwritten answer content recognition model 300 may include:

[0092] Step 1: Construct a training dataset. The training dataset may include questions and images containing handwritten answers corresponding to the questions, and may also include text labels corresponding to the images of the handwritten answers. For example, construct a dataset with M questions {topic1, topic2, …, topic M} training data set, for the mth question, collect the corresponding N m images of handwritten answers and corresponding text labels The text label can be any correct or incorrect possible answer. For example, if the question is "Compare the size: -5______-2 (fill in "<", "=" or ">")", the possible answers can include "<", "=" and ">". The present invention uses all possible answers because the question-assisted network is to learn the fuzzy semantic relationship between the question and all possible answers, rather than simply memorizing the correct answer, to prevent the question-assisted network from only working when the handwritten answer image is the correct answer.

[0093] Step 2: Build a handwritten answer content recognition model 300. The handwritten answer content recognition model 300 can be composed of three parts, namely a handwritten content recognition network 310, a question auxiliary network 320 and a fusion network 330. The handwritten content recognition network and the question auxiliary network both use an encoding-decoding structure using an attention mechanism, and the question auxiliary network uses an encoding-decoding structure using an attention mechanism to make it focus on the question area related to the current symbol to be decoded. The fusion network consists of a predictor and a classifier, wherein the predictor allows the handwritten answer content recognition module to adaptively refer to the question information.

[0094] Step 3: Train the handwritten answer content recognition model 300.

[0095] In one embodiment, training the handwritten answer content recognition model 300 may include: pre-training the handwritten content recognition network 310, the question auxiliary network 320, and the fusion network 330. In order to further make the three networks compatible with each other, in one embodiment, training the handwritten answer content recognition model 300 may also include: jointly training the handwritten content recognition network 310, the question auxiliary network 320, and the fusion network 330.

[0096] Specifically, in one embodiment, the step of pre-training the handwritten answer content recognition model 300 may include:

[0097] Step a: Input an image containing handwritten answer content in the training data set into the handwritten content recognition network 310, obtain multiple first prediction results and the hidden states of each symbol of the handwritten answer content in the image, and obtain a first loss function based on the multiple first prediction results and the text label.

[0098] Among them, in this pre-training stage, the initial hidden state of the symbol is a full 0 matrix, and the hidden state of the symbol is updated based on the text label of the previous moment. The update process can be expressed as follows:

[0099]

[0100] In one embodiment, the first loss function L R It can be obtained by the following formula:

[0101]

[0102] Since it is difficult to collect paired training data with questions and the amount of data is small, and the handwriting content recognition network requires a large amount of training data to enhance the robustness against different handwriting styles, in one embodiment, data without questions can be used as training data in the pre-training process to improve the recognition ability of the handwriting content recognition network.

[0103] Step b: Input the question text sequence of the questions in the training data set into the question auxiliary network, obtain multiple second prediction results and the hidden states of each symbol of the possible answer corresponding to the question, and obtain a second loss function based on the multiple second prediction results and the text labels.

[0104] Among them, in this pre-training stage, the initial hidden state of the symbol of the possible answer corresponding to the question is is a matrix of all zeros. The hidden state of the symbol that may be answered is updated based on the text label of the previous moment. The update process can be expressed as follows:

[0105]

[0106] Since the question-assisted network needs to learn the mapping relationship between questions and answers and the potential semantic relationship, in one embodiment, the second loss function L S It can be obtained by the following formula:

[0107]

[0108] Since training the question-assisting network does not require image data of handwritten answers, in one embodiment, question information and answer information without handwritten answers can be collected for pre-training to enrich the semantic representation capabilities of the question-assisting network.

[0109] Step c: Keep the parameters of the handwritten content recognition network and the question auxiliary network unchanged, input the hidden states of each symbol of the handwritten answer content in the image and the hidden states of each symbol of the possible answers to the question into the fusion network, obtain the third prediction result, and obtain the third loss function based on the third prediction result and the text label.

[0110] After the pre-training of the handwriting content recognition network and the question-assisted network is completed, the two networks already have strong text recognition capabilities and the ability to establish question-answer semantic relationships, but the parameters of the fusion network are still random. If joint training is performed directly, it will affect the performance of the handwriting content recognition network and the question-assisted network. Therefore, in one embodiment, the parameters of the handwriting content recognition network and the question-assisted network can be kept unchanged, and only the parameters of the fusion network can be updated.

[0111] In one embodiment, the third loss function L F It can be obtained by the following formula:

[0112]

[0113] Step d: adjusting the parameters of the handwriting content recognition network, the topic auxiliary network and the fusion network based on the first loss function, the second loss function and the third loss function respectively.

[0114] In one embodiment, after the handwriting content recognition network, the question-assisted network, and the fusion network are trained independently, all three can reach a relatively optimal convergence point. In order to make the three networks compatible with each other, the handwriting content recognition network 310, the question-assisted network 320, and the fusion network 330 can be jointly trained.

[0115] In one embodiment, the step of jointly training the handwriting content recognition network 310, the topic auxiliary network 320 and the fusion network 330 may include:

[0116] Step p: obtaining a joint loss function based on the first loss function, the second loss function and the third loss function;

[0117] In one embodiment, the joint loss function can be obtained as follows:

[0118] L=αL R +βL S +γL F

[0119] Among them, α∈[0,1), β∈[0,1), γ∈(0,1).

[0120] Step q: fine-tune the parameters of the handwriting content recognition network 310, the question auxiliary network 320 and the fusion network 330 based on the joint loss function to obtain a trained handwriting answer content recognition model.

[0121] refer to Figure 4 , Figure 4 An image showing an exemplary ambiguous symbol according to one embodiment of the present invention. Figure 4 The symbols in can be considered as ∠, C, c, <, From the image content alone, it is difficult to determine what the user actually wants to write. However, after inputting the image containing the symbol into the handwriting content recognition network of the handwriting answer content recognition model of the present invention and inputting the corresponding question into the question auxiliary network of the handwriting answer content recognition model of the present invention, the handwriting answer content recognition model of the present invention can easily determine that the user wants to write "<", while the existing single-modal recognition model can easily recognize the sample as

[0122] According to the method for identifying handwritten answer content based on a question according to an embodiment of the present invention, multimodal information is used as input, and question text information can be adaptively used for auxiliary recognition, thereby effectively eliminating ambiguity in identifying handwritten answer content and improving the recognition accuracy of ambiguous symbols.

[0123] In another embodiment, the present invention provides a system for identifying handwritten answer content based on a question. Figure 5 A schematic structural block diagram of a system 500 for identifying handwritten answer content based on a question according to an embodiment of the present invention is described. Figure 5 As shown, the system 500 for recognizing handwritten answer content based on a question may include a processor 510 and a memory 520 .

[0124] The processor 510 is used to use one or more neural networks to implement the following processing steps: receiving an image containing handwritten answer content and a question text sequence of a question corresponding to the handwritten answer content; and identifying a recognition result of the handwritten answer content based on the received image and the question text sequence.

[0125] Exemplarily, the processor 510 can be any processing device known in the art, such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, a field programmable gate array (FPGA), etc., and the present invention is not limited to this.

[0126] The memory 520 is used to store the network parameters of one or more neural networks. Exemplarily, the memory 520 may be a RAM, a ROM, an EEPROM, a flash memory or other storage technology, a CD-ROM, a digital versatile disk (DVD) or other optical disk storage device, a cassette, a magnetic tape, a magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the desired information and can be accessed by the processor 510.

[0127] The system 500 for identifying handwritten answer content based on a question according to an embodiment of the present invention can execute the method 200 for identifying handwritten answer content based on a question according to an embodiment of the present invention described above. Those skilled in the art can understand the specific operation of the system 500 for identifying handwritten answer content based on a question according to an embodiment of the present invention in combination with the above content, and for the sake of brevity, it will not be described here.

[0128] According to an embodiment of the present invention, a system for identifying handwritten answer content based on a question uses multimodal information as input, and can adaptively use question text information for auxiliary recognition, thereby effectively eliminating ambiguity when identifying handwritten answer content and improving the recognition accuracy of ambiguous symbols.

[0129] In yet another embodiment, the present invention provides a computing device. Figure 6 , Figure 6 FIG. 6 shows a schematic structural block diagram of a computing device 600 according to an embodiment of the present invention. Figure 6 As shown, the computing device 600 may include a receiving module 610 and a recognition module 620. The receiving module 610 is used to receive an image containing handwritten answer content and a question text sequence of a question corresponding to the handwritten answer content; the recognition module 620 is used to identify the recognition result of the handwritten answer content based on the received image and the question text sequence. The computing device 600 according to an embodiment of the present invention may execute the method 200 for identifying handwritten answer content based on a question according to an embodiment of the present invention as described above. Those skilled in the art may understand the specific operation of the computing device 600 according to an embodiment of the present invention in combination with the above-described content, and for the sake of brevity, it will not be repeated here.

[0130] In yet another embodiment, the present invention provides a computing device. Figure 7 , Figure 7 FIG. 7 is a schematic structural block diagram of a computing device 700 according to another embodiment of the present invention. Figure 7As shown, the computing device 700 may include a memory 710 and a processor 720, wherein the memory 710 stores a computer program, and when the computer program is executed by the processor 720, the processor 720 executes the method 200 for identifying handwritten answer content based on a question as described above.

[0131] Those skilled in the art can understand the specific operation of the computing device 700 according to the embodiment of the present invention in combination with the above contents. For the sake of brevity, the specific details are not repeated here, and only some main operations of the processor 720 are described as follows:

[0132] receiving an image including handwritten answer content and a question text sequence of a question corresponding to the handwritten answer content; and

[0133] The trained handwritten answer content recognition model identifies the handwritten answer content based on the received image and the question text sequence.

[0134] The computing device according to an embodiment of the present invention uses multimodal information as input and can adaptively use question text information for auxiliary recognition, thereby effectively eliminating ambiguity when recognizing handwritten answer content and improving the recognition accuracy of ambiguous symbols.

[0135] In another embodiment, the present invention provides a computer-readable medium having a computer program stored thereon, wherein the computer program executes the method for identifying handwritten answer content based on the question as described in the above embodiment when it is run. Any tangible, non-temporary computer-readable medium can be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CD-ROMs, DVDs, Blu-ray discs, etc.), flash memory and / or the like. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing device to form a machine, so that these instructions executed on a computer or other programmable data processing device can generate a device that implements a specified function. These computer program instructions can also be stored in a computer-readable memory, which can instruct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory can form a manufactured product, including an implementation device that implements a specified function. The computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device can provide steps for implementing the specified function.

[0136] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0137] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0138] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0139] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0140] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0141] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0142] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment, and the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for identifying handwritten answer content based on a question, characterized in that: The method comprises: receiving an image including handwritten answer content and a question text sequence of a question corresponding to the handwritten answer content; and The handwritten answer content recognition model trained by the user recognizes the handwritten answer content based on the received image and the question text sequence, wherein the handwritten answer content recognition model includes a handwritten content recognition network, a question auxiliary network and a fusion network. The step of training the handwritten answer content recognition model includes: Obtain a training dataset including questions, images containing handwritten answers, and their corresponding text labels; Inputting the images in the training data set into the handwritten content recognition network, obtaining a plurality of first prediction results and hidden states of each symbol of the handwritten answer content in the image, and obtaining a first loss function based on the plurality of first prediction results and the text label; Inputting a text sequence of questions in the training data set into the question auxiliary network, obtaining a plurality of second prediction results and hidden states of each symbol of a possible answer corresponding to the question, and obtaining a second loss function based on the plurality of second prediction results and the text label; Keeping the parameters of the handwritten content recognition network and the question auxiliary network unchanged, inputting the hidden state of each symbol of the handwritten answer content and the hidden state of each symbol of the possible answer in the image into the fusion network to obtain a third prediction result, and obtaining a third loss function based on the third prediction result and the text label; The parameters of the handwriting content recognition network, the topic auxiliary network and the fusion network are adjusted based on the first loss function, the second loss function and the third loss function respectively.

2. The method according to claim 1, characterized in that The method further comprises: The handwritten content recognition network obtains the hidden state of each symbol of the handwritten answer content in the image based on the received image; The question auxiliary network obtains the hidden state of each symbol of the possible answer corresponding to the question based on the question text sequence; The fusion network is used to obtain a recognition result of the handwritten answer content based on the hidden state of each symbol of the handwritten answer content and the hidden state of each symbol of the possible answers.

3. The method according to claim 2, characterized in that The handwriting content recognition network includes a first encoder and a first decoder, and the method further includes: encoding the received image into an image feature vector by the first encoder; The first decoder uses an attention mechanism to perform a decoding operation based on the image feature vector to obtain a hidden state of each symbol of the handwritten answer content in the image.

4. The method according to claim 2, characterized in that The topic auxiliary network includes a word vector extractor, a second encoder and a second decoder, and the method further includes: The word vector extractor extracts word vectors from the title text sequence; The second encoder encodes the extracted word vector into a text feature vector; The second decoder uses an attention mechanism to perform a decoding operation based on the text feature vector to obtain the hidden state of each symbol of the possible answer.

5. The method according to claim 2, characterized in that Wherein the fusion network comprises a predictor and a classifier, and the method further comprises: The predictor obtains a prediction weight based on the hidden state of each symbol of the handwritten answer content and the hidden state of each symbol of the possible answer in the image, The classifier obtains a recognition result of the handwritten answer content based on the hidden state of each symbol of the handwritten answer content in the image, the hidden state of each symbol of the possible answers and the prediction weight.

6. The method according to claim 2, characterized in that The training of the handwritten answer content recognition model includes: pre-training the handwritten content recognition network, the question auxiliary network and the fusion network respectively.

7. The method according to claim 6, characterized in that The training of the handwritten answer content recognition model further includes: after pre-training the handwritten content recognition network, the question auxiliary network and the fusion network respectively, jointly training the handwritten content recognition network, the question auxiliary network and the fusion network.

8. The method according to claim 1, characterized in that The step of training the handwritten answer content recognition model also includes: Obtaining a joint loss function based on the first loss function, the second loss function and the third loss function; Based on the joint loss function, the parameters of the handwriting content recognition network, the question auxiliary network and the fusion network are fine-tuned to obtain a trained handwriting answer content recognition model.

9. The method according to claim 3, characterized in that The method also includes: the first decoder uses an attention mechanism to perform a decoding operation based on the hidden state of the symbols of the handwritten answer content at a previous moment and the recognition result.

10. The method according to claim 4, characterized in that The method also includes: the second decoder uses an attention mechanism to perform a decoding operation based on the hidden state of the symbol of the possible answer at a previous moment and the recognition result.

11. A system for identifying handwritten answers based on questions, characterized in that: The system comprises: A processor configured to use one or more neural networks to: receiving an image including handwritten answer content and a question text sequence of a question corresponding to the handwritten answer content; and recognizing a recognition result of the handwritten answer content based on the received image and the question text sequence, wherein the one or more neural networks include a handwritten content recognition network, a question auxiliary network, and a fusion network, and a memory for storing network parameters of the one or more neural networks, The one or more neural networks are trained using the following steps: Obtain a training dataset including questions, images containing handwritten answers, and their corresponding text labels; Inputting the images in the training data set into the handwritten content recognition network, obtaining a plurality of first prediction results and hidden states of each symbol of the handwritten answer content in the image, and obtaining a first loss function based on the plurality of first prediction results and the text label; Inputting a text sequence of questions in the training data set into the question auxiliary network, obtaining a plurality of second prediction results and hidden states of each symbol of a possible answer corresponding to the question, and obtaining a second loss function based on the plurality of second prediction results and the text label; Keeping the parameters of the handwritten content recognition network and the question auxiliary network unchanged, inputting the hidden state of each symbol of the handwritten answer content and the hidden state of each symbol of the possible answer in the image into the fusion network to obtain a third prediction result, and obtaining a third loss function based on the third prediction result and the text label; The parameters of the handwriting content recognition network, the topic auxiliary network and the fusion network are adjusted based on the first loss function, the second loss function and the third loss function respectively.

12. A computing device, characterized in that: The computing device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method according to any one of claims 1 to 10.

13. A computer readable medium, characterized in that The computer readable medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 10 is executed.

Images