Title identification method and device, electronic equipment and computer storage medium

Abstract

Embodiments of the present application provide a question recognition method and device, electronic equipment and computer storage medium. The question recognition method comprises: obtaining a target image containing a question; inputting the target image into a pre-trained detection model to obtain a question region of the question and a text region in the question; encoding an image part of the question region by using a pre-trained encoder model to obtain an encoding vector of the question; compared with a scheme of identifying a question type by using a question type identification model, obtaining question type information of the question based on a matching result of the encoding vector and a pre-stored plurality of question type feature vectors, thereby improving the accuracy of the question type information; performing text recognition based on the text region in the question to obtain a text recognition result; constructing a question based on the question type information and the text recognition result to obtain a constructed question corresponding to the question in the target image, thereby improving the recognition efficiency and reducing the recognition cost.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, and particularly relate to a question recognition method and device, electronic equipment and computer storage medium. BACKGROUND

[0002] With the rapid development of online education, many teaching auxiliary products have emerged. In the teaching process, the above products can provide technical support for teachers and reduce the workload of teachers, for example: automatic correction of questions, question search, etc. can be realized. Question recognition on an image containing a question is the basis for realizing question correction and question explanation.

[0003] In the prior art, when a question is recognized, a question type recognition model is used to distinguish the type of the question in the image, the question corresponding to the type of the question is boxed with a text box, so as to determine the type of the question, and then the text boxes are merged in combination with the horizontal text line detection result, so as to perform subsequent processing on the question content in the question.

[0004] However, in the scheme of recognizing the type of the question according to the question type recognition model, when a new type of question appears in the image, new training samples related to the new type of question need to be collected, and these training samples also need to be labeled and the question type recognition model needs to be retrained, which is low in question recognition efficiency and high in cost. SUMMARY

[0005] Therefore, embodiments of the present application provide a question recognition method and device, electronic equipment and computer storage medium to overcome the defects of low question recognition efficiency and high cost in the prior art.

[0006] In a first aspect, the embodiments of the present application provide a question recognition method, which includes: acquiring a target image containing a question; inputting the target image into a pre-trained detection model to obtain a question region of the question and a text region in the question; encoding an image part of the question region through a pre-trained encoder model to obtain an encoding vector of the question, and obtaining question type information of the question based on a matching result of the encoding vector and a pre-stored plurality of question type feature vectors; performing text recognition based on the text region in the question to obtain a text recognition result; and performing question construction according to the question type information and the text recognition result to obtain a constructed question corresponding to the question in the target image.

[0007] In a second aspect, an embodiment of the present application provides a question recognition device, the device comprising: an acquisition module configured to acquire a target image containing a question; a detection module configured to input the target image into a pre-trained detection model to obtain a question region of the question and a text region in the question; a matching module configured to encode an image part of the question region by using a pre-trained encoder model to obtain an encoding vector of the question, and obtain question type information of the question based on a matching result of the encoding vector and a plurality of pre-stored question type feature vectors; a recognition module configured to perform text recognition based on the text region in the question to obtain a text recognition result; and a construction module configured to construct a question based on the question type information and the text recognition result to obtain a constructed question corresponding to the question in the target image.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the question recognition method in the first aspect or any one of the embodiments of the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides a computer storage medium having a computer program stored thereon, and the program causes a processor to implement the question recognition method in the first aspect or any one of the embodiments of the first aspect when executed.

[0010] The subject identification scheme provided by the embodiment of the application comprises the following steps: obtaining a target image containing a subject; inputting the target image into a pre-trained detection model to obtain a subject region of the subject and a text region in the subject; encoding an image part of the subject region by using a pre-trained encoder model to obtain an encoding vector of the subject, wherein the encoding is irrelevant to subject type information, the definition of the subject type is weakened, and the image part of the subject region only needs to be encoded regardless of the subject type, and even if a new subject type appears, the encoder model does not need to be retrained. Furthermore, the subject type information of the subject is obtained based on a matching result of the encoding vector and a pre-stored plurality of subject type feature vectors, compared with the scheme of identifying the subject type by using a subject type identification model, the subject type information is obtained based on the matching result of the encoding vector by using the encoder model, and the subject type identification model is not relied on, and the accuracy of the subject type information is improved. Text recognition is performed based on the text region in the subject to obtain a text recognition result; and a constructed subject corresponding to the subject in the target image is obtained by constructing the subject based on the subject type information and the text recognition result. When a new subject type appears, the subject identification method provided by the embodiment of the application does not need to collect new training samples related to the new subject type, label the training samples, and retrain the subject type identification model, and the identification efficiency is improved and the identification cost is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0012] Figure 1 A flowchart of a subject identification method provided by the embodiment of the present application;

[0013] Figure 2 A schematic diagram of a subject type provided by the embodiment of the present application;

[0014] Figure 3 A schematic diagram of a text recognition result provided by the embodiment of the present application;

[0015] Figure 4 A structure schematic diagram of a variational autoencoder model provided by the embodiment of the present application;

[0016] Figure 5 A flowchart of another subject identification method provided by the embodiment of the present application;

[0017] Figure 6A structural block diagram of a question identification device provided by an embodiment of the present application is shown in the figure;

[0018] Figure 7 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0019] In order to make the personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application should belong to the scope of protection of the present application.

[0020] It should be noted that the target in the present application is only used to represent the singular concept and is not used for limitation, and is not specifically for one, for example, target image. The first and the second in the present application are only used to distinguish the names and do not represent the sequential relationship, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features, for example, first detection branch, second detection branch, third detection branch, first loss value, second loss value, third loss value, fourth loss value, first training image sample, second training image sample. The plurality in the present application refers to two and more than two, for example, multiple question similarities.

[0021] The specific implementation of the embodiments of the present application will be further described below in combination with the drawings.

[0022] Embodiment one,

[0023] The embodiment one of the present application provides a question identification method, as shown in the figure, Figure 1 Figure 1 A flowchart of a question identification method provided by an embodiment of the present application is shown in the figure, and the question identification method comprises the following steps:

[0024] Step S101, obtaining a target image containing a question.

[0025] The question identification method in the embodiments of the present application can be applied to the identification of various types of questions, for example, the more common ones in mathematics homework: ordinary horizontal calculation questions, vertical calculation questions for simplifying the calculation process, and detached calculation questions (also called recursive equation calculation questions) that completely write out the calculation process; and more special ones: tree calculation questions, operation variant questions, etc. As shown in the figure, Figure 2 Figure 2 A schematic diagram of a question type provided by an embodiment of the present application is shown in the figure, Figure 2 ​​The middle column lists the schematic diagrams of 2.1-2.6, including six different types of questions. Alternatively, the questions in the present application can be logical operation questions, including various logical correctable conventional and unconventional question types.

[0026] Step S102, input the target image into the pre-trained detection model to obtain the question region of the question and the text region in the question.

[0027] The question region of the question represents the region where the entire question in the target image is located. The question region can include all the stem parts of a question, or the question region can include the stem part in the question and the answer part after answering; the text region in the question represents the text region corresponding to the text line in the question. As shown in Figure 3 Figure 3 A schematic diagram of a text recognition result provided by an embodiment of the present application, Figure 3 3.1 represents the text region in the question, and 3.2 represents the question region of the question. The text region corresponding to the text line in the question can include the text region of printed text, the text region of handwritten text, or the text region of printed text and handwritten text, which is not limited by the present application as long as the text region corresponding to the text line.

[0028] The detection model can be any appropriate neural network model capable of detecting the question region of the question and the text region in the question. The specific structure of the detection model used is not limited here.

[0029] Step S103, encode the image part of the question region through the pre-trained encoder model to obtain the encoding vector of the question, and obtain the question type information of the question based on the matching result of the encoding vector and the pre-stored multiple question type feature vectors.

[0030] The encoder model in the present application can be any appropriate encoder model including an encoder and a decoder part, including but not limited to a variational auto-encoder (VAE), a generative adversarial network (GAN), a regression model, etc., which is not limited by the present application. The encoder model can realize the corresponding encoding function after being trained. Specifically, the present application can realize the function of encoding the data of the image part of the question region to generate an encoding vector.

[0031] ​The encoding vector can be a high-dimensional vector, a parameter of a certain distribution, a feature map, etc. The high-dimensional vector can be understood as a multi-dimensional vector, which can effectively represent the characteristics of the question. Although the image part of the question region input into the pre-trained encoder model does not carry any information related to the question type, the encoding vector obtained by the encoder model can carry sufficient information to represent the characteristics of the question. Based on this, the encoding vector is matched with the pre-stored multiple question type feature vectors, wherein the question type feature vectors include multiple known feature vectors that can represent the question type, which are also obtained based on image features, for example, the 8 question types in the above Figure 2 correspond to 8 question type feature vectors, so that the question type of the target image can be determined by comparing the image features.

[0032] In this example, the image part of the question region is converted into an encoding vector regardless of the question type information when encoding, for example, there are 100 image parts of the question region, and 100 encoding vectors are obtained, which weakens the definition of the question type. Regardless of the question type, only the image part of the question region needs to be encoded, and when dealing with irregular question types or new question types, the encoder model does not need to be retrained, effectively reducing the development workload, improving the development efficiency, and saving the development cost.

[0033] In the scheme of identifying the question type according to the question type identification model, when a new question type appears, new training samples related to the question type need to be collected, and these training samples also need to be labeled, and the question type identification model needs to be retrained. The question type identification model is a neural network model, and the new question type is relatively rare, and a workbook may only have a few of them. For neural network models that require a large number of training samples, data collection is very difficult, and the scheme using the question type identification model can still only solve common question types in a time-consuming and laborious manner, and cannot solve all special question types. That is, the question type identification model scheme has the following problems: once a new question type is added, the question type identification model needs to be retrained; the data of special question types is difficult to collect, and the labeling cost is high, which increases the cost of model training. Therefore, compared with the scheme of identifying the question type according to the question type identification model, the present application encodes by using the encoder model, matches the encoding vector with the pre-stored multiple question type feature vectors, obtains a matching result, and thus obtains the question type information of the question, which weakens the definition of the question type. When dealing with irregular question types or new question types, only the image part of the question region needs to be encoded, and the encoder model does not need to be retrained, which effectively reduces the development workload, improves the development efficiency, and saves the development cost.

[0034] Step S104, text recognition is performed based on the text regions in the question, to obtain a text recognition result.

[0035] Optionally, the text recognition result represents a text recognition result of the text line in the question.

[0036] In the embodiments of the present application, the execution order of step S103 and step S104 is not limited. It can be understood that, in order to improve the speed of question recognition, the question type matching operation and the text recognition operation can also be performed in parallel to obtain the corresponding question type information and the corresponding text recognition result, respectively.

[0037] Step S105, according to the question type information and the text recognition result, a constructed question corresponding to the question in the target image is obtained.

[0038] According to the question type information obtained in step S103 and the text recognition result obtained in step S104, it is determined which text regions in the question are root node regions and which text regions are child node regions according to the text recognition result. Then, according to the operation rule corresponding to the question type information, the relationship between the child node and the root node is determined, and the constructed question is constructed in combination with the above obtained text recognition result, to obtain a constructed question corresponding to the question in the target image.

[0039] Exemplarily, according to the question type information and the text recognition result, one specific example of determining which text regions in the question are root node regions and which text regions are child node regions is as follows. Taking a tree-shaped calculation question as an example, it is exemplarily explained that the text regions in the question are obtained in step 103; in step 103, it is obtained that the question type of the question in the target image is a tree-shaped calculation question, and the text recognition result of the question is also obtained. It can be determined that which text regions in the tree-shaped calculation question are root node regions and which text regions are child node regions according to the text regions in the question; next, according to the operation rule of the tree-shaped calculation question: the sum of the child nodes is equal to the root node, the constructed question is constructed in combination with the above obtained text recognition result, to obtain a constructed question corresponding to the question in the target image.

[0040] In the embodiment of the present application, the question region of the question and the text region in the question are obtained through the detection model. Then, the image part of the question region is encoded by using the encoder model to obtain the encoding vector of the question. The encoding is irrelevant to the question type information, which weakens the definition of the question type. Based on the matching result of the encoding vector and the pre-stored characteristic vector of multiple question types, the question type information of the question is obtained. Then, the text recognition is performed on the text region in the question to obtain the text recognition result. The question is constructed according to the question type information, the text region and the text recognition result, and finally the constructed question is automatically obtained, and the recognition process of the question is completed. Compared with the prior art, when a new question type appears, the question recognition method of the embodiment of the present application does not need to collect new training samples related to the question type, label these training samples, and retrain the question type recognition model. Therefore, the efficiency of question recognition is improved, and the recognition cost is reduced.

[0041] The question recognition method of the embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to a server, a PC, even a high-performance mobile terminal and the like.

[0042] Embodiment two,

[0043] The second embodiment of the present application is based on the scheme of the first embodiment. In an embodiment of the present application, step S103 can be implemented as step S103a and step S103b.

[0044] In step S103a, the target image is captured according to the question region to obtain the image part of the question region.

[0045] Before encoding by using the encoder model, the target image is captured according to the question region to obtain the image part of the question region, and then the image part of the question region is encoded. This can avoid the problem of large data calculation amount and low efficiency in vector encoding caused by inputting other irrelevant regions in the target image except the question region into the encoder model.

[0046] In step S103b, the image part of the question region is input into the pre-trained encoder model to obtain the encoding vector of the question.

[0047] Optionally, in an embodiment of the present application, the encoder model is a variational autoencoder model.

[0048] As shown in Figure 4 , the encoder model is a variational autoencoder model. Figure 4A structural diagram of a variational autoencoder model provided in an embodiment of the present application is shown in FIG. 1. The variational autoencoder is a generative model, which includes an encoder and a decoder. Optionally, the encoder is composed of 8 layers of convolution, and the decoder is composed of 8 layers of deconvolution. Each layer of the encoder is connected to a corresponding layer of the decoder (U-Net structure). An input image is encoded by the encoder to obtain an encoding vector, which is then input into the decoder to obtain an output image. The output image is the same as the input image. Although the quality of the generated image output by the variational autoencoder is often not as good as that of the generative adversarial network and the regression model, the output image is very similar to the input image and can be used to restore the original input image. Compared with other generative models, the variational autoencoder is more stable and faster to train.

[0049] Optionally, in an embodiment of the present application, step S104 can be implemented as step S104a and step S104b.

[0050] Step S104a: calculating the question similarity between the encoding vector of the question and the pre-stored question type feature vectors to obtain a plurality of question similarities corresponding to the question region.

[0051] The question similarity in the present application represents the similarity of the question types between two questions. Taking the encoding vector as a multi-dimensional vector as an example, the question similarity can be represented by the cosine distance when matching the similarity between the encoding vectors. The question similarity is obtained by calculating the cosine distance between the encoding vector of the question and the question type feature vector. The calculation method of the cosine similarity is simple and can clearly indicate the similarity between the vectors. It can be understood that the question similarity can also be a similarity represented by the Euclidean distance or a similarity represented by the norm, which is not limited in the present application.

[0052] Step S104b: if the plurality of question similarities include a question similarity greater than the preset similarity, determining the question type information of the question type feature vector corresponding to the maximum question similarity in the plurality of question similarities as the question type information of the question.

[0053] The preset similarity can be set by a person skilled in the art according to experience or actual conditions. If the plurality of question similarities include a question similarity greater than the preset similarity, it indicates that the pre-stored question type feature vectors have a relatively high possibility of being similar to the question type information of the question. In the present example, the question type information of the question type feature vector corresponding to the maximum question similarity is determined as the question type information of the question, which improves the accuracy of the question type information of the question.

[0054] If the plurality of question similarities are all less than or equal to the preset similarity, the encoding vector of the question is added to the pre-stored plurality of question type feature vectors.

[0055] If the plurality of question similarities are all less than or equal to the preset similarity, it indicates that the possibility of the pre-stored plurality of question type feature vectors having similar question type information of the question is small, and the question type of the question can be a new question type. In the example, the encoding vector of the question is added to the pre-stored plurality of question type feature vectors as a new question type, and the construction strategy of the question when constructing the question and the judgment strategy when correcting the question according to the constructed question need to be added, only a small amount of work needs to be added, and the encoder model does not need to be retrained, thereby effectively reducing the development workload, improving the development efficiency, and saving the development cost.

[0056] In an implementable manner, before the encoding vector corresponding to the new question type is added to the pre-stored plurality of question type feature vectors, an image sample of the same new question type can also be obtained; the image sample of the same new question type is input into the pre-trained encoder model to obtain a plurality of encoding vectors; the average of the plurality of encoding vectors is calculated, and the average of the plurality of encoding vectors is taken as the encoding vector of the new question type, and then added to the pre-stored plurality of question type feature vectors, thereby improving the accuracy of the encoding vector of the new question type, so that more accurate matching results can be obtained when the same question type of the question is encountered next time.

[0057] In the example, the encoder model is used for encoding, the question similarity between the encoding vector and the pre-stored plurality of question type feature vectors is calculated, the question type information of the question type feature vector corresponding to the question similarity greater than the preset similarity and the maximum question similarity is determined as the question type information of the question, thereby improving the accuracy of the question type information of the question and weakening the definition of the question type. No matter what question type appears, only the image part of the question area needs to be encoded, and then the question similarity is calculated, so that when a non-conventional question type is processed or a new question type appears, the encoder model does not need to be retrained, thereby effectively reducing the development workload, improving the development efficiency, and saving the development cost.

[0058] It should be noted that in the embodiments of the present application, the execution order of steps S104b and S104c is not limited, and it can be understood that steps S104b and S104c can also be executed in parallel in order to improve the speed of question recognition.

[0059] Embodiment three,

[0060] In the scheme of the first embodiment and the second embodiment, the encoder model is pre-trained through the following steps: a first training image sample containing a sample question is obtained, the sample question in the first training image sample belongs to a same question type; the first training image sample is input into an initial encoder model to obtain a question prediction encoding vector of the sample question; the question prediction encoding vector of the sample question is input into an initial decoder model to obtain a sample prediction question; a first loss value is obtained by using an L1 loss function according to the sample prediction question; and the initial encoder model and the initial decoder model are trained according to the first loss value to obtain the encoder model.

[0061] The initial encoder model and the initial decoder model are trained according to the first loss value until a training termination condition is reached, for example, the number of training reaches a preset number, or the loss value reaches a preset threshold, and so on, to obtain the encoder model and the decoder model. After the training of the encoder model and the decoder model is completed, only the encoder model is used in the application stage.

[0062] When the first training image sample containing the sample question is obtained, it can be obtained in the following manner: a training image sample set is obtained, wherein the image samples in the training image sample set are image samples corresponding to a plurality of different question types; a question type is determined from the plurality of different question types, and the image sample corresponding to the question type is determined as the first training image sample.

[0063] By determining a question type from the plurality of different question types and determining the image sample corresponding to the question type as the first training image sample, it is ensured that the sample question in the first training image sample belongs to the same question type, which provides training samples for training the encoder model and improves the accuracy of the encoder model obtained after the training is completed.

[0064] After the training of the encoder model is completed, the application further includes a step of obtaining a plurality of pre-stored question type feature vectors, for example, the first training image sample is input into the pre-trained encoder model to obtain an encoding vector corresponding to the first training image sample; an average value of the encoding vector corresponding to the first training image sample is calculated; the average value of the encoding vector is taken as a question type feature vector of the first training image sample; and the plurality of pre-stored question type feature vectors are determined according to the question type feature vector of the first training image sample.

[0065] The sample question in the first training image sample belongs to the same question type, and in this example, the average value of the encoding vector of the first training image sample is calculated as the question type feature vector of the question type, which improves the accuracy of the question type feature vector. A variety of different question type feature vectors can be obtained, and a question type feature vector library can be constructed, which is used as a pre-stored plurality of question type feature vectors.

[0066] Embodiment Four,

[0067] In the embodiment four, step S102 can also be implemented as the following steps based on the schemes in the embodiments one to three.

[0068] The target image is input into the pre-trained detection model, and the question region of the question is obtained through the first detection branch of the detection model; the text region of the printed text in the question is obtained through the second detection branch of the detection model; the text region of the handwritten text in the question is obtained through the third detection branch of the detection model, and the text region of the question includes the text region of the printed text in the question and the text region of the handwritten text in the question.

[0069] The target image in the present application can contain printed text and handwritten text. The printed text can be the stem part of the question, and the handwritten text can be the answer part after the student or the question answerer answers according to the stem part.

[0070] The present example takes the target image containing printed text and handwritten text as an example for illustration, and the detection model can be any model with the first detection branch, the second detection branch and the third detection branch. The specific structure of the detection model used is not limited here.

[0071] The detection model can be a CenterNet model improved to have the three detection branches described above. The existing CenterNet model usually has only one detection head, that is, can only perform one type of detection. In the embodiments of the present application, the existing CenterNet model can be improved as follows: two detection heads are added on the basis of the original. That is, the original CenterNet model with one detection head is improved to a CenterNet model with three detection branches, so that three regions can be obtained by the improved CenterNet model: the question region, the text region of the printed text, and the text region of the printed text. The structure of the CenterNet model is relatively simple, and only the full convolution method is needed to realize the detection of the target without the need for complex operations such as NMS (Non-Maximum Suppression). Therefore, using the CenterNet model improved based on the existing CenterNet model and having the three detection branches described above as the detection model can improve the detection efficiency while ensuring the detection accuracy.

[0072] It can be understood that the detection model can also be an SSD model having the three detection branches described above, or the detection model can also be another model having the three detection branches described above, and the like. In the training process, the three detection branches described above can use the same loss function and training method.

[0073] The question region of the question can be obtained by the first detection branch of the detection model, that is, the region where the entire question in the target image is located. The text region of the printed text in the question can be obtained by the second detection branch of the detection model, that is, the text region where each printed text in the question is located. The text region of the handwritten text in the question can be obtained by the third detection branch of the detection model, that is, the text region where each handwritten text in the question is located.

[0074] In the embodiments of the present application, three types of detection are performed simultaneously by one detection model, and three regions are obtained: the question region, the text region of the printed text, and the text region of the handwritten text. From the principle of model calculation, since the three types of detection have certain commonality in the demand for input features, compared with the way of using different detection models to obtain the above regions, the way of using one detection model to obtain the above three regions in the embodiments of the present application can effectively improve the detection efficiency.

[0075] In addition, compared with a manner of simultaneously performing text detection and question type classification by using the same model, the detection model in the embodiment of the present application does not need to distinguish different question types when performing region detection, weakens the distinction of question types, and can improve the accuracy and speed of detection.

[0076] The question recognition method in the embodiment of the present application is implemented based on a pre-trained detection model. For the convenience of description, the detection model is briefly described as follows. In the example, the detection model includes: a feature extraction part; a first detection branch, a second detection branch and a third detection branch connected in parallel after the feature extraction part. The step S102 can also be specifically implemented as steps S102a and S102d.

[0077] The step S102a inputs the target image into the pre-trained detection model, and extracts features of the target image by the feature extraction part in the detection model to obtain image features corresponding to the target image.

[0078] Optionally, in an embodiment of the present application, the feature extraction part is a residual network including a plurality of residual blocks; and the step S102a can be implemented as the following steps: inputting the target image into the residual network in the pre-trained detection model, respectively obtaining an initial image feature corresponding to the target image by each residual block in the residual network; and performing feature fusion on the plurality of initial image features to obtain the image features.

[0079] For example, the detection model can be a CenterNet model, and the feature extraction part can be a Resnet18 composed of four residual blocks. The target image is input into the Resnet18 in the pre-trained CenterNet model, and an initial image feature corresponding to the target image is respectively obtained by each residual block in the Resnet18, for example: the first residual block outputs an initial feature map with a size of 1 / 4 of the target image; the second residual block outputs an initial feature map with a size of 1 / 8 of the target image; the third residual block outputs an initial feature map with a size of 1 / 16 of the target image; and the fourth residual block outputs an initial feature map with a size of 1 / 32 of the target image.

[0080] Since the sizes of the four initial feature maps are different, in order to facilitate the fusion of the initial feature maps, the four initial feature maps can be first respectively subjected to up-sampling processing to obtain initial feature maps of the same size (for example, the sizes of the feature maps obtained after up-sampling processing can all be 1 / 4). Then, the feature maps subjected to the up-sampling processing are subjected to feature fusion, for example, the elements at the same positions in the four feature maps are respectively subjected to multiplication operation to obtain a fused image feature. The fused image feature can be taken as the input of the three detection branches of the detection model, and the question region, the text region of the printed text in the question and the text region of the handwritten text in the question are respectively obtained through the three detection branches.

[0081] Step S102b, a question region is predicted based on the image feature through the first detection branch to obtain the question region of the question; step S102c, a printed text region is predicted based on the image feature through the second detection branch to obtain the text region of the printed text in the question; and step S102d, a handwritten text region is predicted based on the image feature through the third detection branch to obtain the text region of the handwritten text in the question.

[0082] The fused image feature is input into the three detection branches of the detection model, and each detection branch can output a set of feature maps with a channel number of 6, which respectively represent a score map of the center point (the value of each pixel point in the score map is between 0 and 1, representing the probability that the pixel point is the center point of the object), the offset of the center point (one is the X-axis offset and the other is the Y-axis offset), the length and width of the rectangular region corresponding to the center point, and the rotation angle of the rectangular region corresponding to the center point.

[0083] Specifically, the first detection branch outputs a feature map with 6 channels, which respectively represent a score map of the question region center point, an offset of the question region center point, a length and a width of a rectangular region corresponding to the question region center point, and a rotation angle of the rectangular region corresponding to the question region center point. Based on the feature map with 6 channels output by the first detection branch, a question region is predicted to obtain the question region of the question. The second detection branch outputs a feature map with 6 channels, which respectively represent a score map of the text region center point of the printed text, an offset of the text region center point of the printed text, a length and a width of a rectangular region corresponding to the text region center point of the printed text, and a rotation angle of the rectangular region corresponding to the text region center point of the printed text. Based on the feature map with 6 channels output by the second detection branch, a printed text region is predicted to obtain the text region of the printed text in the question. The third detection branch outputs a feature map with 6 channels, which respectively represent a score map of the text region center point of the handwritten text, an offset of the text region center point of the handwritten text, a length and a width of a rectangular region corresponding to the text region center point of the handwritten text, and a rotation angle of the rectangular region corresponding to the text region center point of the handwritten text. Based on the feature map with 6 channels output by the third detection branch, a handwritten text region is predicted to obtain the text region of the handwritten text in the question.

[0084] The step S104 can also be implemented as the following steps: based on the text region of the printed text and the text region of the handwritten text, respectively performing text recognition through the recognition model trained in advance to obtain corresponding printed text recognition result and handwritten text recognition result; and the encoder model and the recognition model are two models executed in parallel.

[0085] In the embodiment of the present application, two models executed in parallel are adopted to perform the encoding, matching operation and text recognition operation of the question in parallel. Since the models performing the above two operations are executed in parallel, the question type information, the printed text recognition result and the handwritten text recognition result can be obtained at the same time, and therefore, the efficiency of question recognition can be further improved in the embodiment of the present application.

[0086] In the embodiment of the present application, the recognition model can be any recognition model, and the specific structure of the recognition model adopted is not limited herein, for example, a long short-term memory neural network (LSTM), a recurrent neural network (RNN), and a convolutional recurrent neural network (CRNN).

[0087] Optionally, in an embodiment of the present application, the step S104 can be further implemented as the following steps S104a and S104b.

[0088] The step S104a, according to the text area of the printed text and the text area of the handwritten text, respectively, the target image is screenshot, and the corresponding printed text area image and handwritten text area image are obtained; the step S104b, the printed text area image and the handwritten text area image are respectively input into the pre-trained recognition model, and text recognition is performed to obtain the corresponding printed text recognition result and handwritten text recognition result.

[0089] Before text recognition, the target image is screenshot according to the text area of the printed text and the text area of the handwritten text, and the printed text area image and the handwritten text area image are obtained. Then, the printed text area image and the handwritten text area image are respectively input into the pre-trained recognition model for text recognition, which can avoid the problem of large data calculation amount and low recognition efficiency in the text recognition process caused by inputting other irrelevant areas in the target image except the text area into the recognition model.

[0090] Embodiment five,

[0091] In the embodiments one to four of the present application, the detection model is pre-trained through the following steps S201-S205. The step S201, a second training image sample containing a sample question, a sample question area of the sample question in the second training image sample and a text area in the sample question are obtained; the step S202, the second training image sample is input into an initial detection model to obtain a sample question prediction area and a text prediction area in the sample question; based on the sample question prediction area and the sample question area, a second loss value is obtained; the step S203, based on the text prediction area and the text area in the sample question, a text loss value is obtained; the step S204, the second loss value and the text loss value are numerically fused to obtain a fused loss value; the step S205, according to the fused loss value, the initial detection model is trained to obtain the detection model.

[0092] According to the fused loss value, the initial detection model is trained until the training termination condition is reached, for example, the training number reaches a preset number, or the loss value reaches a preset threshold, etc., to obtain the detection model. In the above training process, for each detection branch, the loss value corresponding to the detection branch is equal to the sum of the loss values of the six channels, wherein, for the channel representing the score map of the center point, the loss value of the channel can be calculated by the Focal Loss loss function, and the loss values of the remaining channels can be calculated by the L1 smooth loss function.

[0093] Since the sample question prediction region output in the detection model and the text prediction region in the sample question represent different meanings respectively, in order to balance the influence of the second loss value and the text loss value on the loss value of the entire detection model, the weight coefficients can be set for the second loss value and the text loss value respectively, and the second loss value and the text loss value are correspondingly fused to obtain a fusion loss value. The fusion manner can be realized by the following manner: the product of the second loss value and the first weight coefficient, the product of the text loss value and the text weight coefficient are calculated respectively, and the two products are added to obtain the fusion loss value. The fusion loss value is the loss value of the entire detection model.

[0094] The first weight coefficient and the text weight coefficient can be set according to experience or actual situation, and the specific values of the first weight coefficient and the text weight coefficient are not limited in the embodiment of the application. For example, the first weight coefficient can be set to 5, and the text weight coefficient can be set to 1.

[0095] The text region in the sample question includes the text region of the printed text in the sample question and the text region of the handwritten text in the sample question; step S202 is realized by the following steps: obtaining the sample question prediction region through the first detection branch of the initial detection model; obtaining the text prediction region of the printed text in the sample question through the second detection branch of the initial detection model; obtaining the text prediction region of the handwritten text in the sample question through the third detection branch of the initial detection model; the first detection branch corresponds to the second loss value; step S203 is realized by the following steps: based on the text prediction region of the printed text in the sample question and the text region of the printed text in the sample question, a third loss value corresponding to the second detection branch is obtained; based on the text prediction region of the handwritten text in the sample question and the text region of the handwritten text in the sample question, a fourth loss value corresponding to the third detection branch is obtained, and the text loss value includes the third loss value and the fourth loss value.

[0096] The detection results output by the three detection branches in the detection model represent different meanings respectively, and based on the same reason as above, the second loss value, the third loss value and the fourth loss value are fused in the example to obtain a fusion loss value. The fusion manner can be realized by the following manner: the product of the second loss value and the first weight coefficient, the product of the third loss value and the second weight coefficient, and the product of the fourth loss value and the third weight coefficient are calculated respectively, and the three products are added to obtain the fusion loss value. The fusion loss value is the loss value of the entire detection model.

[0097] The first weight coefficient, the second weight coefficient, and the third weight coefficient can be set according to experience or actual conditions. In the embodiment of the application, the specific values of the first weight coefficient, the second weight coefficient, and the third weight coefficient are not limited. For example, the first weight coefficient can be set to 5, and the second weight coefficient and the third weight coefficient can be set to 1.

[0098] After obtaining the constructed question, the application can further perform the following steps: performing question correction according to the constructed question to obtain a correction result.

[0099] After obtaining the constructed question corresponding to the question in the target image, the correct text content actually should be filled in each handwritten text in the constructed question can be calculated according to the operation rule of the question, and then the correct text content is compared with each handwritten text recognition result to perform question correction to obtain a correction result.

[0100] In the embodiment of the application, after completing the question construction and obtaining the constructed question, the automatic correction of the question is performed according to the constructed question to obtain a correction result. Compared with the scheme of identifying the type of the question in the image by using a question type recognition model, and then merging the text boxes combined with the horizontal text line detection result to process the question content in the question, the efficiency of question correction can be effectively improved, and the cost of question correction can be saved.

[0101] Further, a specific example is listed to illustrate the question recognition in the image in the embodiment of the application, as shown in Figure 5 Figure 5 The flowchart of another question recognition method provided in the embodiment of the application can include the following steps:

[0102] Step S501, a large number of training image samples containing sample questions are collected.

[0103] Specifically, a large number of images containing different types of questions can be collected as training image samples.

[0104] Step S502, training image samples are labeled.

[0105] ​Specifically, the training image samples collected in step S501 can be labeled. For each training image sample, the question region where the question is located in the training image sample can be framed, and the text regions where the printed text and the handwritten text are located in the training image sample can be framed as training data of the detection model. In addition, each question can be classified according to a pre-defined question type, and the question region where the question is located can be cropped from the training image sample as training data of the encoder model. In addition, the text content of the printed text and the text content of the handwritten text can be labeled as training data of the recognition model.

[0106] Step S503: constructing a CenterNet model with three detection branches as the detection model.

[0107] The loss value of the entire detection model is the sum of the loss values of the three detection branches. The feature extraction part of the detection model can be a Resnet18 composed of 4 residual blocks.

[0108] Specifically, the target image is input into the Resnet18 in the pre-trained CenterNet model, and an initial feature map with a size of 1 / 4 of the target image, an initial feature map with a size of 1 / 8 of the target image, an initial feature map with a size of 1 / 16 of the target image, and an initial feature map with a size of 1 / 32 of the target image are obtained through the residual blocks in the Resnet18, respectively. The above four initial feature maps are respectively up-sampled to obtain initial feature maps with the same size. The elements at the same positions in the four feature maps after sampling are multiplied to obtain fused image features. The fused image features can be input into the three detection branches of the CenterNet model, respectively. Each detection branch outputs a set of feature maps with a channel number of 6, which respectively represent the score map of the center point, the offset of the center point, the length and width of the rectangular region corresponding to the center point, and the rotation angle of the above rectangular region.

[0109] During training, for each detection branch, the channel of the score map of the center point can use the FocalLoss loss function to calculate the loss value, and the remaining channels can use the L1 smooth loss function to calculate the loss value. A weight coefficient is set for the loss value corresponding to each detection branch. The loss value of each detection branch is multiplied by the corresponding weight coefficient, and three products are obtained. The sum of the three products is the loss value of the entire CenterNet model.

[0110] Step S504: constructing a variational autoencoder as the model for encoding the question region.

[0111] Specifically, both the encoder and the decoder are included, wherein the encoder is composed of 8 layers of convolution, and the decoder is composed of 8 layers of deconvolution, each layer of the encoder is jump-connected to the corresponding layer of the deconvolution of the decoder (U-Net structure), and the training is performed using an L1 loss function.

[0112] Step S505, the CRNN model is used as the recognition model.

[0113] It should be noted that the specific process is an example of performing step S504 first and then performing step S505, but those skilled in the art should understand that in actual application, the execution order of step S504 and step S505 can be performed in any order, or can be performed in parallel.

[0114] Step S506, the training data in step S502 are used to train the CenterNet model in step S503, the variational autoencoder in step S504, and the CRNN model in step S505, respectively.

[0115] It should be noted that the above steps S501 to S506 are steps in the training phase, and those skilled in the art should understand that in the application phase, the above steps do not need to be performed.

[0116] Step S507, a question type vector library is constructed.

[0117] Specifically, after the training of all models is completed, only the encoder model part of the variational autoencoder is retained, and the training data used in step S502 are divided into training data of multiple question types according to the categories of the question types, each training data of the question type is input into the encoder model to obtain an encoding vector, and the average value of the encoding vectors of the encoding vectors belonging to the same category of the question type is calculated. The average value of the encoding vectors is used as the encoding vector of the question type, thereby constructing a question type vector library, and the question type vector library is pre-stored with multiple question type feature vectors.

[0118] Step S508, in the application phase, a target image containing a question is obtained.

[0119] Step S509, the target image in step S508 is subjected to the detection model in step S503, and the first detection branch outputs a question area, according to the question area, the target image is screenshot, and the screenshot is input into the encoder model in step S504 to obtain an encoding vector of the question.

[0120] Step S510, determine the question type information of the question according to the encoding vector of step S509 and the question type vector library of step S507, in parallel, the second detection branch and the third detection branch output the text area of the printed text and the text area of the handwritten text respectively, according to the above text area, the target image is screenshot, and the screenshot is sent to the CRNN model in step S505 to obtain the text recognition result.

[0121] Specifically, the encoding vector of step S509 and all question type vectors in the question type vector library of step S507 are calculated for question similarity (cosine distance can be used for measurement), so as to determine the question type information, if it cannot be determined, it means that the question is a new question type, at this time, the encoding vector of the question can be stored in the question type vector library, and a new question construction strategy and a new question solving strategy are developed for the new question type.

[0122] Step S511, according to the question type information, the printed text recognition result and the handwritten text recognition result, the question construction is carried out, and the constructed question corresponding to the question in the target image is obtained.

[0123] Step S512, according to the constructed question, the question is corrected, and the correction result is obtained. Specifically, after obtaining the constructed question corresponding to the question in the target image, the correct text content actually filled in each handwritten text in the constructed question can be calculated according to the operation rule of the question, and then the correct text content is compared with each handwritten text recognition result, the question is corrected, and the correction result is obtained.

[0124] The same model is used to simultaneously perform text detection and question type recognition. When training the model, a large amount of artificial annotation data is required, which is high in cost. At the same time, the model needs to complete both the detection task and the question type classification task. From the principle, the needs of the two for input features are opposite. Using the same model to achieve the effect is inevitably poor. More importantly, in the logical operation question, how many irregular question types are very difficult to define, and it is almost impossible to count. That is to say, every time a new question type appears, the model needs to add a question type category. At the same time, a large amount of manpower is spent to collect the data of this question type and mark, and then update the model. At the same time, in order to adapt to the output of the model, the text detection part also needs to be updated. The whole solution is full development (that is, every time a new question type comes, everything needs to be done again), which cannot realize incremental development (only a small amount of change is needed every time a new question type comes). Therefore, the embodiment of the present application Figure 5In the flow of the above-mentioned question recognition method, the CenterNet model is used for detection, and the detection model is only used for detection, but not for question type recognition. After the detection is completed, the question area output by the CenterNet model is used to encode the question area by using the encoder model to obtain the encoding vector of the question, the encoding vector of the question is used to calculate the question similarity with the question type vector in the question type vector library, and the question type corresponding to the maximum question similarity greater than the preset similarity is determined as the question type information of the question. Therefore, after the training of the CenterNet model is completed, even if a new question type appears, the trained CenterNet model does not need to be retrained, and the trained encoder model does not need to be retrained, and only the new question type needs to be developed. When a new question type appears, compared with the way of detecting the question area and recognizing the question type in the same model, Figure 5 The flow can avoid full development, realize incremental development, effectively reduce the development workload, improve the development efficiency, and save the development cost.

[0125] Meanwhile, in the above-mentioned flow, the CenterNet model with three detection branches is used to simultaneously detect the question area, the text area of printed text, and the text area of handwritten text in the target image. Since the above-mentioned three types of detection have certain commonality in the requirement of input features, the CenterNet model can be used to simultaneously obtain the above-mentioned three types of areas, which can effectively improve the detection efficiency, and further improve the overall efficiency of question recognition.

[0126] In addition, in the above-mentioned flow, two models, the encoder model and the CRNN model, are used to perform the operations of question encoding and text recognition in parallel. Since the encoder model and the CRNN model performing the above-mentioned two operations are executed in parallel, the encoding vector of the question, the recognition result of the printed text, and the recognition result of the handwritten text can be obtained at the same time. Therefore, the above-mentioned flow can further improve the efficiency of question recognition, and further improve the efficiency of question correction.

[0127] Embodiment six,

[0128] The embodiment of the present application provides a question recognition device, such as Figure 6 As shown, Figure 6A subject recognition device is provided in an embodiment of the present application. The subject recognition device 60 comprises: an acquisition module 601 configured to acquire a target image containing a subject; a detection module 602 configured to input the target image into a pre-trained detection model to obtain a subject region of the subject and a text region in the subject; a matching module 603 configured to encode an image part of the subject region by using a pre-trained encoder model to obtain an encoding vector of the subject, and obtain subject type information of the subject based on a matching result of the encoding vector and a plurality of pre-stored subject type feature vectors; an identification module 604 configured to perform text recognition based on the text region in the subject to obtain a text recognition result; and a construction module 605 configured to perform subject construction based on the subject type information and the text recognition result to obtain a constructed subject corresponding to the subject in the target image.

[0129] Optionally, in an embodiment of the present application, the matching module 603 is further configured to capture a screenshot of the target image according to the subject region to obtain the image part of the subject region, and input the image part of the subject region into the pre-trained encoder model to obtain the encoding vector of the subject.

[0130] Optionally, in an embodiment of the present application, the matching module 603 is further configured to calculate a subject similarity between the encoding vector of the subject and the plurality of pre-stored subject type feature vectors to obtain a plurality of subject similarities corresponding to the subject region, and determine subject type information of a subject type feature vector corresponding to a maximum subject similarity in the plurality of subject similarities as the subject type information of the subject if the maximum subject similarity is greater than a preset similarity.

[0131] Optionally, in an embodiment of the present application, the matching module 603 is further configured to add the encoding vector of the subject into the plurality of pre-stored subject type feature vectors if the plurality of subject similarities are all less than or equal to the preset similarity.

[0132] Optionally, in an embodiment of the present application, the subject recognition device 60 further comprises an encoder model training module configured to acquire a first training image sample containing a sample subject, the sample subject in the first training image sample belongs to a same subject type, input the first training image sample into an initial encoder model to obtain a subject prediction encoding vector of the sample subject, input the subject prediction encoding vector of the sample subject into an initial decoder model to obtain a sample prediction subject, and obtain a first loss value by using an L1 loss function according to the sample prediction subject, and train the initial encoder model and the initial decoder model according to the first loss value to obtain the encoder model.

[0133] Optionally, in an embodiment of the present application, the encoder model training module is configured to obtain a training image sample set, wherein the image samples in the training image sample set are image samples corresponding to a plurality of different question types; determine a question type from the plurality of different question types, and determine the image sample corresponding to the question type as the first training image sample.

[0134] Optionally, in an embodiment of the present application, the question identification apparatus 60 further comprises a pre-storage module, which is further configured to input the first training image sample into a pre-trained encoder model to obtain an encoding vector corresponding to the first training image sample; calculate the average value of the encoding vector corresponding to the first training image sample; take the average value of the encoding vector as the question type feature vector of the first training image sample; and determine a plurality of pre-stored question type feature vectors according to the question type feature vector of the first training image sample.

[0135] Optionally, in an embodiment of the present application, the encoder model is a variational autoencoder model.

[0136] Optionally, in an embodiment of the present application, the detection module 602 is further configured to input the target image into a pre-trained detection model, obtain the question region of the question through a first detection branch of the detection model, obtain the text region of the printed text in the question through a second detection branch of the detection model, obtain the text region of the handwritten text in the question through a third detection branch of the detection model, and the text region of the question comprises the text region of the printed text in the question and the text region of the handwritten text in the question; the recognition module 604 is further configured to perform text recognition based on the text region of the printed text and the text region of the handwritten text through a pre-trained recognition model to obtain corresponding printed text recognition result and handwritten text recognition result; wherein the encoder model and the recognition model are two models executed in parallel.

[0137] Optionally, in an embodiment of the present application, the recognition module 604 is further configured to respectively take screenshots of the target image according to the text region of the printed text and the text region of the handwritten text to obtain corresponding printed text region image and handwritten text region image; and input the printed text region image and the handwritten text region image into the pre-trained recognition model to perform text recognition to obtain corresponding printed text recognition result and handwritten text recognition result.

[0138] Optionally, in an embodiment of the present application, the detection model comprises: a feature extraction part; a first detection branch, a second detection branch and a third detection branch connected in parallel after the feature extraction part; the detection module 602 is further configured to input the target image into the pre-trained detection model, perform feature extraction on the target image through the feature extraction part in the detection model, and obtain image features corresponding to the target image; perform question region prediction based on the image features through the first detection branch, and obtain the question region of the question; perform printed text region prediction based on the image features through the second detection branch, and obtain the text region of the printed text in the question; and perform handwritten text region prediction based on the image features through the third detection branch, and obtain the text region of the handwritten text in the question.

[0139] Optionally, in an embodiment of the present application, the feature extraction part is a residual network comprising a plurality of residual blocks; the detection module 602 is further configured to input the target image into the residual network in the pre-trained detection model, and obtain an initial image feature corresponding to the target image through each residual block in the residual network; and perform feature fusion on a plurality of initial image features to obtain the image features.

[0140] Optionally, in an embodiment of the present application, the question recognition device 60 further comprises a detection model training module, and the detection model training module is further configured to obtain a second training image sample comprising a sample question, a sample question region of the sample question in the second training image sample, and a text region in the sample question; input the second training image sample into an initial detection model to obtain a sample question prediction region and a text prediction region in the sample question; obtain a second loss value based on the sample question prediction region and the sample question region; obtain a text loss value based on the text prediction region and the text region in the sample question; perform numerical fusion on the second loss value and the text loss value to obtain a fusion loss value; and train the initial detection model according to the fusion loss value to obtain the detection model.

[0141] Optionally, in an embodiment of the present application, the text region in the sample question includes a text region of printed text in the sample question and a text region of handwritten text in the sample question; the detection model training module is further configured to obtain a sample question prediction region by a first detection branch of the initial detection model; obtain a text prediction region of printed text in the sample question by a second detection branch of the initial detection model; obtain a text prediction region of handwritten text in the sample question by a third detection branch of the initial detection model; the first detection branch corresponds to the second loss value; obtain a third loss value corresponding to the second detection branch based on the text prediction region of printed text in the sample question and the text region of printed text in the sample question; obtain a fourth loss value corresponding to the third detection branch based on the text prediction region of handwritten text in the sample question and the text region of handwritten text in the sample question, and the text loss value includes the third loss value and the fourth loss value.

[0142] Optionally, in an embodiment of the present application, the question identification apparatus 60 further includes a correction module, and the correction module is further configured to perform question correction according to the constructed question to obtain a correction result.

[0143] The question identification apparatus 60 of the present embodiment is configured to implement the corresponding question identification method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described herein again. In addition, the functions of each module in the question identification apparatus 60 of the present embodiment can be implemented by referring to the descriptions of the corresponding parts in the foregoing method embodiments, which will not be described herein again.

[0144] Embodiment seven,

[0145] Based on any one of the question identification methods described in the foregoing embodiments one to five, the present embodiment provides an electronic device. It should be noted that the question identification method of the present embodiment can be performed by any appropriate electronic device with data processing capability, including but not limited to: a server, a mobile terminal (such as a mobile phone, a PAD, etc.), and a PC, etc. For example, Figure 7 As shown in Figure 7 A structural diagram of an electronic device provided in the present embodiment. The specific implementation of the electronic device is not limited in the present embodiment. The electronic device 70 can include a processor 702, a communications interface 704, a memory 706, and a communications bus 708.

[0146] The processor 702, the communications interface 704, and the memory 706 can communicate with each other through the communications bus 708.

[0147] The communication interface 704 is configured to communicate with other electronic devices or servers.

[0148] The processor 702 is configured to execute the computer program 710, and specifically, execute the steps of the question identification method described above.

[0149] Specifically, the computer program 710 can include computer program codes, which include computer operation instructions.

[0150] The processor 702 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0151] The memory 706 is configured to store the computer program 710. The memory 706 can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.

[0152] The computer program 710 can be specifically used to cause the processor 702 to perform the operations of the question identification method in any one of Embodiment One to Embodiment Five. It can be understood that, for the convenience and brevity of description, the specific working processes of the devices and modules described above can refer to the corresponding process descriptions in the foregoing method embodiments, and will not be described herein.

[0153] Embodiment Eight,

[0154] Based on the question identification method described in Embodiment One to Embodiment Five, the embodiments of the present application provide a computer storage medium, which stores a computer program, and the computer program is executed by a processor to implement the question identification method described in Embodiment One to Embodiment Five.

[0155] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or part of the operations of the components / steps can be combined into a new component / step, to achieve the purpose of the embodiments of the present application.

[0156] The above-described methods according to the embodiments of the present application can be implemented in hardware, firmware or software, or any combination thereof. And the methods described herein can be stored as software or computer code on a recording medium such as a CD ROM, RAM, floppy disks, hard disks, or optical disk, or be stored in a remote recording medium or non-transitory machine-readable medium originally stored in a remote recording medium or downloaded from a network and stored in a local recording medium, so that the methods described herein can be processed by such software on a recording medium using a general purpose computer, a special purpose processor, or programmable or dedicated hardware such as ASIC or FPGA. It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, processor, or hardware, the subject recognition method described herein is implemented. In addition, when a general purpose computer accesses the code for implementing the subject recognition method shown herein, the execution of the code will convert the general purpose computer into a special purpose computer for executing the subject recognition method shown herein.

[0157] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0158] Those of ordinary skill in the art can realize that the example units and method steps described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0159] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0160] The above embodiments are only used for describing the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, all equivalent technical solutions belong to the scope of the present application, and the patent protection scope of the present application should be defined by the claims.

Claims

1. A subject recognition method characterized by comprising: The method comprises: acquiring a target image containing a question; inputting the target image into a pre-trained detection model to obtain a question region of the question and a text region in the question, the text region in the question including a text region of printed text in the question and a text region of handwritten text in the question; encoding an image part of the question region through a pre-trained encoder model to obtain an encoding vector of the question, and obtaining question type information of the question based on a matching result of the encoding vector and a plurality of pre-stored question type feature vectors; the encoder model is an encoder model including an encoder and a decoder part; performing text recognition based on the text region in the question to obtain a text recognition result; constructing a question according to the question type information and the text recognition result to obtain a constructed question corresponding to the question in the target image; wherein, based on the matching result of the encoding vector of the question and the plurality of pre-stored question type feature vectors, the question type information of the question is obtained, which comprises: calculating the question similarity between the encoding vector of the question and the plurality of pre-stored question type feature vectors to obtain a plurality of question similarities corresponding to the question region; if the plurality of question similarities include a question similarity greater than a preset similarity, the question type information of the question type feature vector corresponding to the maximum question similarity in the plurality of question similarities is determined as the question type information of the question; performing text recognition based on the text region in the question to obtain a text recognition result, which comprises: performing text recognition based on the text region of printed text and the text region of handwritten text to obtain corresponding printed text recognition result and handwritten text recognition result.

2. The method of claim 1, wherein, The encoding of the image part of the question region through the pre-trained encoder model to obtain the encoding vector of the question comprises: capturing the target image according to the question region to obtain the image part of the question region; inputting the image part of the question region into the pre-trained encoder model to obtain the encoding vector of the question.

3. The method of claim 1, wherein, The method further comprises: if the plurality of question similarities are all less than or equal to the preset similarity, the encoding vector of the question is added to the plurality of pre-stored question type feature vectors.

4. The method of claim 1, wherein, The encoder model is pre-trained in the following way: acquiring a first training image sample containing a sample question, the sample question in the first training image sample belonging to the same question type; inputting the first training image sample into an initial encoder model to obtain a question prediction encoding vector of the sample question; inputting the question prediction encoding vector of the sample question into an initial decoder model to obtain a sample prediction question; obtaining a first loss value by using an L1 loss function according to the sample prediction question; training the initial encoder model and the initial decoder model according to the first loss value to obtain the encoder model.

5. The method of claim 4, wherein, The acquisition of the first training image sample containing the sample question comprises: obtaining a training image sample set, wherein image samples in the training image sample set are image samples corresponding to a plurality of different question types; determining a question type from the plurality of different question types, and determining an image sample corresponding to the question type as the first training image sample.

6. The method of claim 5, wherein, The method further comprises: inputting the first training image sample into the pre-trained encoder model to obtain an encoding vector corresponding to the first training image sample; calculating a mean value of the encoding vector corresponding to the first training image sample; taking the mean value of the encoding vector as a question type feature vector of the first training image sample; determining the pre-stored plurality of question type feature vectors according to the question type feature vector of the first training image sample.

7. The method according to any one of claims 1 to 6, characterized in that, The encoder model is a variational autoencoder model.

8. The method of claim 1, wherein, inputting the target image into the pre-trained detection model to obtain a question region of the question and a text region in the question, comprising: inputting the target image into the pre-trained detection model to obtain a question region of the question through a first detection branch of the detection model, to obtain a text region of printed text in the question through a second detection branch of the detection model, and to obtain a text region of handwritten text in the question through a third detection branch of the detection model; performing text recognition based on the text region of printed text and the text region of handwritten text respectively to obtain corresponding printed text recognition result and handwritten text recognition result, comprising: performing text recognition based on the text region of printed text and the text region of handwritten text respectively through a pre-trained recognition model to obtain corresponding printed text recognition result and handwritten text recognition result; wherein the encoder model and the recognition model are two models executed in parallel.

9. The method of claim 8, wherein, The performing text recognition based on the text region of printed text and the text region of handwritten text respectively through a pre-trained recognition model to obtain corresponding printed text recognition result and handwritten text recognition result, comprising: performing screenshot on the target image according to the text region of printed text and the text region of handwritten text respectively to obtain corresponding printed text region image and handwritten text region image; inputting the printed text region image and the handwritten text region image into the pre-trained recognition model respectively to perform text recognition and obtain corresponding printed text recognition result and handwritten text recognition result. The detection model comprises: a feature extraction part; a first detection branch, a second detection branch and a third detection branch connected in parallel after the feature extraction part; 10. The method of claim 8, wherein, The inputting the target image into the pre-trained detection model to obtain a question region of the question through a first detection branch of the detection model, to obtain a text region of printed text in the question through a second detection branch of the detection model, and to obtain a text region of handwritten text in the question through a third detection branch of the detection model, comprising: ​ The target image is input into a pre-trained detection model, feature extraction of the target image is performed by a feature extraction part in the detection model, and an image feature corresponding to the target image is obtained; A question area of the question is obtained by question area prediction based on the image feature through the first detection branch; a text area of printed text in the question is obtained by printed text area prediction based on the image feature through the second detection branch; and a text area of handwritten text in the question is obtained by handwritten text area prediction based on the image feature through the third detection branch.

11. The method of claim 10, wherein, The feature extraction part is a residual network comprising a plurality of residual blocks; The target image is input into a pre-trained detection model, feature extraction of the target image is performed by a feature extraction part in the detection model, and an image feature corresponding to the target image is obtained, including: The target image is input into a residual network in the pre-trained detection model, and an initial image feature corresponding to the target image is obtained by each residual block in the residual network; Feature fusion is performed on a plurality of initial image features to obtain an image feature.

12. The method of claim 1, wherein, The detection model is pre-trained in the following manner: A second training image sample comprising a sample question, a sample question area of the sample question in the second training image sample, and a text area in the sample question are obtained; The second training image sample is input into an initial detection model to obtain a sample question prediction area and a text prediction area in the sample question; A second loss value is obtained based on the sample question prediction area and the sample question area; A text loss value is obtained based on the text prediction area and the text area in the sample question; Numerical fusion is performed on the second loss value and the text loss value to obtain a fusion loss value; The initial detection model is trained according to the fusion loss value to obtain the detection model.

13. The method of claim 12, wherein, The text area in the sample question comprises a text area of printed text in the sample question and a text area of handwritten text in the sample question; The second training image sample is input into an initial detection model to obtain a sample question prediction area and a prediction area in the sample question, including: a sample question prediction area is obtained by a first detection branch of the initial detection model; a text prediction area of printed text in the sample question is obtained by a second detection branch of the initial detection model; and a text prediction area of handwritten text in the sample question is obtained by a third detection branch of the initial detection model; The first detection branch corresponds to the second loss value. The text loss value is obtained based on the text prediction region in the sample question and the text region in the sample question, including: the third loss value corresponding to the second detection branch is obtained based on the text prediction region of the printed text in the sample question and the text region of the printed text in the sample question; the fourth loss value corresponding to the third detection branch is obtained based on the text prediction region of the handwritten text in the sample question and the text region of the handwritten text in the sample question, and the text loss value includes the third loss value and the fourth loss value.

14. The method of claim 1, wherein, The method further includes: Question correction is performed according to the constructed question to obtain a correction result.

15. A title recognition apparatus characterized by comprising: The device includes: An acquisition module is configured to acquire a target image containing a question; A detection module is configured to input the target image into a pre-trained detection model to obtain a question region of the question and a text region in the question, the text region in the question including a text region of printed text in the question and a text region of handwritten text in the question; A matching module is configured to encode an image part of the question region by using a pre-trained encoder model to obtain an encoding vector of the question, and obtain question type information of the question based on a matching result of the encoding vector and a plurality of pre-stored question type feature vectors; the encoder model is an encoder model including an encoder and a decoder part; An identification module is configured to perform text identification based on the text region in the question to obtain a text identification result; A construction module is configured to construct a question according to the question type information and the text identification result to obtain a constructed question corresponding to the question in the target image; The matching module is further configured to calculate a question similarity between the encoding vector of the question and the plurality of pre-stored question type feature vectors to obtain a plurality of question similarities corresponding to the question region; if the plurality of question similarities include a question similarity greater than a preset similarity, a question type information of a question type feature vector corresponding to a maximum question similarity in the plurality of question similarities is determined as the question type information of the question; The identification module is further configured to perform text identification based on the text region of the printed text and the text region of the handwritten text to obtain corresponding printed text identification results and handwritten text identification results.

16. An electronic device, comprising: It includes: A processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the question identification method in any one of claims 1-14.

17. A computer storage medium, comprising, A computer program is stored thereon, and the program is executed by a processor to implement the question identification method in any one of claims 1-14.

Images