Image processing method and device

Abstract

The present specification provides an image processing method and device, wherein the image processing method comprises: acquiring an image to be recognized; inputting the image to be recognized into a recognition model for character recognition to obtain a recognition result of the image to be recognized, wherein the recognition result comprises a first character and a second character; based on the recognition result, screening a target formula in a preset formula database, and comparing the second character with the target formula and the operation result of the first character; generating and displaying a target image based on the comparison result and the image to be recognized.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method and device. Background Art

[0002] With the development of Internet technology, the Internet education industry has attracted more and more attention from users. In this industry, tool-based educational products can not only help students with tutoring, but also provide auxiliary education for parents and teachers. Many of the current tool-based educational products are able to provide the function of taking photos and correcting them, but most of them can only solve the conversion of common fractions and units in the lower grades. For question types such as number series with many categories and complex rules, they can only be corrected through the preset question bank, that is, according to the identified number series questions, select the target number series questions that are the same as the number series questions in the preset question bank for correction. If the richness of the question bank is low, it may lead to the inability to correctly correct the number series questions, which greatly affects the user experience. Therefore, an effective solution is urgently needed to solve this problem. Summary of the invention

[0003] In view of this, the embodiment of this specification provides an image processing method. This specification also relates to an image processing device, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, there is provided an image processing method, including:

[0005] Obtain an image to be recognized;

[0006] Inputting the image to be recognized into a recognition model for character recognition to obtain a recognition result of the image to be recognized, wherein the recognition result includes a first character and a second character;

[0007] Based on the recognition result, a target formula is selected from a preset formula database, and the second character is compared with the target formula and the operation result of the first character;

[0008] A target image is generated and displayed according to the comparison result and the image to be identified.

[0009] Optionally, obtaining the image to be identified includes:

[0010] Get the captured images uploaded by the user;

[0011] When it is detected that the captured image contains a target question type, the to-be-recognized image corresponding to the target question type is generated according to the captured image.

[0012] Optionally, generating the image to be recognized corresponding to the target question type according to the captured image includes:

[0013] Locating the target question type in the captured image by using a question type positioning frame to obtain an intermediate image to be recognized corresponding to the target question type;

[0014] The target characters contained in the target question type in the intermediate image to be recognized are detected based on a target detection algorithm, and the image to be recognized is generated according to the detection result.

[0015] Optionally, inputting the image to be recognized into a recognition model for character recognition to obtain a recognition result of the image to be recognized includes:

[0016] Determining the question type of the target question type contained in the image to be recognized;

[0017] Annotating the image to be identified according to the question type;

[0018] The annotated image to be recognized is input into the recognition model for character recognition to obtain the recognition result.

[0019] Optionally, inputting the annotated image to be recognized into the recognition model for character recognition to obtain the recognition result includes:

[0020] Determine a target recognition module in the recognition model according to the labeled image to be recognized, and process the image to be recognized through a convolutional neural network in the target recognition module to obtain a feature vector;

[0021] The feature vector is identified according to the long short-term memory network in the target identification module to obtain the identification result.

[0022] Optionally, before the step of selecting a target formula in a preset formula database based on the recognition result is performed, the method further includes:

[0023] Filtering sub-characters from the recognition results according to a union-find algorithm to form a character set;

[0024] Accordingly, the step of selecting a target formula from a preset formula database based on the recognition result includes:

[0025] The recognition result is adjusted according to the sub-characters included in the character set, and the target formula is selected in the preset formula database according to the adjusted recognition result.

[0026] Optionally, the step of selecting sub-characters from the recognition result according to a union-find algorithm to form a character set includes:

[0027] Adding character frames for the sub-characters included in the first character and the sub-characters included in the second character in the image to be recognized;

[0028] The spacing between adjacent character boxes is calculated according to the adding result, and sub-characters whose spacing is less than a spacing threshold are selected to form the character set.

[0029] Optionally, the step of screening a target formula in a preset formula database based on the recognition result includes:

[0030] Based on the first character in the preset formula database, screening the target formula according to the priority order of the formula;

[0031] or,

[0032] The target formula is filtered in the preset formula database based on the first character and the second character according to the priority order of the formulas.

[0033] Optionally, comparing the second character with the target formula and a calculation result of the first character includes:

[0034] Obtaining a third character according to a calculation result of the first character and the target formula;

[0035] The third character is compared with the second character.

[0036] Optionally, also include:

[0037] In a case where the second character includes at least two sub-characters, comparing the at least two sub-characters with the third character respectively;

[0038] Accordingly, generating and displaying a target image according to the comparison result and the image to be identified includes:

[0039] Determining whether any sub-character is equal to the third character according to at least two comparison results;

[0040] If yes, generating and displaying the target image according to the first display rule and the image to be recognized;

[0041] If not, the target image is generated and displayed according to the second display rule and the image to be recognized.

[0042] Optionally, generating and displaying a target image according to the comparison result and the image to be identified includes:

[0043] When the second character and the operation result are equal, selecting a first display template from the preset display templates to integrate with the image to be recognized, generating the target image and displaying it;

[0044] In the case that the second character is not equal to the operation result, a second display template is selected from the preset display templates to be integrated with the image to be recognized to generate the target image and display it.

[0045] According to a second aspect of the embodiments of this specification, there is provided an image processing apparatus, including:

[0046] An image acquisition module is configured to acquire an image to be recognized;

[0047] A model recognition module is configured to input the image to be recognized into a recognition model for character recognition, and obtain a recognition result of the image to be recognized, wherein the recognition result includes a first character and a second character;

[0048] a character comparison module, configured to select a target formula in a preset formula database based on the recognition result, and compare the second character with the target formula and the operation result of the first character;

[0049] The image display module is configured to generate and display a target image based on the comparison result and the image to be identified.

[0050] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0051] Memory and processor;

[0052] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions:

[0053] Obtain an image to be recognized;

[0054] Inputting the image to be recognized into a recognition model for character recognition to obtain a recognition result of the image to be recognized, wherein the recognition result includes a first character and a second character;

[0055] Based on the recognition result, a target formula is selected from a preset formula database, and the second character is compared with the target formula and the operation result of the first character;

[0056] A target image is generated and displayed according to the comparison result and the image to be identified.

[0057] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the image processing method are implemented.

[0058] The image processing method provided by the present application, after acquiring the image to be recognized, will recognize the characters in the image to be recognized through a recognition model, and obtain a recognition result including a first character and a second character, and then screen out a target formula in the formula database based on the recognition result, and compare the second character with the target formula and the operation result of the first character, and finally generate and display the target image according to the comparison result and the image to be recognized, so as to realize accurate recognition of the second character in the image to be recognized, and by combining the target formula and the operation result of the first character for comparison with the second character, it is possible to judge the correctness of the characters in any type of image to be recognized, which can not only improve the accuracy of character recognition, but also realize application in more scenarios, and effectively solve the problem of over-reliance on the richness of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of an image processing method provided by an embodiment of this specification;

[0060] Figure 2 is a schematic diagram of a captured image provided by an embodiment of this specification;

[0061] Figure 3 is a schematic diagram of an image to be recognized provided in an embodiment of this specification;

[0062] Figure 4 is a schematic diagram corresponding to the recognition result provided in an embodiment of this specification;

[0063] Figure 5 is a schematic diagram corresponding to a screening character set provided in an embodiment of this specification;

[0064] Figure 6 is a schematic diagram of a target image provided in an embodiment of this specification;

[0065] Figure 7 It is a processing flow chart of an image processing method applied to a sequence question correction scenario provided in an embodiment of this specification;

[0066] Figure 8 is a structural schematic diagram of an image processing device provided by an embodiment of this specification;

[0067] Fig. 9 It is a structural block diagram of a computing device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0068] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0069] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0070] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0071] First, the terms involved in one or more embodiments of this specification are explained.

[0072] Computer vision: Computer vision is the science of using computers to mimic the human visual system, giving computers the ability to extract, process, understand and analyze images and image sequences similar to humans. Autonomous driving, robotics, smart healthcare and other fields all require computer vision technology to extract and process information from visual signals.

[0073] Sequence question type: Each number in a sequence is called a term of the sequence. The first number is called the first term of the sequence (usually also called the first term), the second number is called the second term of the sequence, and the nth number is called the nth term of the sequence. Depending on the type of sequence, it can be divided into arithmetic sequence, geometric sequence, odd-even sequence, Fibonacci sequence, etc.

[0074] Photo correction: After the user takes a photo of the exercise results and uploads it, the computer vision recognition CV (Computer Vision) method is used to perform text detection tasks on the photo results and correct them, and finally the correction results are returned to the user.

[0075] Union-find set: It is a tree-like data structure used to handle the merging and querying of some disjoint sets (Disjoint Sets), and is often represented by a forest in use.

[0076] Long Short-Term Memory Network (LSTM) is a time recurrent neural network designed to solve the long-term dependency problem of general RNN (recurrent neural network). All RNNs have a chain form of repeated neural network modules. In the standard RNN, this repeated structural module has only a very simple structure, such as a tanh layer.

[0077] In this specification, an image processing method is provided. This specification also relates to an image processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0078] Figure 1 A flowchart of an image processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps:

[0079] Step S102, obtaining an image to be recognized.

[0080] In actual applications, users can use educational tools to correct the homework or test papers written by their children. Users only need to take a photo of the homework or test paper and upload it to the homework correction software. The process of correcting homework or test papers by the homework correction software is to select the same questions as the homework or test paper in the preset question bank, and extract the answers to correct them. In this case, the question bank must contain the questions in the homework or test paper, and the richness of the question bank is relatively high. However, due to the increasing number of question types and questions, the question bank cannot be expanded in time, which results in some questions or question types cannot be corrected by this type of tool, which greatly affects the user experience and has large limitations.

[0081] The image processing method provided by the present application is to be able to correct any changes in questions without using a question bank. After obtaining the image to be recognized, the characters in the image to be recognized are recognized by a recognition model to obtain a recognition result including a first character and a second character. Then, based on the recognition result, a target formula is screened out from the formula database, and the second character is compared with the target formula and the operation result of the first character. Finally, the target image is generated and displayed based on the comparison result and the image to be recognized, so that the second character in the image to be recognized can be accurately recognized, and by combining the target formula and the operation result of the first character for comparison with the second character, the correctness of the characters in any type of image to be recognized can be judged. Not only can the accuracy of character recognition be improved, but also more application scenarios can be realized, effectively solving the problem of over-reliance on the richness of the database.

[0082] In specific implementation, the image to be recognized specifically refers to an image taken by a user, which can be used for subsequent recognition and correction of characters in the image after preprocessing. The characters specifically refer to the characters that need to be recognized in the image to be recognized, and the characters can be numbers or letters; accordingly, the target formula specifically refers to a formula that can correct the recognized characters, such as a geometric series formula, a speed of light calculation formula, a multiplication formula calculation formula, etc. The target image specifically refers to an image that generates a correction result after recognizing and correcting the characters contained in the image to be recognized.

[0083] Based on this, when the image uploaded by the user contains a number series question (including a question stem and the answer filled in by the user), the recognition model can be used to identify the numbers included in the question stem and the numbers filled in by the user in the number series question, and then the target formula can be screened out from the formula database based on the number recognition result (the target formula corresponds to the number series question, and the number of any item in the number series question can be calculated by the formula). At this time, the answer number that needs to be filled in the question can be calculated through the numbers in the question stem and the target formula. At this time, the answer number is compared with the number filled in by the user to determine whether the number filled in by the user is correct or incorrect. Finally, by displaying the comparison result on the image, the user can determine whether the answer to the number series question is correct.

[0084] Alternatively, when the image uploaded by the user contains a calculation problem (including a question stem and an answer filled in by the user), the recognition model can be used to identify the numbers included in the question stem and the numbers filled in by the user in the letter-filled problem, and then the target formula is screened out from the formula database based on the number recognition result (the target formula corresponds to the calculation problem, and the result of the calculation problem can be calculated by the formula). At this time, the answer numbers that need to be filled in the problem can be calculated through the numbers in the question stem and the target formula. At this time, the answer numbers are compared with the numbers filled in by the user to determine whether the numbers filled in by the user are correct or incorrect. Finally, by displaying the comparison results on the image, the user can determine whether the answer to the number series problem is correct.

[0085] In practical applications, the image processing method provided in this embodiment can perform calculations and corrections on any type of questions without matching answers through a question bank. This not only saves the operation of expanding the question bank, but also improves the efficiency of correcting questions in the image, further improving the user experience. It should be noted that in the process of performing calculations and corrections on different question types, it is necessary to combine different recognition modules to complete the recognition of characters, so as to ensure the accuracy of the recognition results and promote the accuracy of correction. This embodiment will take the correction process of series questions as an example to describe the process. The recognition and correction processes of other question types can refer to the corresponding description content of this embodiment, and no further details will be given here.

[0086] Furthermore, the image to be identified specifically refers to an image obtained after preprocessing. The preprocessing process specifically refers to detecting the target question type from the image taken by the user. When the target question type is detected in the image, the image to be identified is generated. In this embodiment, the specific implementation method is as follows:

[0087] (1) Obtain the captured images uploaded by the user.

[0088] Specifically, the captured image specifically refers to an image uploaded by a user through a terminal device, and the terminal device specifically refers to a terminal equipped with an image acquisition device, such as a mobile phone, computer and other wearable devices; this indicates that the user needs to correct questions in homework or test papers, and in order to provide users with richer services, that is, to be able to correct more types of questions, it is necessary to detect the question types contained in the captured image, and after determining the target question type according to the detection result, the characters are recognized and the operation is performed to complete the correction of the questions in the captured image. The target question type is the question type corresponding to the question that can be corrected. The target question type can be a series question type, a calculation question type or a fill-in-the-blank question type, etc. This embodiment does not make too many restrictions here.

[0089] (2) When it is detected that the captured image contains a target question type, the to-be-recognized image corresponding to the target question type is generated based on the captured image.

[0090] Specifically, based on the above-mentioned acquisition of the captured images uploaded by the user, it is further necessary to detect whether the captured images contain the target question type. If not, it means that the captured images do not contain questions that can be identified and corrected. This may be caused by the user's incorrect shooting method, such as the captured image is blurry or incomplete, or the question contained in the captured image cannot be identified and corrected. For example, if the question contained in the image is a relatively deep question such as university advanced mathematics or university physics, it may lead to the inability to correctly correct it, and a reminder that it cannot be corrected can be sent to the user.

[0091] Based on this, when it is detected that the captured image contains the target question type, it means that the captured image contains questions that can be identified and corrected. At this time, the image to be identified corresponding to the target question type can be generated based on the captured image. In specific implementation, generating the image to be identified based on the captured image specifically means that the target question type is selected in the captured image for positioning, and cropped or framed, and then the processed image is detected to obtain the image to be identified for use in the subsequent recognition process.

[0092] Furthermore, in the process of generating the image to be recognized according to the captured image, in order to accurately correct the target question type in the image to be recognized, thereby meeting the needs of the user, the captured image may be processed twice to improve the accuracy of subsequent recognition. In this embodiment, the specific implementation method is as follows:

[0093] Locating the target question type in the captured image by using a question type positioning frame to obtain an intermediate image to be recognized corresponding to the target question type;

[0094] The target characters contained in the target question type in the intermediate image to be recognized are detected based on a target detection algorithm, and the image to be recognized is generated according to the detection result.

[0095] Specifically, the question type positioning frame specifically refers to the position of the target question type located in the captured image, which can be understood as selecting the target question type in the captured image. At this time, the intermediate image to be identified corresponding to the target question type can be obtained. The intermediate image to be identified specifically refers to the image obtained after the positioning processing of the question type positioning frame.

[0096] Based on this, it is now possible to capture the intermediate image to be recognized corresponding to the target question type in the captured image. In order to further improve the subsequent processing effect and the accuracy of subsequent recognition, the target characters contained in the target question type in the intermediate image to be recognized can be detected. The target characters specifically refer to the numbers, letters or punctuation marks contained in the target question type in the intermediate image to be recognized. The target characters can be located according to the detection results, and the position of each target character can be determined. At this time, the image to be recognized can be generated for subsequent processing.

[0097] In practical applications, the process of locating the target question type in the captured image and detecting the target characters can be realized through a text detection algorithm module, which includes a large frame detection model and a small frame detection model. The large frame detection model is mainly constructed based on the target detection algorithm of YOLOv3, and the small frame detection model is mainly implemented based on the target detection algorithm of YOLOv3+FPN (feature pyramid networks). That is, the target question type in the captured image can be located by the large frame detection model, and the target question type can be framed by the question type positioning frame to obtain the intermediate image to be recognized; the small frame detection model can detect the target characters contained in the target question type in the intermediate image to be recognized, that is, determine the target characters belonging to the target question type in the intermediate image to be recognized, and the image to be recognized can be determined according to the detection results.

[0098] At this time, the target question type and the character position contained in the target question type can be determined based on the image to be identified for use in subsequent processing, thereby improving recognition accuracy while speeding up the efficiency of correcting questions in the captured image, further improving the user experience.

[0099] This embodiment is described by taking the example of a user needing to correct a number series question type. Figure 2 As shown, the user uses a mobile phone to take images of questions including "1. Find the pattern and draw it", and "2. Find the pattern first and then fill in the appropriate number". At this time, the homework grading software installed on the user's mobile phone uploads the captured images to the server for processing. The server will detect whether the captured images contain number series questions. If not, it will directly return instructions to the user to upload the correct captured images. If so, it means that the captured images contain number series questions and the subsequent grading process can be carried out.

[0100] Based on this, the captured image is input into the text detection algorithm module for detection. First, the large frame detection model is used to locate the number series question type in the captured image using the question type positioning frame. The intermediate image to be identified is as follows: Figure 3As shown in (a), the number series question type is selected by the question type positioning box corresponding to the label 209, and then the target numbers and target symbols contained in the number series question type in the middle image to be recognized are detected by the small box detection model (target detection algorithm), and the positions of the target numbers and target symbols are determined according to the detection results. Finally, the following image can be generated: Figure 3 The image to be recognized as shown in (b) is used for subsequent correction of the answers filled in by the user in the image.

[0101] In summary, after obtaining the captured image uploaded by the user, in order to speed up the subsequent processing process and improve the subsequent recognition accuracy of the characters in the image to be recognized, the target question type will be located in combination with the question type positioning box, and the target characters will be detected through the target detection algorithm to obtain the image to be recognized. In the subsequent processing process, it is only necessary to recognize and correct the characters corresponding to the target question type, without the need to recognize and correct other characters in the image, which greatly improves the efficiency of question correction.

[0102] Step S104: input the image to be recognized into a recognition model for character recognition to obtain a recognition result of the image to be recognized, wherein the recognition result includes a first character and a second character.

[0103] Specifically, on the basis of the above-mentioned acquisition of the image to be recognized, it is further explained that the characters contained in the image to be recognized need to be recognized for subsequent correction of the questions in the image. Based on this, the image to be recognized is input into the recognition model for character recognition, and the recognition result output by the recognition model after recognizing the image to be recognized is obtained, wherein the recognition result includes a first character and a second character, the first character specifically refers to the character corresponding to the stem of the question contained in the image to be recognized, and the second character specifically refers to the character corresponding to the answer to the question contained in the image to be recognized.

[0104] For example, the question contained in the image to be recognized is 123456(7)(8)(9), where 123456 belongs to the question stem and (7)(8)(9) belongs to the answer filled in by the user. After the image to be recognized is recognized by the recognition model, the first character 123456 and the second character 789 can be obtained.

[0105] Furthermore, in the process of identifying the characters contained in the image to be identified by the recognition model, the server will deploy more models for different scenarios in order to meet more user needs, that is, to identify more types of questions. At this time, if the image uploaded by the user is obtained, before correcting the question in the image, it is necessary to accurately recognize the text in the image in order to correctly correct it. Therefore, it is necessary to select the model corresponding to the question type in the image to perform the character recognition process. In this embodiment, the character recognition is realized in the following way:

[0106] Determining the question type of the target question type contained in the image to be recognized;

[0107] Annotating the image to be identified according to the question type;

[0108] The annotated image to be recognized is input into the recognition model for character recognition to obtain the recognition result.

[0109] In actual applications, in the process of identifying the image to be identified through the recognition model, there are two scenarios. On the one hand, the recognition model is integrated with sub-models that can recognize characters in different question types, such as the recognition model is integrated with a digital recognition model, a graphic recognition model, a letter recognition model and a Chinese character recognition model. The recognition model is used to complete the recognition of numbers, graphics, letters and Chinese characters. However, after the image to be identified is input into the recognition model, if the target question type contained in the image to be identified is not determined, the correct model cannot be selected to identify it. Therefore, at this time, the image to be identified needs to be labeled according to the question type of the target question type, and then the labeled image to be identified is input into the recognition model to select the correct model for character recognition to obtain the recognition result.

[0110] On the other hand, the recognition model can recognize characters in any question type. For example, the recognition model includes a digital recognition module, a letter recognition module and a symbol recognition module. The three modules influence and check each other and introduce influence relationships to each other. At this time, any one of the digital recognition module, the letter recognition module and the symbol recognition module can introduce the influence relationship of the other two modules during the character recognition process to improve the accuracy of the recognition model; and in this process, in order to satisfy the recognition model to select the correct module to recognize the characters in the image to be recognized, the image to be recognized can be annotated according to the target question type, and then the annotated image to be recognized can be input into the recognition model so that character recognition can be performed on it through the correct module to obtain the recognition result.

[0111] Furthermore, after the annotated image to be recognized is input into the recognition model, both the first aspect and the second aspect mentioned above perform character recognition by machine learning. In this embodiment, the specific implementation is as follows:

[0112] Determine a target recognition module in the recognition model according to the labeled image to be recognized, and process the image to be recognized through a convolutional neural network in the target recognition module to obtain a feature vector;

[0113] The feature vector is identified according to the long short-term memory network in the target identification module to obtain the identification result.

[0114] Specifically, the target recognition module specifically refers to a module capable of recognizing characters in the image to be recognized, and may be the model in the first aspect or the module in the second aspect; the convolutional neural network specifically refers to a network capable of extracting features from the image to be recognized, and may be a ResNet network; the feature vector specifically refers to a vector expression of the image to be recognized; the long short-term memory network specifically refers to a network for classifying and recognizing feature vectors, and may be an LSTM network.

[0115] Based on this, after obtaining the annotated image to be identified, the target recognition module can be determined in the recognition model according to the annotation information of the image to be identified, and then the image to be identified is vectorized by the convolutional neural network in the target recognition module to obtain the feature vector, and then the feature vector is identified by the long short-term memory network in the target recognition module to obtain the recognition result and determine the characters in the image to be identified.

[0116] Using the above example, the image to be recognized is as follows Figure 3 As shown in (b), at this time, it is determined that the question in the image is a number series question type, and the image to be recognized is labeled according to the number series question type. The image labeled with the number series question type label is input into the recognition model. The recognition model determines that the numbers in the number series question need to be recognized. In the recognition model, the number recognition module is determined according to the number series question type label. The feature vector Si = [S1, S2...Sn] of the image to be recognized is obtained through the ResNet network in the number recognition module. Then, the numbers in the number series question are recognized through the LSTM network in the number recognition module to obtain the recognition results of each number. See Figure 4 As shown, the numbers above and below the question are the recognition results obtained after recognizing the various numbers in the question, where the question stem number is the first character and the answer number is the second character.

[0117] In addition, if the recognition model needs to correctly recognize the characters in the image to be recognized, it needs to be trained with a large number of samples. The specific training process refers to: screening out a large number of sample images in the sample set, and annotating each sample image, annotating the characters in each sample image, and then forming a large number of sample pairs based on the annotation results and the sample images, and training the recognition model to be trained until a recognition model that meets the training stop condition is obtained, which can be used as the recognition model used in practical applications. The training stop condition can be to judge the size of the loss value, or to judge the recognition accuracy of the recognition model, which is not limited in this embodiment.

[0118] To sum up, in the process of character recognition of the image to be recognized, the image to be recognized is marked by the question type of the target question type and then input into the recognition model for character recognition. A module that matches the question type can be selected in the recognition model for character recognition, so that the first character and the second character can be accurately identified from the image to be recognized, avoiding affecting the subsequent process of correcting the target question type and further improving the correction accuracy.

[0119] Step S106, selecting a target formula in a preset formula database based on the recognition result, and comparing the second character with the target formula and the calculation result of the first character.

[0120] Specifically, based on the recognition results including the first character and the second character obtained by performing character recognition through the recognition model mentioned above, it is further necessary to judge the second character based on the recognition result, and before that it is necessary to determine the correct answer for the position of the second character. In order to be able to recognize questions corresponding to any target question type, the correct answer for the position of the second character will be calculated in combination with the target formula, and then compared with the recognized second character to determine whether the user's answer is correct.

[0121] Based on this, it is first necessary to filter out the target formula that meets the calculation requirements in the formula database according to the recognition result, and then determine the correct answer corresponding to the second character according to the calculation result of the first character and the target formula. Finally, the second character and the correct answer are compared to determine the correctness of the second character. That is, the user's answer can be corrected according to the comparison result to determine whether the answer filled in by the user is correct.

[0122] In actual applications, when the target question type contained in the image to be identified is a sequence question type, the preset formula database is a database that stores sequence formulas, such as arithmetic sequence formulas, geometric sequence formulas, Fibonacci sequence formulas, odd and even sequence formulas, multi-row sequence formulas, multi-column sequence formulas, recursive sequence formulas, multiple groups of pattern-finding sequence formulas, etc.; when the target question type contained in the image to be identified is an operation question type, the preset formula database is a database that stores operation formulas, such as addition formulas, subtraction formulas, multiplication formulas, division formulas, etc. Different formula databases can be preset for different question types, and this embodiment does not make too many restrictions here.

[0123] Furthermore, in the process of selecting the target formula according to the recognition result, in order to accurately correct the target question type in the image to be recognized later, the following two methods can be used to implement the target formula selection process. In this embodiment, the specific contents are as follows:

[0124] The target formula is filtered in the preset formula database according to the priority order of the formula based on the first character; or the target formula is filtered in the preset formula database according to the priority order of the formula based on the first character and the second character.

[0125] Specifically, when filtering the target formula in the formula database, the target formula may be filtered according to the priority order of the formula based on the first character, or the target formula may be filtered according to the priority order of the formula based on the first character and the second character.

[0126] In actual applications, since the stems corresponding to the target question types are not uniform, there may be a possibility that the stems provide more or less content; if the first characters corresponding to the stems are too few, the selected target formula may not be accurate enough. At this time, the conditions for querying the formula database can be composed of the first characters and the second characters, and the target formula can be selected according to the priority order of the formulas in the formula database; if the first characters corresponding to the stems are sufficient, the target formula that meets the requirements can be extracted from the formula database only by the first characters. At this time, the first character in the preset formula database can be used to filter the target formula according to the priority order of the formulas.

[0127] For example, in the number series question 123(4)(5), the first character is the number 123 contained in the question stem, and the second character is the answer 45 filled in by the user. At this time, by filtering the target formula in the formula database by the number 123, multiple formulas may be obtained, and the correct answer in the bracket cannot be obtained. At this time, the number 123 and the answer 45 can be combined to filter the target formula in the formula database, and the answer that should be filled in the bracket can be calculated, so that the user's answer can be compared with the calculation result to determine whether it is correct.

[0128] In actual applications, the target formula is filtered based on the first character, or the target formula is filtered based on the first character and the second character. In the selection process of these two methods, the selection can be made by judging the stem length of the target question type, that is, judging whether the stem length of the target question type is greater than a preset length threshold. If so, the target formula is filtered based on the first character in the preset formula database according to the priority order of the formula; if not, the target formula is filtered based on the first character and the second character in the preset formula database according to the priority order of the formula.

[0129] In summary, in the process of selecting the target formula, in order to correctly screen out the target formula, the above two methods can be used to accurately determine the target formula for different scenarios, thereby promoting the accuracy of subsequent correction of the second character.

[0130] Furthermore, in the process of comparing the second character with the target formula and the operation result of the first character, the correct answer is actually calculated through the first character and the target formula, and the second character and the correct answer are compared; for example, in a series question type, since the series question type is regular, the number to be filled in can be calculated according to the number given in the question, that is, the correct answer is calculated according to the first character (the number in the question stem) and the target formula (the calculation formula that meets the series question), and then the correct answer is compared with the second character (the number filled in by the user). In the calculation question type or the function question type, the correct answer can be calculated in this way and compared with the second character to determine the correctness of the second character, so as to realize the correction of the target question type in the image to be recognized.

[0131] Furthermore, when the second character is compared with the calculation result of the first character and the target formula, the third character is actually obtained based on the calculation result of the first character and the target formula; then the third character is compared with the second character; wherein the third character specifically refers to the calculation result of the first character and the target formula, that is, the correct answer to the target question type in the image to be recognized.

[0132] In addition, when comparing the second character with the calculation result of the first character and the target formula, in addition to comparing based on the first character and the third character, the calculation module can also implement the comparison process. The comparison process of the calculation module means: after obtaining the third character, a new formula can be generated based on the third character and the target formula, and then the second character can be substituted into the new formula to determine whether the second character is correct.

[0133] For example, in the number series question 123456(7)(9)(9), the numbers in the brackets are the numbers filled in by the user, and the numbers outside the brackets are the question stem. At this time, the target formula corresponding to the number series question is determined to be an=a1+(n-1)d (n is a positive integer, d represents the common difference=1, and an represents the nth number). After comparing 123456 and the target formula, the correct answers in the brackets are 7, 8, and 9. On the one hand, the (7)(9)(9) filled in by the user can be directly compared with 789 to determine whether it is correct. On the other hand, the basic formulas corresponding to the three brackets can be determined based on the target formula, namely x+0=7, y+0=8, and z+0=9, where x represents the number filled in by the user in the first bracket, y represents the number filled in by the user in the second bracket, and z represents the number filled in by the user in the third bracket. Substitute the (7)(9)(9) filled in by the user into the three formulas, and determine whether the user's answer is correct based on whether the equations hold.

[0134] In addition, when the image to be recognized contains multiple questions, and the multiple questions all belong to the target question type, the problem of inaccurate correction of the second character may occur, such as Figure 5 As shown, the questions include two number series questions (1) and (2), and the second characters in both questions need to be corrected at the same time. If the order of each character is not inserted when recognizing the second character, or each character is not divided, it will cause recognition confusion, resulting in the inability to accurately filter out the target formula from the formula database, and then the second character cannot be accurately corrected. In order to avoid this problem, in this embodiment, the sub-characters are filtered out through the union-find algorithm to form a character set, and then the target formula is filtered. The specific implementation method is as follows:

[0135] Filtering sub-characters from the recognition results according to a union-find algorithm to form a character set;

[0136] Correspondingly, step S106 is executed, and step S106 specifically refers to adjusting the recognition result according to the sub-characters included in the character set, and filtering the target formula in the preset formula database according to the adjusted recognition result, and then comparing the second character with the target formula and the calculation result of the first character.

[0137] In a specific implementation, when sub-characters are selected from the recognition result through a union-find algorithm to form the character set, it is also necessary to select a target formula in the calculation database that can be operated with the first character to obtain the correct answer corresponding to the second character, and the recognition result can be adjusted according to the character set. The adjustment process specifically refers to integrating the characters belonging to the same question in the recognition result according to the character set, and then selecting the target formula in the formula database according to the integrated characters, so that the obtained target formula meets the requirements of the question, and if there are multiple questions, the target formulas corresponding to each question can be selected according to multiple character sets, so as to avoid the problem of character confusion and inability to correctly select the target formula.

[0138] Furthermore, in the process of selecting the sub-characters to form the character set, the union-find algorithm actually calculates the spacing between the sub-characters and puts the sub-characters with closer spacing into the same set. In this embodiment, the specific implementation method is as follows:

[0139] Adding character frames for the sub-characters included in the first character and the sub-characters included in the second character in the image to be recognized;

[0140] The spacing between adjacent character boxes is calculated according to the adding result, and sub-characters whose spacing is less than a spacing threshold are selected to form the character set.

[0141] Specifically, after the first character and the second character are identified by the recognition model, character boxes are added to the sub-characters contained in the first character and the sub-characters contained in the second character. The character box specifically refers to a box that selects the sub-characters contained in the first character and the second character. After the addition is completed, the spacing between two adjacent character boxes is calculated. Since the spacing between vector characters in the same question is basically equal, at this time, sub-characters that are smaller than the spacing can be selected to form the character set, which is used for subsequent adjustment of the recognition result.

[0142] Using the above example, after obtaining Figure 4After the recognition results are obtained, it is determined that (2, 3, 3, 2, 3, 3, 2, 3, 3) and (10, 9, 8, 10, 9, 8, 10, 9, 8) are sub-characters of the first character, and {(2), (3), (3)} and {(10), (9), (8)} are sub-characters of the second character. At this time, character boxes are added to each sub-character, and the spacing between adjacent character boxes is calculated. By calculation, it is determined that (2, 3, 3, 2, 3, 3, 2, 3, 3) and {(2), (3), (3)} If the spacing between adjacent character boxes in is less than the spacing threshold, then (2, 3, 3, 2, 3, 3, 2, 3, 3) and {(2), (3), (3)} are combined into a first character set, and if the spacing between adjacent character boxes in (10, 9, 8, 10, 9, 8, 10, 9, 8) and {(10), (9), (8)} are less than the spacing threshold, then 10, 9, 8, 10, 9, 8, 10, 9, 8) and {(10), (9), (8)} are combined into a second character set. In this case, the following can be obtained: Figure 5 As shown in the content, the characters marked as 0 belong to the first character set, and the characters marked as 1 belong to the second character set. The recognition results can be adjusted according to the first character set and the second character set, so as to select the target formulas corresponding to the two questions respectively and correct the answers in the brackets.

[0143] In addition, when the characters are confused due to the large tilt angle of the captured image, the sub-characters belonging to the same question can be filtered out through the union-find algorithm to obtain a character set to correct the question. Among them, the spacing threshold can be set according to the actual application scenario, and different questions may correspond to different spacing thresholds, which are not limited in this embodiment.

[0144] The characters belonging to the same question are filtered out to form a character set through the union-find algorithm, which solves the problem of not being able to accurately divide each question after identifying multiple questions. By adjusting the recognition results based on the character set, the recognition results that meet each question in the target question type can be obtained, so that the target formula can be more accurately filtered out from the formula database to correct the second character.

[0145] Step S108, generating and displaying a target image according to the comparison result and the image to be identified.

[0146] Specifically, on the basis of comparing the second character with the first character and the target formula as mentioned above, further, the correctness of the second character can be derived according to the comparison result, that is, the comparison result obtained includes the second character being correct and the second character being incorrect. At this time, in order to facilitate user viewing, a target image can be generated for the correct second character combined with the image to be recognized, or a target image can be generated for the erroneous second character combined with the image to be recognized, and then displayed to the user. It should be noted that the target images generated by the correct second character and the erroneous second character combined with the image to be recognized are not the same, so that users can more easily distinguish the correctness of the second character.

[0147] For example, if the second character is determined to be correct based on the comparison results, a correct mark, such as a check mark, a circle or a green box, can be added to the image to be recognized to generate a target image to display to the user, informing the user that the answer is correct; if the second character is determined to be incorrect, an error mark, such as a cross, a black filled box or a red box, can be added to the image to be recognized to generate a target image to display to the user, informing the user that the answer is incorrect.

[0148] Furthermore, in order to facilitate display to users and improve the recognizability of the target image, the correctness of the second character can be determined more intuitively. The target image can be generated according to a preset display template. In this embodiment, the specific implementation method is as follows:

[0149] When the second character and the operation result are equal, selecting a first display template from the preset display templates to integrate with the image to be recognized, generating the target image and displaying it;

[0150] In the case that the second character is not equal to the operation result, a second display template is selected from the preset display templates to be integrated with the image to be recognized to generate the target image and display it.

[0151] During specific implementation, in order to be able to display a target image with relatively high recognition to the user, a display template will be selected according to the correctness of the second character. When the second character and the calculation result are equal, it means that the second character is filled in correctly. At this time, the first display template can be selected to be integrated with the image to be recognized, and a target image that can indicate that the second character is correct can be generated for display to the user; when the second character and the calculation result are not equal, it means that the second character is filled in incorrectly. At this time, the second display template can be selected to be integrated with the image to be recognized, and a target image that can indicate that the second character is incorrect can be generated for display to the user.

[0152] In practical applications, in order to increase user interest, different display templates can be set according to the actual application scenarios. When the target image needs to be displayed, the template that meets the comparison results is selected according to various dimensional factors and integrated with the image to be identified to generate the target image and display it.

[0153] Using the above example, when determining Figure 5 After the target formulas corresponding to the two questions are calculated, it is determined that the answer to be filled in the brackets of the first question is 2, 3, 3, and the answer filled in by the user is 2, 3, 3, and it is determined that the user filled in correctly, and the answer to be filled in the brackets of the second question is 10, 9, 8, and the answer filled in by the user is 10, 9, 8, and it is determined that the user filled in correctly. At this time, the first display template and the image to be recognized can be selected according to the correct filling of the second character to integrate, and the following is obtained: Figure 6 The target image shown indicates that the answer filled in by the user is correct by selecting the answer filled in by the user and adding the result on top.

[0154] In order to improve the user's recognition, the display template is selected according to the correctness of the second character and combined with the image to be identified to generate the target image. This can not only correctly express whether the answer to the second character is correct, but also facilitate user viewing, further improving the user experience.

[0155] In addition, since the recognition model cannot achieve 100% accuracy, incorrect recognition may occur. In order to avoid affecting the correctness judgment of the subsequent second character, the recognized characters can be processed subsequently. In this embodiment, the specific implementation method is as follows:

[0156] In a case where the second character includes at least two sub-characters, comparing the at least two sub-characters with the third character respectively;

[0157] Accordingly, generating and displaying a target image according to the comparison result and the image to be identified includes:

[0158] Determining whether any sub-character is equal to the third character according to at least two comparison results;

[0159] If yes, generating and displaying the target image according to the first display rule and the image to be recognized;

[0160] If not, the target image is generated and displayed according to the second display rule and the image to be recognized.

[0161] Specifically, when the recognition model cannot accurately recognize the characters in the image to be recognized, two recognition results of the same character may be output together. For example, when recognizing the number 1, the recognition result that may be output may be 1 or 7. At this time, the second character cannot be corrected correctly. In order to avoid this problem, at least two sub-characters can be compared with the target formula and the operation result of the first character respectively. If there is at least one sub-character equal to the third character, the sub-character is used as the recognition result of the second character, and the second character is determined to be correct. The target image is generated and displayed according to the first display rule and the image to be recognized; if there is not at least one sub-character equal to the third character, it is determined that the second character is wrong, and the target image is generated and displayed according to the second display rule and the image to be recognized.

[0162] For example, the recognized second character is 1 or 7. When the calculation result of the first character and the target formula is 7, the recognized 7 is used as the recognition result of the second character. If the second character is determined to be correct, a target image that can represent the correct second character is generated and displayed according to the first display rule and the image to be recognized. When the calculation result of the first character and the target formula is 9, it is determined that 1 or 7 is not equal to 9. If the second character is determined to be wrong, a target image that can represent the wrong second character is generated and displayed according to the second display rule and the image to be recognized.

[0163] In summary, when the accuracy of the recognition model is not high, any sub-character contained in the recognition result can be compared with the third character. As long as there are equal sub-characters, it can be used as the correct recognition result of the second character. In this way, the target question type in the image to be recognized can be corrected, which improves the processing capability and solves the problem of inaccurate recognition model output.

[0164] The image processing method provided by the present application, after acquiring the image to be recognized, will recognize the characters in the image to be recognized through a recognition model, and obtain a recognition result including a first character and a second character, and then screen out a target formula in the formula database based on the recognition result, and compare the second character with the target formula and the operation result of the first character, and finally generate and display the target image according to the comparison result and the image to be recognized, so as to realize accurate recognition of the second character in the image to be recognized, and by combining the target formula and the operation result of the first character for comparison with the second character, it is possible to judge the correctness of the characters in any type of image to be recognized, which can not only improve the accuracy of character recognition, but also realize application in more scenarios, and effectively solve the problem of over-reliance on the richness of the database.

[0165] The following combination Figure 7Taking the application of the image processing method provided in this specification in the scenario of correcting number series questions as an example, the image processing method is further explained. Figure 7 A processing flow chart of an image processing method applied to a sequence question correction scenario provided by an embodiment of this specification is shown, which specifically includes the following steps:

[0166] Step S702: Acquire the captured image uploaded by the user.

[0167] In practical applications, in order to meet the user's needs for number series problem recognition, any number series problem can be corrected, which will be completed through the combination of the recognition model and the formula database. The specific implementation method is as follows.

[0168] Step S704, when it is detected that the captured image contains a number series question, the number series question is located by using the question type location frame to obtain the middle image to be recognized.

[0169] Step S706, detecting the target numbers contained in the number series question in the middle image to be recognized based on the target detection algorithm to obtain the image to be recognized.

[0170] Specifically, when it is determined that the captured image contains a number series problem, in order to accurately correct the number series problem and avoid the influence of other content in the image on the recognition process, the number series problem will be located and the numbers in the number series problem will be detected to obtain the image to be recognized for the subsequent correction process.

[0171] Step S708, adding a number series question type label to the image to be recognized according to the question type of the number series question, and inputting the labeled image to be recognized into the recognition model.

[0172] Step S710, determine the target recognition module corresponding to the series question type label in the recognition model, and process the image to be recognized through the convolutional neural network in the target recognition module to obtain a feature vector.

[0173] Step S712, identify the feature vector according to the long short-term memory network in the target recognition module to obtain the stem numbers and fill-in numbers.

[0174] Step S714, adding character boxes for the question stem numbers and filled-in numbers in the image to be recognized, and calculating the spacing between adjacent character boxes based on the adding results.

[0175] Step S716, select numbers whose spacing is less than the spacing threshold to form a number set, and filter out the target formula in the formula database based on the numbers in the number set that belong to the question stem.

[0176] Step S718, calculate the correct number based on the target formula and the numbers in the number set that belong to the question stem numbers, and determine the actual numbers in the number set that belong to the filled-in numbers.

[0177] Step S720, determine whether the actual number and the correct number are equal; if so, execute step S722, if not, execute step S724.

[0178] Step S722, adding a correct mark to the captured image, and generating a target image based on the adding result to display to the user.

[0179] Step S724, adding an error mark to the captured image, and generating a target image based on the adding result for display to the user.

[0180] The image processing method provided in the present application can realize accurate recognition of the second character in the image to be recognized, and by combining the target formula and the operation result of the first character for comparison with the second character, it can realize the correctness judgment of the characters in any type of image to be recognized, which can not only improve the accuracy of character recognition, but also realize application in more scenarios, effectively solving the problem of over-reliance on the richness of the database.

[0181] Corresponding to the above method embodiment, this specification also provides an image processing device embodiment, Figure 8 FIG. 1 shows a schematic diagram of the structure of an image processing device provided by an embodiment of the present specification. Figure 8 As shown, the device comprises:

[0182] The image acquisition module 802 is configured to acquire an image to be recognized;

[0183] The model recognition module 804 is configured to input the image to be recognized into a recognition model for character recognition, and obtain a recognition result of the image to be recognized, wherein the recognition result includes a first character and a second character;

[0184] A character comparison module 806 is configured to select a target formula in a preset formula database based on the recognition result, and compare the second character with the target formula and the operation result of the first character;

[0185] The image display module 808 is configured to generate and display a target image according to the comparison result and the image to be identified.

[0186] In an optional embodiment, the image acquisition module 802 includes:

[0187] A captured image acquisition unit, configured to acquire captured images uploaded by a user;

[0188] The unit for generating an image to be recognized is configured to generate the image to be recognized corresponding to the target question type according to the captured image when it is detected that the captured image contains the target question type.

[0189] In an optional embodiment, generating the image unit to be identified includes:

[0190] A target question type positioning subunit is configured to locate the target question type in the captured image through a question type positioning frame to obtain an intermediate image to be recognized corresponding to the target question type;

[0191] The subunit for generating the image to be recognized is configured to detect the target characters contained in the target question type in the intermediate image to be recognized based on the target detection algorithm, and generate the image to be recognized according to the detection result.

[0192] In an optional embodiment, the model identification module 804 includes:

[0193] A question type determination unit, configured to determine the question type of the target question included in the image to be identified;

[0194] An image annotation unit, configured to annotate the image to be identified according to the question type;

[0195] The model recognition unit is configured to input the annotated image to be recognized into the recognition model for character recognition to obtain the recognition result.

[0196] In an optional embodiment, the model recognition unit includes:

[0197] A neural network processing subunit is configured to determine a target recognition module in the recognition model according to the labeled image to be recognized, and process the image to be recognized through a convolutional neural network in the target recognition module to obtain a feature vector;

[0198] The recognition subunit is configured to recognize the feature vector according to the long short-term memory network in the target recognition module to obtain the recognition result.

[0199] In an optional embodiment, the image processing device further includes:

[0200] A screening set module is configured to screen out sub-characters from the recognition result to form a character set according to a union-find algorithm;

[0201] Accordingly, the character comparison module 806 is further configured to:

[0202] The recognition result is adjusted according to the sub-characters included in the character set, and the target formula is selected in the preset formula database according to the adjusted recognition result.

[0203] In an optional embodiment, the screening set module includes:

[0204] A character frame adding unit is configured to add character frames for the sub-characters included in the first character and the sub-characters included in the second character in the image to be recognized;

[0205] The spacing calculation unit is configured to calculate the spacing between adjacent character frames according to the adding result, and select sub-characters whose spacing is less than a spacing threshold to form the character set.

[0206] In an optional embodiment, the character comparison module 806 is further configured to:

[0207] The target formula is filtered in the preset formula database according to the priority order of the formula based on the first character; or the target formula is filtered in the preset formula database according to the priority order of the formula based on the first character and the second character.

[0208] In an optional embodiment, the character comparison module 806 includes:

[0209] an operation unit configured to obtain a third character according to an operation result of the first character and the target formula;

[0210] The comparing unit is configured to compare the third character with the second character.

[0211] In an optional embodiment, the character comparison module 806 is further configured to:

[0212] In a case where the second character includes at least two sub-characters, comparing the at least two sub-characters with the third character respectively;

[0213] Accordingly, the image display module 808 includes:

[0214] A judging unit configured to judge whether there is any sub-character equal to the third character according to at least two comparison results;

[0215] If yes, run a first display unit, wherein the first display unit is configured to generate and display the target image according to a first display rule and the image to be recognized;

[0216] If not, a second display unit is run, where the second display unit is configured to generate and display the target image according to a second display rule and the image to be recognized.

[0217] In an optional embodiment, the image display module 808 includes:

[0218] A third display unit is configured to select a first display template from preset display templates to integrate with the image to be recognized, generate the target image and display it when the second character and the operation result are equal;

[0219] The fourth display unit is configured to select a second display template from the preset display templates to integrate with the image to be recognized, generate the target image and display it when the second character and the operation result are not equal.

[0220] The image processing device provided in this embodiment, after acquiring the image to be recognized, will recognize the characters in the image to be recognized through a recognition model, and obtain a recognition result including a first character and a second character, and then screen out a target formula in the formula database based on the recognition result, and compare the second character with the target formula and the operation result of the first character, and finally generate and display the target image according to the comparison result and the image to be recognized, thereby realizing accurate recognition of the second character in the image to be recognized, and by combining the target formula and the operation result of the first character for comparison with the second character, it is possible to judge the correctness of the characters in any type of image to be recognized, which not only improves the accuracy of character recognition, but also enables application in more scenarios, and effectively solves the problem of over-reliance on the richness of the database.

[0221] The above is a schematic scheme of an image processing device of this embodiment. It should be noted that the technical scheme of the image processing device and the technical scheme of the above-mentioned image processing method belong to the same concept, and the details not described in detail in the technical scheme of the image processing device can be referred to the description of the technical scheme of the above-mentioned image processing method.

[0222] Fig. 9 The structure block diagram of a computing device 900 provided according to an embodiment of the present specification is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0223] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0224] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0225] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 900 may also be a mobile or stationary server.

[0226] The processor 920 is used to execute the following computer executable instructions:

[0227] Obtain an image to be recognized;

[0228] Inputting the image to be recognized into a recognition model for character recognition to obtain a recognition result of the image to be recognized, wherein the recognition result includes a first character and a second character;

[0229] Based on the recognition result, a target formula is selected from a preset formula database, and the second character is compared with the target formula and the operation result of the first character;

[0230] A target image is generated and displayed according to the comparison result and the image to be identified.

[0231] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned image processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned image processing method.

[0232] An embodiment of the present specification further provides a computer-readable storage medium storing computer instructions, which are used when executed by a processor to:

[0233] Obtain an image to be recognized;

[0234] Inputting the image to be recognized into a recognition model for character recognition to obtain a recognition result of the image to be recognized, wherein the recognition result includes a first character and a second character;

[0235] Based on the recognition result, a target formula is selected from a preset formula database, and the second character is compared with the target formula and the operation result of the first character;

[0236] A target image is generated and displayed according to the comparison result and the image to be identified.

[0237] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned image processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned image processing method.

[0238] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0239] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0240] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that this specification is not limited by the order of the actions described, because according to this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this specification.

[0241] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0242] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: Obtain an image to be recognized; Inputting the image to be recognized into a recognition model for character recognition to obtain a recognition result of the image to be recognized, the recognition result including a first character and a second character, wherein the image to be recognized includes a calculation question; Preset different formula databases for different question types, select a target formula from the preset formula database based on the recognition result, and compare the second character with the target formula and the calculation result of the first character, wherein the target formula refers to the formula for correcting the recognized character, the target formula corresponds to the calculation question, and is used to calculate the result of the calculation question; Generate and display a target image based on the comparison result and the image to be identified; Among them, the filtering of the target formula in a preset formula database based on the recognition result includes: filtering the target formula in the preset formula database according to the priority order of the formula based on the first character; or filtering the target formula in the preset formula database according to the priority order of the formula based on the first character and the second character.

2. The image processing method according to claim 1, characterized in that: The step of obtaining the image to be identified comprises: Get the captured images uploaded by the user; When it is detected that the captured image contains a target question type, the to-be-recognized image corresponding to the target question type is generated according to the captured image.

3. The image processing method according to claim 2, characterized in that: The step of generating the image to be recognized corresponding to the target question type according to the captured image includes: Locating the target question type in the captured image by using a question type positioning frame to obtain an intermediate image to be recognized corresponding to the target question type; The target characters contained in the target question type in the intermediate image to be recognized are detected based on a target detection algorithm, and the image to be recognized is generated according to the detection result.

4. The image processing method according to claim 1, characterized in that: The step of inputting the image to be recognized into a recognition model for character recognition to obtain a recognition result of the image to be recognized includes: Determining the question type of the target question type contained in the image to be recognized; Annotating the image to be identified according to the question type; The annotated image to be recognized is input into the recognition model for character recognition to obtain the recognition result.

5. The image processing method according to claim 4, characterized in that: The step of inputting the annotated image to be recognized into the recognition model for character recognition to obtain the recognition result includes: Determine a target recognition module in the recognition model according to the labeled image to be recognized, and process the image to be recognized through a convolutional neural network in the target recognition module to obtain a feature vector; The feature vector is identified according to the long short-term memory network in the target identification module to obtain the identification result.

6. The image processing method according to claim 1, characterized in that: Before the step of selecting a target formula in a preset formula database based on the recognition result is performed, the method further includes: Filtering sub-characters from the recognition results according to a union-find algorithm to form a character set; Accordingly, the step of selecting a target formula from a preset formula database based on the recognition result includes: The recognition result is adjusted according to the sub-characters included in the character set, and the target formula is selected in the preset formula database according to the adjusted recognition result.

7. The image processing method according to claim 6, characterized in that: The step of selecting sub-characters from the recognition result according to the union-find algorithm to form a character set includes: Adding character frames for the sub-characters included in the first character and the sub-characters included in the second character in the image to be recognized; The spacing between adjacent character boxes is calculated according to the adding result, and sub-characters whose spacing is less than a spacing threshold are selected to form the character set.

8. The image processing method according to claim 1, characterized in that: The comparing the second character with the target formula and the operation result of the first character includes: Obtaining a third character according to a calculation result of the first character and the target formula; The third character is compared with the second character.

9. The image processing method according to claim 8, characterized in that: Also includes: In a case where the second character includes at least two sub-characters, comparing the at least two sub-characters with the third character respectively; Accordingly, generating and displaying a target image according to the comparison result and the image to be identified includes: Determining whether any sub-character is equal to the third character according to at least two comparison results; If yes, generating and displaying the target image according to the first display rule and the image to be recognized; If not, the target image is generated and displayed according to the second display rule and the image to be recognized.

10. The image processing method according to claim 1, characterized in that: The step of generating and displaying a target image according to the comparison result and the image to be identified includes: When the second character and the operation result are equal, selecting a first display template from the preset display templates to integrate with the image to be recognized, generating the target image and displaying it; In the case that the second character is not equal to the operation result, a second display template is selected from the preset display templates to be integrated with the image to be recognized to generate the target image and display it.

11. An image processing device, characterized in that: include: An image acquisition module is configured to acquire an image to be recognized; A model recognition module is configured to input the image to be recognized into a recognition model for character recognition, and obtain a recognition result of the image to be recognized, wherein the recognition result includes a first character and a second character, wherein the image to be recognized includes a calculation question; a character comparison module, configured to preset different formula databases for different question types, select a target formula from the preset formula database based on the recognition result, and compare the second character with the target formula and the calculation result of the first character, wherein the target formula refers to the formula for correcting the recognized character, the target formula corresponds to the calculation question, and is used to calculate the result of the calculation question; An image display module is configured to generate and display a target image according to the comparison result and the image to be identified; Among them, the character comparison module is further configured to filter the target formula according to the priority order of the formulas in the preset formula database based on the first character; or to filter the target formula according to the priority order of the formulas in the preset formula database based on the first character and the second character.

12. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions: Obtain an image to be recognized; Inputting the image to be recognized into a recognition model for character recognition to obtain a recognition result of the image to be recognized, the recognition result including a first character and a second character, wherein the image to be recognized includes a calculation question; Preset different formula databases for different question types, select a target formula from the preset formula database based on the recognition result, and compare the second character with the target formula and the calculation result of the first character, wherein the target formula refers to the formula for correcting the recognized character, the target formula corresponds to the calculation question, and is used to calculate the result of the calculation question; Generate and display a target image based on the comparison result and the image to be identified; Among them, the filtering of the target formula in a preset formula database based on the recognition result includes: filtering the target formula in the preset formula database according to the priority order of the formula based on the first character; or filtering the target formula in the preset formula database according to the priority order of the formula based on the first character and the second character.

13. A computer-readable storage medium, characterized in that: It stores computer instructions, which, when executed by a processor, implement the steps of the image processing method described in any one of claims 1 to 10.

14. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the image processing method according to any one of claims 1 to 10.

Images