Text end-to-end recognition method and device, electronic equipment and storage medium

Abstract

The invention discloses a text end-to-end recognition method and device, electronic equipment and a storage medium, and relates to the technical field of computer vision, and the main technical scheme comprises the steps that a to-be-recognized image is acquired, the to-be-recognized image comprises target question type information, and the target question type information is a question related to at least one of pinyin and Chinese characters; and inputting the to-be-recognized image into a pre-trained target recognition model for image recognition, and outputting at least one of structure information, text information and answer information corresponding to the target question type information. The problem of overall result failure caused by a single model fault is effectively avoided, an information combination process brought by multi-model cooperation is simplified, and unified end-to-end identification of structural information, text information and answer information is realized.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and more particularly to an end-to-end text recognition method and apparatus, electronic device and storage medium. Background Technology

[0002] Visual AI grading is an important application of educational technology in recent years, combining multiple artificial intelligence technologies to automatically grade subjective and objective questions. Currently, visual AI grading solutions involve using multiple models to separately acquire text information, answer information, symbol information, etc., and then combining them. If any one model malfunctions, the final result will be incorrect. Summary of the Invention

[0003] This disclosure provides an end-to-end text recognition method, apparatus, electronic device, and storage medium. Its main purpose is to simplify the information combination process brought about by multi-model collaboration and achieve unified end-to-end recognition of structural information, textual information, and response information.

[0004] According to a first aspect of this disclosure, an end-to-end text recognition method is provided, comprising: Obtain an image to be identified, wherein the image contains target question type information, and the target question type information is a question related to at least one of Pinyin and Chinese characters; The image to be recognized is input into a pre-trained target recognition model for image recognition, and at least one of the structural information, text information, and answer information corresponding to the target question type information is output.

[0005] In some embodiments, before inputting the image to be recognized into a pre-trained target recognition model for image recognition and outputting at least one of the structural information, text information, and answer information corresponding to the target question type information, the method further includes: Generate training images, which include at least one of the following: images of Chinese characters written from pinyin, images of target Chinese characters written from pinyin, images of four-line grid questions, and images of pinyin annotation questions with dots. The target recognition model is obtained by training the recognition model based on the training images; wherein the recognition model is an end-to-end model.

[0006] In some embodiments, generating training images includes: Configure the font information in the training images, including font category, font size, and font color; the font category in the answer area is handwritten font, and the font category in the printing area is printed font. Configure the background information of the training images, wherein the background information includes at least one of the background texture of real paper, paper wrinkles, and slight stain effects; Configure the target question type information; Configure the text fill format corresponding to the target question type information; According to the configured target question type information, the text filling format corresponding to the target question type information, and the font information, the corresponding text is randomly selected from the preset corpus to generate text information; Image information is generated based on the background information and text information of the training image, and the image information is labeled to obtain the training image.

[0007] In some embodiments, training the recognition model based on the training images to obtain the target recognition model includes: A first learning rate is set, and the recognition model is trained iteratively for multiple rounds based on the first learning rate and the training images until the recognition accuracy of the recognition model reaches a preset initial threshold, thus obtaining the initial training recognition model. Obtain real-life images corresponding to the target question type information in a real-world scenario, wherein the real-life images include correct structural information, text information, and answer information; A second learning rate is set, and the cross-entropy loss function value of the initial training recognition model is calculated based on the second learning rate and the real-shot image. The parameters of the initial training recognition model are then adjusted using gradient descent based on the loss function value until the loss function value is less than a preset loss threshold, thereby obtaining the target recognition model.

[0008] In some embodiments, according to the configured target question type information, the text filling format corresponding to the target question type information, and the font information, the corresponding text is randomly selected from a preset corpus to generate text information, including: If the text filling format is a question type of writing Chinese characters based on pinyin or a question type of writing target characters based on Chinese characters, then the target pinyin will be filled to the area directly above the outside of the grid, and the target Chinese character of the target pinyin will be filled to; If the text filling format is the filling format of the four-line grid question type, then the target pinyin will be filled into the specified grid line area of ​​the four-line grid. If the text filling format is a format for adding pinyin to characters, then the target Chinese character is marked, and the target pinyin corresponding to the marked target Chinese character is filled into the area directly above the target Chinese character.

[0009] In some embodiments, before annotating the image information to obtain the training image, the method further includes: The visual feature extraction module of the recognition model is used to detect edge regions in the image information and remove blank edge regions in the image information that have no title information or text content. Based on the recognition model, the text recognizability of the edge-processed image information is detected, and images containing unrecognizable blurry text or rare characters are removed.

[0010] According to a second aspect of this disclosure, an end-to-end text recognition device is provided, comprising: An acquisition unit is used to acquire an image to be identified, wherein the image to be identified contains target question type information, and the target question type information is a question related to at least one of pinyin and Chinese characters. The determining unit is used to input the image to be recognized into a pre-trained target recognition model for image recognition, and output at least one of the structural information, text information, and answer information corresponding to the target question type information.

[0011] In some embodiments, before inputting the image to be recognized into a pre-trained target recognition model for image recognition and outputting at least one of the structural information, text information, and answer information corresponding to the target question type information, the apparatus further includes: The generation unit is used to generate training images, which include at least one of the following: images of Chinese characters written from pinyin, images of target Chinese characters written from pinyin, images of four-line grid questions, and images of pinyin annotation questions with dots. The training unit is used to train the recognition model based on the training images to obtain the target recognition model; wherein the recognition model is an end-to-end model.

[0012] In some embodiments, the generating unit is further configured to: Configure the font information in the training images, including font category, font size, and font color; the font category in the answer area is handwritten font, and the font category in the printing area is printed font. Configure the background information of the training images, wherein the background information includes at least one of the background texture of real paper, paper wrinkles, and slight stain effects; Configure the target question type information; Configure the text fill format corresponding to the target question type information; According to the configured target question type information, the text filling format corresponding to the target question type information, and the font information, the corresponding text is randomly selected from the preset corpus to generate text information; Image information is generated based on the background information and text information of the training image, and the image information is labeled to obtain the training image.

[0013] In some embodiments, the training unit is further configured to: A first learning rate is set, and the recognition model is trained iteratively for multiple rounds based on the first learning rate and the training images until the recognition accuracy of the recognition model reaches a preset initial threshold, thus obtaining the initial training recognition model. Obtain real-life images corresponding to the target question type information in a real-world scenario, wherein the real-life images include correct structural information, text information, and answer information; A second learning rate is set, and the cross-entropy loss function value of the initial training recognition model is calculated based on the second learning rate and the real-shot image. The parameters of the initial training recognition model are then adjusted using gradient descent based on the loss function value until the loss function value is less than a preset loss threshold, thereby obtaining the target recognition model.

[0014] In some embodiments, according to the configured target question type information, the text filling format corresponding to the target question type information, and the font information, the corresponding text is randomly selected from a preset corpus to generate text information, including: If the text filling format is a question type of writing Chinese characters based on pinyin or a question type of writing target characters based on Chinese characters, then the target pinyin will be filled to the area directly above the outside of the grid, and the target Chinese character of the target pinyin will be filled to; If the text filling format is the filling format of the four-line grid question type, then the target pinyin will be filled into the specified grid line area of ​​the four-line grid. If the text filling format is a format for adding pinyin to characters, then the target Chinese character is marked, and the target pinyin corresponding to the marked target Chinese character is filled into the area directly above the target Chinese character.

[0015] In some embodiments, the generation unit is further configured to Before annotating the image information to obtain the training image, the visual feature extraction module of the recognition model is used to detect edge regions of the image information and remove blank edge regions in the image information that have no title information or text content. Based on the recognition model, the text recognizability of the edge-processed image information is detected, and images containing unrecognizable blurry text or rare characters are removed.

[0016] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.

[0017] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect above.

[0018] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0019] The end-to-end text recognition method, apparatus, electronic device, and storage medium disclosed herein mainly include the following technical solutions: acquiring an image to be recognized, wherein the image to be recognized contains target question type information, wherein the target question type information is a question related to at least one of pinyin and Chinese characters; inputting the image to be recognized into a pre-trained target recognition model for image recognition, and outputting at least one of structural information, text information, and answer information corresponding to the target question type information. Compared with the prior art scheme of using multiple models to recognize and then combine them separately, this effectively avoids the problem of overall result failure due to the failure of a single model, simplifies the information combination process brought about by multi-model collaboration, and realizes unified end-to-end recognition of structural information, text information, and answer information.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating an end-to-end text recognition method provided in an embodiment of the present disclosure. Figure 2 A flowchart illustrating another end-to-end text recognition method provided in this application embodiment; Figure 3 A flowchart illustrating another end-to-end text recognition method provided in this application embodiment; Figure 4 A schematic diagram of the structure of an end-to-end text recognition device provided in an embodiment of this disclosure; Figure 5 A schematic diagram of the structure of an end-to-end text recognition device provided in an embodiment of this disclosure; Figure 6 A schematic block diagram of an example electronic device provided for embodiments of this disclosure. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] The following description, with reference to the accompanying drawings, describes an end-to-end text recognition method, apparatus, electronic device, and storage medium according to embodiments of the present disclosure.

[0024] Figure 1 This is a flowchart illustrating an end-to-end text recognition method provided in an embodiment of the present disclosure.

[0025] like Figure 1 As shown, the method includes the following steps: Step 101: Obtain the image to be identified, wherein the image to be identified contains target question type information, wherein the target question type information is a question related to at least one of Pinyin and Chinese characters.

[0026] In some embodiments, the image to be identified is specific and directional, meaning that the image contains target question type information. The target question type information can cover various forms, including the spelling of pinyin, such as combinations of initials, finals, and tones, to allow users to judge whether the answer is correct or not; it may also include writing the corresponding pinyin based on Chinese characters, or filling in appropriate Chinese characters or performing numerical calculations based on pinyin, etc.

[0027] The images to be recognized can be obtained by photographing the target question types on paper test papers or workbooks using electronic devices (such as mobile phones or cameras); they can also be images specifically designed for target question type recognition downloaded from online resource libraries; or screenshots containing the target question types can be extracted from the question banks provided by educational software. During the acquisition of the images, it is crucial to ensure image quality, including sufficient clarity, to guarantee accurate identification of the target question type information and avoid recognition errors or failures due to image blurring. Preprocessing operations, such as image cropping and brightness adjustment, can also be performed on the acquired images to better extract the target question type information, preparing them thoroughly for subsequent recognition steps.

[0028] Step 102: Input the image to be recognized into the pre-trained target recognition model for image recognition, and output at least one of the structural information, text information, and answer information corresponding to the target question type information.

[0029] In some embodiments, the target recognition model is pre-designed and trained to accurately analyze and extract various types of information from an image. When an image to be recognized is input into the target recognition model, the model uses its complex internal algorithms and pattern recognition mechanisms to meticulously scan and analyze the image. During the text recognition process, the target recognition model identifies and processes the text portions of the image. Regarding structural information, the model can identify the layout structure of the question within the image, such as whether the question is a pinyin-based question, a character-based question, a four-line grid question, or a question with added pinyin. This structural information helps to more clearly understand the organization of the question.

[0030] Text information recognition targets the specific text content within images, including the question stem, pinyin characters, and Chinese characters in pinyin questions. Accurately identifying this text information is fundamental for subsequent analysis. Answer information includes completed answers, such as handwritten pinyin answers and selected options. By recognizing answer information, the answer performance can be evaluated and analyzed. Symbolic information is also crucial in target question types, such as tone marks and apostrophes in pinyin. The target recognition model can accurately identify these symbols to correctly understand the pronunciation and meaning of pinyin.

[0031] During the recognition process, the target recognition model may focus on identifying at least one of the following: structural information, textual information, answer information, and symbolic information, depending on different application scenarios and needs. Alternatively, it may simultaneously recognize multiple types of information to comprehensively acquire key information from the image, providing accurate and detailed data for subsequent processing, such as question grading and data analysis. This step plays a crucial role in the entire image-based target question processing workflow, transforming the acquired image information into processable and analyzable structured data, laying a solid foundation for intelligent question processing and evaluation.

[0032] The end-to-end text recognition method disclosed herein acquires an image to be recognized, the image containing target question type information, the target question type information being a question related to at least one of pinyin and Chinese characters; the image to be recognized is input into a pre-trained target recognition model for image recognition, and outputs at least one of structural information, text information, and answer information corresponding to the target question type information. Compared with the existing technology that uses multiple models to recognize and then combine them separately, this method effectively avoids the problem of overall result failure due to the failure of a single model, simplifies the information combination process brought about by multi-model collaboration, and achieves unified end-to-end recognition of structural information, text information, and answer information.

[0033] In some embodiments, before inputting the image to be recognized into a pre-trained target recognition model for image recognition and outputting at least one of the structural information, text information, answer information, and symbol information corresponding to the target question type information, such as Figure 2 As shown, the method further includes: Step 201: Generate training images, which include at least one of the following: images of Chinese characters written from pinyin, images of target Chinese characters written from pinyin, images of four-line grid questions, and images of pinyin annotation questions with dots.

[0034] Step 202: Train the recognition model based on the training images to obtain the target recognition model; wherein the recognition model is an end-to-end model.

[0035] In some embodiments, the training images include at least one of the following: grid-based question images (such as images of writing Chinese characters based on pinyin, images of writing target characters based on Chinese characters), grid-based question images, four-line grid question images, and images of question images with pinyin annotations for dotted characters.

[0036] The recognition model (which is an untrained recognition model or a recognition model to be updated) is trained based on the training images, so that the end-to-end recognition model has the ability to recognize the target question type.

[0037] In some embodiments, the training images encompass target question type information for different question types. For example, some training images include pinyin spelling questions to help the object recognition model learn to recognize the features of various pinyin combinations; others include questions that require writing Chinese characters based on pinyin and vice versa, allowing the recognition model to become familiar with how these types of questions are presented in images. The training images are also diverse in format, covering various image formats such as common JPEG and PNG formats, as well as BMP formats for some specific scenarios. Using training images containing target question type information of different types and formats to train the object recognition model addresses the problem in real-world application scenarios where the images containing target question types are from a wide range of sources and are diverse in form.

[0038] Through learning from a large number of training images, the recognition model can accurately distinguish between different styles of Pinyin and Chinese characters, whether they are standard printed fonts or handwritten fonts. The model can also learn the layout characteristics of different question types; for example, multiple-choice questions may have specific option arrangements, while fill-in-the-blank questions have unique text box designs. During training, the target recognition model continuously optimizes its parameters, gradually improving the accuracy of recognizing various types of information in images. When faced with a new image to be recognized, the well-trained target recognition model can accurately identify structural information in the image, understand the positional relationships of different parts of the question, recognize text information, clearly distinguish the text in the question stem and answer, recognize answer information, obtain the answers left by the test taker, and recognize symbol information, making correct judgments about various symbols in Pinyin. Through this comprehensive and in-depth training, the target recognition model can truly possess the image recognition capabilities to handle complex real-world scenarios, providing strong and reliable technical support for the subsequent input of images to be recognized into the model for question text recognition, ensuring the smooth progress of the entire image recognition and analysis process.

[0039] In some embodiments, the method of generating training images in step 201 can be implemented in, but is not limited to, the following ways: Figure 3 As shown, the method includes: Step 301: Configure the font information in the training image. The font information includes font category, font size, and font color. The font category in the answer area is handwritten font, and the font category in the printing area is printed font.

[0040] Set font parameters; wherein, the font parameters include font type, font size and color, and the font type is divided into printed font and handwritten font.

[0041] In some embodiments, font parameters encompass font type, such as different styles like Song, Kai, and Hei; font size, as different font sizes have varying effects on image presentation and readability; and attributes such as font color and weight. These parameters all affect the visual representation of text and pinyin in simulated real-world scenarios. A simulated real-world scenario is the foundational environment for constructing training images. It can simulate the appearance and layout of exam papers, workbooks, etc., in actual exams or practice sessions, including paper color and texture, and the presence of elements such as borders, headers, and footers. The target question type format determines the arrangement and organization of text and pinyin in the simulated real-world scenario. For example, different question types, such as multiple-choice, fill-in-the-blank, and matching questions, have their own unique format requirements.

[0042] Step 302: Configure the background information of the training image, wherein the background information includes at least one of the background texture of real paper, paper wrinkles, and slight stain effects; Determine the target question format and simulate a real-world scenario; the simulated real-world scenario includes at least one of the following: background texture of real paper, paper wrinkles, and slight stain effects.

[0043] Step 303: Configure the target question type information; configure the text filling format corresponding to the target question type information.

[0044] Due to the different formats of the target questions, the types of text filled into the simulated real-world scenarios will also vary. The text types mainly include pinyin and regular characters. For example, when the target question format is a pinyin spelling question, Chinese characters are filled into the simulated scene, allowing the target recognition model to learn to recognize the characters and generate the correct pinyin according to pinyin rules. In questions requiring writing Chinese characters based on pinyin, pinyin is filled first, requiring the model to accurately recognize the pinyin and associate it with the corresponding Chinese characters. In multiple-choice questions, both the question stem and the options may contain both characters and pinyin; the model needs to learn to recognize this mixed information and their layout relationships under specific formats. For fill-in-the-blank questions, depending on the specific question requirements, either characters or pinyin are filled in, and the model must be able to adapt to different filling needs and accurately recognize them. During the filling process, font parameters play a crucial role, determining the presentation style of the characters and pinyin, making the generated training images more realistic and diverse. By reasonably filling in the target question format and font parameters, various training images that conform to actual application scenarios can be generated. These images contain rich text and pinyin information, providing high-quality training data for the target recognition model. This helps the model to better learn and recognize image content in different situations, improving its recognition ability and accuracy in practical applications. This enables the model to extract structural information, text information, answer information, and symbol information more efficiently and accurately when processing images containing target question information in the future.

[0045] Step 304: According to the configured target question type information, the text filling format corresponding to the target question type information, and the font information, randomly select the corresponding text from the preset corpus to generate text information.

[0046] Construct a pre-defined corpus, and randomly select texts (target-specific terms) corresponding to the target question type information from the pre-defined corpus.

[0047] Taking the "Recognize and Write Chinese Characters" question type as an example, characters are randomly selected from a pre-set corpus, and their corresponding standard pinyin is generated. The characters and their corresponding pinyin are then combined to form a complete text, yielding the text information. The pre-set corpus is a vast collection of language resources, containing a rich variety of textual content, covering different vocabulary, grammatical structures, and semantic categories. When selecting characters from the pre-set corpus, various factors are considered, such as the commonness, diversity, and frequency of use in practical applications. The selected characters may include everyday expressions, academic terms, idioms, and colloquialisms. After determining the selected characters, the next step is to generate their corresponding pinyin. Using pinyin rules, each Chinese character is converted into its correct pinyin form, including accurate marking of initials, finals, and tones.

[0048] In some embodiments, if the text filling format is a text filling format for a question type of writing Chinese characters based on pinyin or a question type of writing target characters based on Chinese characters, then the target pinyin is filled to the area directly above the outside of the grid, and the target Chinese character of the target pinyin is filled to; If the text filling format is the filling format of the four-line grid question type, then the target pinyin will be filled into the specified grid line area of ​​the four-line grid. If the text filling format is a format for adding pinyin to characters, then the target Chinese character is marked, and the target pinyin corresponding to the marked target Chinese character is filled into the area directly above the target Chinese character.

[0049] Specifically, the present disclosure does not limit the type of target question, and the above description is only an example and not a specific limitation.

[0050] In some embodiments, the specific filling operation needs to match different target question formats: If it is a grid filling format, the target Chinese character is filled into the center of the grid, ensuring that the Chinese character is evenly distributed and does not exceed the boundary. The corresponding standard pinyin is filled into the outer area of ​​the grid, such as above or to the right, and appropriate spacing is maintained to avoid overlap. If it is a four-line grid filling format, the initials and finals of the target standard pinyin are filled into the designated grid areas of the four-line grid according to the pinyin writing standard, ensuring that the positions of each part are accurate. If it is a dotted character annotation format, the target special term is first annotated, such as marking a dot above the Chinese character, and then the corresponding standard pinyin is filled into the outer area of ​​the target special term, such as to the right, maintaining a fixed horizontal spacing to ensure clear correspondence, thereby generating an initial training image that meets the training requirements.

[0051] Step 305: Generate image information according to the background information and text information of the training image.

[0052] The image information is generated by filling the background information of the training image with text information (simulating a real-world scenario). The target question type format is determined based on the question types that may appear in actual applications. For example, different question types such as multiple choice, fill-in-the-blank, and true / false questions each have their own unique formats and layouts. When simulating a real-world scenario, various situations that may occur during the actual answering process are considered, such as the layout of the test paper, the size and style of the font, and the spacing between questions. The previously selected and generated pinyin text is reasonably filled into the simulated scene according to the requirements of the target question type format. For example, when generating the training image for multiple choice questions, the questions and options are arranged in a suitable way, and the positions of the text and pinyin are adjusted according to the format requirements; for fill-in-the-blank questions, corresponding fill-in-the-blank areas are set, and the text and pinyin are accurately placed in the appropriate positions. During the filling process, some details are also considered, such as whether to add a background pattern or whether to use different colors to distinguish different parts, so that the generated training image is as close as possible to the real exam or practice scenario. Through this series of operations, image information containing rich text and pinyin information and conforming to the target question type format is finally generated.

[0053] To generate training data adapted to the recognition model, target-specific terms and corresponding standard pinyin need to be filled into a simulated real-world scenario according to the target question format and preset font parameters, thereby generating image information. Target-specific terms refer to Chinese characters used for pinyin-specific question training, selected from commonly used corpora in the Chinese language field, covering commonly tested characters at different grade levels to ensure the training data aligns with actual teaching needs; standard pinyin is generated according to the national common language and script pinyin rules, fully including initials, finals, and tones, ensuring the standardization of pinyin annotation; the simulated real-world scenario recreates the paper texture, slight wrinkles, natural edge wear, and minor stains of assignments or test papers, making the generated image closer to the actual question scene; font parameters include font type, font size, and color. Printed fonts are used to present the fixed text of the question, handwritten fonts simulate user answers, the font size adaptively adjusts according to the question size to ensure text clarity, and the color is primarily black, with contrasting colors optional for annotation. Specifically, this disclosure does not limit the scope of the embodiments.

[0054] This image information provides sufficient and effective data support for the subsequent training of the target recognition model, enabling the recognition model to learn the characteristics and patterns of target question information under different question types and formats from diverse training data, thereby improving its ability to recognize images and laying a solid foundation for the subsequent accurate recognition of structural information, text information, answer information and symbol information in the image to be recognized.

[0055] Step 306: Use the visual feature extraction module of the recognition model to perform edge region detection on the image information and remove blank edge regions in the image information that have no title information or text content; Based on the recognition model, perform text recognizability detection on the edge-processed image information and remove images containing unrecognizable blurry text or rare characters.

[0056] During the generation of training images, due to the construction of simulated real-world scenes and infilling operations, the edges of the images may contain some irrelevant or distracting information. This edge information may include boundary decorations of the simulated scene, redundant blank areas, or irrelevant elements generated during the generation process. After previous design and initial training, the object recognition model possesses a certain image analysis capability, able to identify which parts of the image are related to the core target question information and which parts are removable edges. By using the object recognition model to process the training images and remove these edges, the images become more concise and focused, highlighting the key information in the image, namely the text, pinyin, and the question content they constitute. This allows the model to focus more on learning the core information when subsequently trained using these training images, avoiding interference from irrelevant edge information, thereby improving training efficiency and accuracy.

[0057] Recognizableness is a crucial metric for evaluating the quality of training images, directly impacting the accuracy with which target recognition models can extract structural, textual, response, and symbolic information from images. When selecting training images, multiple factors are considered to assess their recognizability. Image clarity is critical; if an image is blurry and the strokes of characters and pinyin are difficult to discern, the target recognition model cannot accurately identify the information, rendering such images unsuitable for training. The layout of characters and pinyin within the image, as well as any overlap or occlusion, also affects recognizability. If characters and pinyin overlap or are obscured by other elements, the target recognition model will encounter difficulties or errors during recognition. Furthermore, factors such as color contrast and brightness also influence recognizability. By comprehensively considering these factors, training images are selected, retaining those with high recognizability and discarding those with obvious recognition problems. This ensures that the images used to train the target recognition model are of high quality, thereby improving the training effect of the model and enabling it to more accurately identify various information in the image to be recognized in practical applications, providing reliable support for subsequent task processing, evaluation and other work.

[0058] Step 307: Annotate the image information to obtain the training image.

[0059] In some embodiments, when training the recognition model based on the training images to obtain the target recognition model, the following methods can be used, but are not limited to: setting a first learning rate, performing multiple rounds of iterative training on the recognition model based on the first learning rate and the training images until the recognition accuracy of the recognition model reaches a preset initial threshold, thereby obtaining an initial training recognition model; acquiring real-world images corresponding to the target question type information in a real-world scenario, wherein the real-world images include correct structural information, text information, and answer information; setting a second learning rate, calculating the cross-entropy loss function value of the initial training recognition model based on the second learning rate and the real-world images, and adjusting the parameters of the initial training recognition model using gradient descent based on the loss function value until the loss function value is less than a preset loss threshold, thereby obtaining the target recognition model.

[0060] At the start of training, using a larger learning rate allows the recognition model's parameters to move rapidly within the parameter space, accelerating the model's convergence. The training images, as mentioned earlier, are carefully generated and contain target question information of different types and formats, providing a rich and diverse set of learning samples for the model. By continuously inputting these training images into the recognition model and adjusting the model's parameters based on the difference between the model's output and the actual results (if labeled), multiple iterations of training are performed. In each iteration, the model processes the training images according to the current parameter settings, attempting to learn various information features in the images, such as structural information, textual information, response information, and symbolic information. As the number of iterations increases, the model gradually adapts to the data distribution and patterns in the training images, and the parameters are continuously optimized.

[0061] After a certain number of iterations of training with a large learning rate, an initial training recognition model is obtained, which already possesses a certain ability to process images. At this point, the initial training model is used again to predict the training images, and the model outputs its recognition results for various information in the images. By comparing these prediction results with the true results (such as the labeled correct answer), the value of the loss function can be calculated. The loss function is an indicator that measures the difference between the model's prediction and the true result; the smaller its value, the closer the prediction result of the initial training recognition model is to the true result. A small learning rate is used at this stage because the initial training recognition model is close to convergence, and a smaller learning rate allows the model to fine-tune its parameters, avoiding missing the optimal solution or oscillating around the optimal solution due to excessive parameter adjustments. Based on the calculated loss function value, the parameters of the initial training recognition model are fine-tuned using techniques such as backpropagation, so that the prediction results of the initial training recognition model can more accurately match the true results. By repeatedly performing this process of prediction, loss function calculation, and parameter adjustment, the initial training recognition model continuously optimizes its performance and gradually acquires the ability to accurately identify various images containing target question information, thus obtaining a target recognition model. This lays a solid foundation for the subsequent efficient and accurate processing of images to be recognized.

[0062] Corresponding to the aforementioned end-to-end text recognition method, this disclosure also proposes an end-to-end text recognition device. Since the device embodiments of this disclosure correspond to the aforementioned method embodiments, details not disclosed in the device embodiments can be referred to the aforementioned method embodiments, and will not be repeated in this disclosure.

[0063] Figure 4 This is a schematic diagram of the structure of an end-to-end text recognition device provided in an embodiment of this disclosure, as shown below. Figure 4 As shown, it includes: Acquisition unit 41 is used to acquire an image to be identified, wherein the image to be identified contains information on pinyin-related special questions, wherein the pinyin-related special questions are questions with special structures related to pinyin. The recognition unit 42 is used to input the image to be recognized into a pre-trained end-to-end recognition model, and to perform question text recognition on the image to be recognized through the end-to-end recognition model, and output at least one of the structural information, text information, answer information and symbol information of the pinyin special question; wherein, the end-to-end recognition model includes a visual feature extraction module and a natural language processing module.

[0064] The end-to-end text recognition device provided in this disclosure acquires an image to be recognized, the image containing target question type information, which is a question related to at least one of pinyin and Chinese characters; the image to be recognized is input into a pre-trained target recognition model for image recognition, and outputs at least one of structural information, text information, and answer information corresponding to the target question type information. Compared with the existing technology that uses multiple models to recognize and then combine them separately, this device effectively avoids the problem of overall result failure due to the failure of a single model, simplifies the information combination process brought about by multi-model collaboration, and realizes unified end-to-end recognition of structural information, text information, and answer information.

[0065] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 5 As shown, before inputting the image to be recognized into a pre-trained target recognition model for image recognition and outputting at least one of the structural information, text information, and answer information corresponding to the target question type information, the device further includes: The generation unit 43 is used to generate training images, which include at least one of the following: images of Chinese characters written from pinyin, images of target Chinese characters written from pinyin, images of four-line grid questions, and images of pinyin annotation questions with dots. Training unit 44 is used to train the recognition model based on the training images to obtain the target recognition model; wherein the recognition model is an end-to-end model.

[0066] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 5 As shown, the generation unit 43 is further configured to: Configure the font information in the training images, including font category, font size, and font color; the font category in the answer area is handwritten font, and the font category in the printing area is printed font. Configure the background information of the training images, wherein the background information includes at least one of the background texture of real paper, paper wrinkles, and slight stain effects; Configure the target question type information; Configure the text fill format corresponding to the target question type information; According to the configured target question type information, the text filling format corresponding to the target question type information, and the font information, the corresponding text is randomly selected from the preset corpus to generate text information; Image information is generated based on the background information and text information of the training image, and the image information is labeled to obtain the training image.

[0067] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 5As shown, the training unit 44 is also used for: A first learning rate is set, and the recognition model is trained iteratively for multiple rounds based on the first learning rate and the training images until the recognition accuracy of the recognition model reaches a preset initial threshold, thus obtaining the initial training recognition model. Obtain real-life images corresponding to the target question type information in a real-world scenario, wherein the real-life images include correct structural information, text information, and answer information; A second learning rate is set, and the cross-entropy loss function value of the initial training recognition model is calculated based on the second learning rate and the real-shot image. The parameters of the initial training recognition model are then adjusted using gradient descent based on the loss function value until the loss function value is less than a preset loss threshold, thereby obtaining the target recognition model.

[0068] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 5 As shown, according to the configured target question type information, the text filling format corresponding to the target question type information, and the font information, the corresponding text is randomly selected from the preset corpus to generate text information including: If the text filling format is a question type of writing Chinese characters based on pinyin or a question type of writing target characters based on Chinese characters, then the target pinyin will be filled to the area directly above the outside of the grid, and the target Chinese character of the target pinyin will be filled to; If the text filling format is the filling format of the four-line grid question type, then the target pinyin will be filled into the specified grid line area of ​​the four-line grid. If the text filling format is a format for adding pinyin to characters, then the target Chinese character is marked, and the target pinyin corresponding to the marked target Chinese character is filled into the area directly above the target Chinese character.

[0069] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 5 As shown, the generation unit 44 is further configured to: Before annotating the image information to obtain the training image, the visual feature extraction module of the recognition model is used to detect edge regions of the image information and remove blank edge regions in the image information that have no title information or text content. Based on the recognition model, the text recognizability of the edge-processed image information is detected, and images containing unrecognizable blurry text or rare characters are removed.

[0070] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of the embodiments of this disclosure, and the principle is the same. Therefore, the embodiments of this disclosure are not limited thereto.

[0071] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0072] Figure 6 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0073] like Figure 6 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 402 or a computer program loaded from storage unit 408 into RAM (Random Access Memory) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. I / O (Input / Output) interface 405 is also connected to bus 404.

[0074] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0075] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as end-to-end text recognition methods. For example, in some embodiments, the end-to-end text recognition method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the aforementioned end-to-end text recognition method by any other suitable means (e.g., by means of firmware).

[0076] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0077] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0078] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0079] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0080] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0081] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0082] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0083] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0084] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method of end-to-end recognition of text, characterized in that, The method comprises: acquiring an image to be recognized, wherein the image to be recognized contains subject title type information related to at least one of pinyin and Chinese characters; inputting the image to be recognized into a pre-trained target recognition model to perform image recognition, and outputting at least one of structure information, text information, answer information and symbol information corresponding to the subject title type information.

2. The method of claim 1, wherein, Before the image to be recognized is inputted into the pre-trained target recognition model to perform image recognition and output at least one of structure information, text information and answer information corresponding to the subject title type information, the method further comprises: generating a training image, wherein the training image contains at least one of a see pinyin write Chinese character type image, a see Chinese character write subject title type image, a four-line grid type image and a point word phonetic spelling type image; training a recognition model based on the training image to obtain the target recognition model, wherein the recognition model is an end-to-end model.

3. The method of claim 2, wherein, The generation of the training image comprises: configuring font information in the training image, wherein the font information comprises font category, font size and font color; the font category of an answer area is a handwriting font, and the font category of a printing area is a printed font; configuring background information of the training image, wherein the background information comprises at least one of a real paper background texture, paper creases and slight stain effects; configuring the subject title type information; configuring text filling formats corresponding to the subject title type information; randomly selecting corresponding text from a preset corpus according to the configured subject title type information, text filling formats corresponding to the subject title type information and font information to generate text information; generating image information according to the background information of the training image and the text information, and labeling the image information to obtain the training image.

4. The method of claim 2, wherein, The training of the recognition model based on the training image to obtain the target recognition model comprises: setting a first learning rate, performing multi-round iterative training on the recognition model based on the first learning rate and the training image until the recognition accuracy of the recognition model reaches a preset initial threshold to obtain an initial training recognition model; acquiring a real scene corresponding image of the subject title type information, wherein the real scene corresponding image comprises correct structure information, text information and answer information; setting a second learning rate, calculating a cross-entropy loss function value of the initial training recognition model based on the second learning rate and the real scene corresponding image, and adjusting parameters of the initial training recognition model according to the loss function value by using a gradient descent method until the loss function value is less than a preset loss threshold to obtain the target recognition model.

5. The method of claim 2, wherein, Randomly selecting corresponding text from a preset corpus according to the configured subject title type information, text filling formats corresponding to the subject title type information and font information to generate text information comprises: if the text filling format is a see pinyin write Chinese character type text filling format or a see Chinese character write subject title type text filling format, filling a target pinyin into a region directly above an outside of a Chinese character field, and filling a target Chinese character of the target pinyin into a region directly below the Chinese character field; If the text filling format is the filling format of the four-line grid type, the target pinyin is filled into the specified grid line area of the four-line grid; If the text filling format is the filling format of the dot annotation type, the target Chinese character is annotated, and the target pinyin corresponding to the annotated target Chinese character is filled into the area directly above the target Chinese character.

6. The method of claim 2, wherein, Before the image information is labeled to obtain the training image, the method further comprises: using the visual feature extraction module of the recognition model to detect the edge region of the image information, and removing the blank edge region without topic information and text content in the image information; based on the recognition model, the image information after edge processing is detected for text recognition, and images containing blurred text and rare characters that cannot be recognized are removed.

7. An apparatus for end-to-end recognition of text, characterized by comprising: an acquisition unit configured to acquire a to-be-recognized image, the to-be-recognized image containing topic type information related to at least one of pinyin and Chinese characters; a determination unit configured to input the to-be-recognized image into a pre-trained target recognition model to perform image recognition, and output at least one of structure information, text information, and answer information corresponding to the topic type information.

8. An electronic device, comprising: comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to make the computer execute the method according to any one of claims 1-6.

10. A computer program product, characterised in that, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1-6.

Images