Method and device for identifying image content, equipment and medium

Abstract

The embodiment of the invention provides a method and device for identifying image content, equipment and a medium. The scheme comprises the following steps: acquiring a target image comprising an English text handwritten by a user; inputting the target image into a text detection model to obtain position information and size information of a bounding box of each word in English text content output by the text detection model; cutting the target image based on the position information and the size information of the bounding box corresponding to each word to obtain a plurality of cut sub-images; each sub-image contains a word; inputting the plurality of sub-images into a recognition model to obtain word recognition results for each sub-image output by the recognition model; the identification model comprises a TROCR model; and typesetting the word recognition result of each sub-image based on the position information and the size information of the bounding box corresponding to each word to obtain a recognition result for the English text in the target image. According to the method, the handwritten content in the picture can be accurately recognized and typeset.

Description

Technical Field

[0001] This specification relates to the field of image recognition technology, and in particular to a method, apparatus, device, and medium for recognizing image content. Background Technology

[0002] In practical scenarios of English text recognition, users can take photos of their handwritten English text and upload them. The application can then recognize the English text based on the uploaded image and score or correct it.

[0003] However, handwritten English text may be illegible, unclear, or have a messy background.

[0004] However, existing technologies often cannot accurately recognize illegible handwritten words, requiring users to write in a more standardized manner, which limits the application scenarios and affects user experience.

[0005] Therefore, improving the accuracy of recognizing handwritten English text in images has become an urgent technical problem to be solved. Summary of the Invention

[0006] In view of this, embodiments of this specification provide a method for recognizing image content. One or more embodiments of this specification also relate to an apparatus for recognizing image content, a computing device, and a computer-readable storage medium, to address the technical deficiencies existing in the prior art.

[0007] According to a first aspect of the embodiments of this specification, a method for recognizing image content is provided, comprising: Acquire a target image that includes handwritten English text by the user; The target image is input into the text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model. Based on the position and size information of the bounding boxes corresponding to each word, the target image is cropped to obtain multiple cropped sub-images; each sub-image contains one word. The multiple sub-images are input into the recognition model to obtain the word recognition results for each sub-image output by the recognition model; the recognition model includes the TROCR model; Based on the position and size information of the bounding boxes corresponding to each word, the word recognition results of each sub-image are arranged to obtain the recognition result of the English text in the target image.

[0008] According to a second aspect of the embodiments of this specification, an apparatus for recognizing image content is provided, comprising: The acquisition module is used to acquire target images, including handwritten English text by the user; The first input module is used to input the target image into the text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model; The cropping module is used to crop the target image based on the position and size information of the bounding boxes corresponding to each word, to obtain multiple cropped sub-images; each sub-image contains one word; The second input module is used to input the multiple sub-images into the recognition model to obtain the word recognition results for each sub-image output by the recognition model; the recognition model includes the TROCR model. The layout module is used to layout the word recognition results of each sub-image based on the position and size information of the bounding boxes corresponding to each word, so as to obtain the recognition result of the English text in the target image.

[0009] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described method for recognizing image content.

[0010] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the method for recognizing image content described above.

[0011] At least one embodiment provided in this specification can achieve the following beneficial effects: After obtaining a target image including handwritten English text, the target image can be input into a text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model. Based on the position and size information of the bounding boxes corresponding to each word, the target image is cropped to obtain multiple cropped sub-images; wherein each sub-image contains one word; then, the multiple sub-images can be input into a recognition model to obtain the word recognition results for each sub-image output by the recognition model; wherein the recognition model includes a TROCR model. Finally, based on the position and size information of the bounding boxes corresponding to each word, the word recognition results of each sub-image can be arranged to obtain the recognition results of the English text in the target image. In this application, a text detection model is used to determine the bounding boxes of each handwritten English word in the target image. Based on the bounding boxes, the target image is cropped to obtain multiple sub-images. After the sub-images are input into the TROCR model, the TROCR model can accurately recognize the handwritten English words in the sub-images, obtaining accurate recognition results. Subsequently, based on the position and size information of the bounding boxes corresponding to each word, the word recognition results of each sub-image can be arranged to finally determine the recognition result for the entire handwritten English text. By using the TROCR model instead of the CNN model to recognize words in the sub-images, the attention mechanism can be used to grasp the global information of the input word image, and the contextual relationships between the letters of the words in the image can be understood, which can effectively improve the recognition accuracy of handwritten words. Attached Figure Description

[0012] Figure 1 This is a schematic diagram illustrating an application scenario of a method for recognizing image content provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating a method for recognizing image content according to one embodiment of this specification; Figure 3 This is a schematic diagram of the structure of an image content recognition device provided in one embodiment of this specification; Figure 4 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0013] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0014] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0015] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0016] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0017] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0018] TROCR is an end-to-end Transformer-based Optical Character Recognition (OCR) model. Instead of the CNN backbone commonly used in traditional OCR models, TROCR first resizes the input text image and segments it into fixed-size image slices before feeding them into the image Transformer. Its encoder is typically initialized with a ViT-mode pre-trained model (such as DeiT or BEiT) to extract image slice features, while the decoder is initialized with a BERT-mode pre-trained model (such as RoBERTa) to generate recognized text in wordpiece format. TROCR can be pre-trained on large-scale synthetic data and fine-tuned on manually labeled data, requiring no additional language models or complex pre- and post-processing steps. It demonstrates superior performance compared to traditional models in both printed and handwritten text recognition tasks.

[0019] The Detecting Text in Natural Image with Connectionist Text Proposal Network (CTPN) is a deep learning text detection model designed for natural scenes. Based on the Faster R-CNN framework, CTPN focuses on the accurate detection of horizontal or slightly slanted text lines. CTPN uses VGG16 as its backbone to extract image features and innovatively employs fixed-width fine-grained anchor boxes to adapt to the elongated and variable lengths of text. It also introduces a bidirectional LSTM to capture the sequential relationships within the text and incorporates a side-optimization step to fine-tune the text box boundaries, improving localization accuracy. Its core idea is to first generate discrete small text candidate boxes on the feature map, then filter them through multiple tasks such as text or non-text classification, and finally connect and merge these candidate boxes into a complete text region using specific rules. This approach efficiently handles text detection tasks in natural scenes without complex preprocessing.

[0020] The Efficient and Accurate Scene Textdetection pipeline (EAST) is a lightweight end-to-end scene text detection model. EAST eliminates the redundant steps of traditional multi-stage detection, completing detection in only two stages: a fully convolutional network (FCN) and non-maximum suppression (NMS). EAST typically uses PVANet or similar networks as its backbone to extract multi-scale feature maps, then fuses features at different scales through upsampling and feature concatenation via a feature merging branch, adapting to the detection needs of texts of varying sizes. The output layer generates text score feature maps and geometric feature maps, supporting both rotated rectangle (RBOX) and arbitrary quadrilateral (QUAD) text region annotations. It also optimizes training performance through combined loss functions such as balanced cross-entropy. Subsequent local perception NMS post-processing filters out redundant boxes, enabling efficient detection of text in any orientation within natural scenes.

[0021] The Differentiable Binarization Network (DBNet) model is a real-time scene text detection model based on segmentation algorithms. DBNet's core innovation is its differentiable binarization module, which departs from the traditional model's approach of treating binarization as a separate post-processing step requiring manually set fixed thresholds. By using a differentiable function that approximates binarization, the binarization process is integrated into network training. Parameters can be optimized through backpropagation, and the binarization threshold for each pixel can be adaptively adjusted. The model structure includes a backbone network, an FPN network, and a head network. First, the backbone network, in conjunction with the FPN, extracts and fuses multi-scale image features. Then, the head network outputs a probability map and a threshold map of the text region. Combining these two maps yields an approximate binary map. During training, the loss is calculated and optimized for these three maps. During inference, a simple dilation operation yields the text box. It not only simplifies the post-processing workflow and improves detection speed but also accurately handles text of various shapes, including horizontal and curved text. With a lightweight backbone network, it balances detection accuracy and efficiency, making it suitable for various scene text detection tasks.

[0022] The Fast Oriented Text Spotting With a Unified Network (FOTS) model is an end-to-end unified network model for detecting and recognizing text in any orientation. FOTS uses ResNet-50 as its backbone and leverages FPN to fuse high- and low-level semantic features to obtain a shared feature map. Its core consists of four parts: shared convolutions, a text detection branch, a RoIRotate operation, and a text recognition branch. The detection branch borrows the fully convolutional network design from the EAST model, outputting information such as the probability of positive samples per pixel, distance to the four sides of the bounding box, and the orientation of the bounding box. The results are obtained after thresholding and NMS. RoIRotate uses affine transformation to convert features of slanted text regions into horizontal feature maps, solving the misalignment problem between features and regions. The recognition branch uses VGG-like convolutions, bidirectional LSTM, and a CTC decoder to perform text recognition on the transformed features. This model reduces computational overhead by sharing convolutional features, can be jointly trained end-to-end, and can efficiently handle multi-angle text detection and recognition tasks in natural scenes.

[0023] Current technologies typically train classic Convolutional Neural Network (CNN) models using publicly available datasets, and then use these trained CNN models to recognize English text in images uploaded by users that contain handwritten English text. However, CNN models can only capture local features of images, such as the features of a single letter in an input word image. They struggle to grasp deeper, more global information about the image, such as the relationships between the letters. Therefore, CNN models have low accuracy in recognizing illegible English text. Furthermore, using CNN models requires users to have relatively neat handwriting, which limits their application scenarios and impacts user experience.

[0024] To address the deficiencies in related technologies, this specification provides a method for recognizing image content. One or more embodiments of this specification also relate to an apparatus for recognizing image content, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0025] See Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of a method for recognizing image content provided in one embodiment of this specification.

[0026] like Figure 1 As shown, the application client 101 can be installed on the user's terminal device. After the user completes the English essay by hand, they can take a picture and upload it. The target image, including the user's handwritten English essay, is uploaded to the application client 101, and the application client 101 can send the uploaded target image to the application server 102.

[0027] After receiving the target image, including the user's handwritten English essay, the application server 102 can input the target image into a text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model. Based on the position and size information of the bounding boxes corresponding to each word, the target image is cropped to obtain multiple cropped sub-images, each containing one English word. Further, the application server 102 can input these sub-images into a TROCR model to obtain the word recognition results for each sub-image output by the TROCR model. Based on the position and size information of the bounding boxes corresponding to each word, the word recognition results for each sub-image are formatted to obtain the recognition result of the English essay in the target image.

[0028] In practical applications, after the application server 102 determines the recognition result of the English text in the target image, it can input the recognition result into the scoring model. The scoring model can score the user's handwritten English essay, and can also correct the user's handwritten English essay and provide suggestions for improvement. Alternatively, the recognition result can be scored or corrected manually, without specific limitations.

[0029] Subsequently, the application server 102 can also provide the essay score and / or revision suggestions to the application client 101. After receiving the essay score and / or revision suggestions, the application client 101 can display the essay score and / or revision suggestions to the user.

[0030] The user's terminal device can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, object content computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs).

[0031] The application server 102 can include, but is not limited to, any device, equipment, platform, or server cluster with computing and processing capabilities. Specifically, the server at the application server 102 can be a standalone physical server, a server cluster consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0032] Application client 101 and application server 102 can transmit data through a local area network connection, a wide area network connection, an Internet connection or other types of data network connection, or through other means, without specific limitations.

[0033] Figure 1The method described above uses a text detection model to determine the bounding boxes of each handwritten English word in the target image. Based on these bounding boxes, the target image is cropped into multiple sub-images. These sub-images are then input into the TROCR model, which accurately identifies the handwritten English words within each sub-image, yielding accurate recognition results. Furthermore, based on the position and size information of the bounding boxes corresponding to each word, the word recognition results from each sub-image are arranged to ultimately determine the recognition result for the entire handwritten English text. By using the TROCR model instead of a CNN model to recognize words in sub-images, the attention mechanism can be utilized to grasp the global information of the input word image, understanding the contextual relationships between the letters of the words, thus effectively improving the accuracy of handwritten word recognition.

[0034] See Figure 2 , Figure 2 This is a flowchart illustrating a method for recognizing image content according to one embodiment of this specification. From a hardware perspective, the entity executing this process can be a server. From a program perspective, the entity executing this process can be an image recognition program hosted on the server.

[0035] like Figure 2 As shown, the process may include the following steps: Step 202: Obtain the target image including the user's handwritten English text.

[0036] In this embodiment of the specification, the target image can be an image taken by the user of their handwritten English text. The target image may include the user's handwritten English text. This English text may include, but is not limited to, English essay text, English paragraph text, English sentence text, etc.

[0037] In practical applications, after a user completes handwriting English text, they can take a photo and upload it. The target image, including the user's handwritten English text, is uploaded to the application client, which can then send the uploaded target image to the application server for subsequent recognition processing.

[0038] Step 204: Input the target image into the text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model.

[0039] In the embodiments of this specification, a bounding box can refer to a rectangular region used to accurately select English words in a target image. The bounding box is the core geometric representation for quantifying the spatial position and range of the target word. The core function of the bounding box is to define the specific position and size of the word in the image coordinate system through specific parameters, providing a clear target area for subsequent cropping of the target image and avoiding interference from background information.

[0040] In the embodiments of this specification, the text detection model is an algorithm model that relies on technologies such as deep learning to accurately locate text regions from various images such as natural scenes and printed documents. It is one of the core components of optical character recognition (OCR) technology. Unlike ordinary object detection, the text detection model needs to adapt to the characteristics of text lines of varying lengths, angles, and easy blending with the background.

[0041] In practical applications, text detection models can include the following three core functions: 1. Locating text regions: Accurately outputting the position and extent of text in an image, often presented in the form of bounding box coordinates, clearly distinguishing text from background areas. 2. Adapting to diverse text shapes: Handling text of different angles and shapes, such as horizontal, tilted, and curved text, while also being compatible with different font sizes, fonts, and text detection needs under complex image quality conditions such as blurriness and occlusion. 3. Providing support for recognition: Filtering out effective text regions in the image, reducing background interference in subsequent text recognition processes, and improving the efficiency and accuracy of the overall OCR process.

[0042] In practical applications, text detection models can include, but are not limited to: text detection models based on connected preselection boxes, efficient and accurate scene text detection pipeline models, differentiable binarization network models, and unified network models for fast targeted text detection and recognition.

[0043] In practical applications, text detection models can be used to determine the location and size of the bounding boxes for each English word in a target image. After inputting the target image into the text detection model, it can output the location and size of the bounding boxes for each word in the English text content. The location information of the bounding boxes may include, but is not limited to: the x-coordinates and y-coordinates of the center point, the x-coordinates and y-coordinates of the top-left and bottom-left corners, the x-coordinates and y-coordinates of the top-right and bottom-right corners, etc. The size information of the bounding boxes may include, but is not limited to: the length and width of the bounding boxes.

[0044] Step 206: Based on the position and size information of the bounding boxes corresponding to each word, the target image is cropped to obtain multiple cropped sub-images; each sub-image contains one word.

[0045] In the embodiments of this specification, after determining the position and size information of the bounding boxes corresponding to each word using a text detection model, the target image can be cropped based on the range of the bounding boxes. Specifically, the target region of the bounding box of each word can be accurately extracted in the target image coordinate system to ensure that the sub-image contains only a single word and has no redundant background.

[0046] In practical applications, if a user's handwritten English words are slanted or distorted, the word's bounding box may not be a regular rectangular bounding box. In such cases, the irregular text area can be perspective-transformed into a regular horizontal rectangle using the coordinates of the four vertices of the bounding box quadrilateral. Then, the rectangular area is cropped to ensure the word is not stretched or deformed, thus adapting to severely curved or slanted English words. Specifically, the dimensions of the target horizontal rectangle can be calculated using the four vertices of the bounding box quadrilateral (e.g., width = maximum horizontal span, height = maximum vertical span), and then the quadrilateral area can be mapped to a horizontal rectangle using a perspective transformation matrix.

[0047] In practical applications, the target image can be cropped by calling the cropping interface of image processing libraries (such as OpenCV and PIL), and the cropped sub-image can be scaled to the input size required by the recognition model. The aspect ratio is maintained during scaling (stretching can be avoided by filling black or white borders).

[0048] Step 208: Input the multiple sub-images into the recognition model to obtain the word recognition results for each sub-image output by the recognition model; the recognition model includes the TROCR model.

[0049] In the embodiments described in this specification, the recognition model can identify handwritten English words in each input sub-image, recognizing the word text in the image and obtaining the word recognition result for each sub-image. In practical applications, the TROCR model can be used as the recognition model. The Transformer network in the TROCR model does not only focus on local features but also uses an attention mechanism to construct the entire network structure. This allows it to grasp the global information of the input word image, better understand the contextual relationships between the letters in the image, and thus effectively improve the recognition accuracy of handwritten words.

[0050] In practical applications, handwritten word datasets that match real-world usage scenarios can be used to specifically train the recognition model, thereby further improving the model's accuracy in recognizing handwritten English words.

[0051] Step 210: Based on the position and size information of the bounding boxes corresponding to each word, the word recognition results of each sub-image are arranged to obtain the recognition results of the English text in the target image.

[0052] In the embodiments of this specification, after the recognition model identifies the English text corresponding to each handwritten word, these identified words need to be further formatted before they can be combined into an English sentence, a paragraph, or an essay. Without formatting, the English text in the target image cannot be fully reproduced.

[0053] In practical applications, the row to which each word belongs can be determined sequentially based on the position and size information of the bounding box corresponding to the word. Furthermore, based on the position information of the bounding box corresponding to the word, the sorting position of each word in its row can be determined. Then, the word recognition results of each sub-image can be arranged to obtain the recognition results of English and Chinese text in the target image. Even if the shooting method is not standardized or the text layout is not neat, each line can be correctly divided, improving the accuracy of the recognition results.

[0054] Figure 2 The method described above involves obtaining a target image containing handwritten English text. This image is then input into a text detection model to obtain the position and size information of the bounding boxes for each word in the English text. Based on this information, the target image is cropped to obtain multiple sub-images, each containing one word. These sub-images are then input into a recognition model to obtain word recognition results for each sub-image. The recognition model includes the TROCR model. Finally, based on the position and size information of the bounding boxes for each word, the word recognition results for each sub-image are arranged to obtain the recognition result for the English text in the target image. In this application, a text detection model is used to determine the bounding boxes of each handwritten English word in the target image. Based on the bounding boxes, the target image is cropped to obtain multiple sub-images. After the sub-images are input into the TROCR model, the TROCR model can accurately recognize the handwritten English words in the sub-images, obtaining accurate recognition results. Subsequently, based on the position and size information of the bounding boxes corresponding to each word, the word recognition results of each sub-image can be arranged to finally determine the recognition result for the entire handwritten English text. By using the TROCR model instead of the CNN model to recognize words in the sub-images, the attention mechanism can be used to grasp the global information of the input word image, and the contextual relationships between the letters of the words in the image can be understood, which can effectively improve the recognition accuracy of handwritten words.

[0055] based on Figure 2 In addition to the method described in the embodiments of this specification, some specific implementation schemes of the method are also provided, which will be described below.

[0056] As an implementation method, optionally, in the embodiments of this specification, before inputting the target image into the text detection model to obtain the position and size information of the bounding boxes of each word in the English essay content output by the detection model, the method may further include: The target image is input into the image correction model to obtain the corrected target image output by the image correction model; the image correction model is used to perform trapezoidal correction on the target image to correct the tilt or distortion of the target image; Correspondingly, inputting the target image into the text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model may specifically include: The corrected target image is input into the text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model.

[0057] In practical applications, since the target image may be tilted or distorted due to the shooting angle, the target image can be corrected before recognizing the English text in the target image to ensure the accuracy of the recognition.

[0058] In the embodiments of this specification, the image correction model can be a trained model specifically designed for image correction. Specifically, the image correction model can be used to perform trapezoidal correction on images to correct tilt or distortion. Trapezoidal correction is a technique in computer vision and image processing for correcting geometric distortions in images. It involves analyzing trapezoidal distortions in a target image caused by shooting angle shifts, lens optical characteristics, or changes in the posture of the shooting device. Based on principles such as perspective transformation, projective geometry, or feature point matching, it calculates distortion correction parameters and maps and adjusts the image pixel coordinates, ultimately restoring the distorted image to a rectangular or regular geometric shape that conforms to human visual perception. This ensures that the shape and proportion of objects in the image are consistent with the actual scene, providing an accurate image basis for subsequent image analysis, measurement, or visualization.

[0059] In practical applications, after the target image is input into the image correction model, the model can perform trapezoidal correction to correct any tilt or distortion. The corrected image can then be output. This corrected image can then be input into a text detection model to determine the bounding box position and size of each word in the English text content within the corrected image.

[0060] In practical applications, not only can image correction models be used to correct target images, but other methods can also be used to correct target images. For example, pre-written correction programs can also be used to correct target images, without any specific limitations.

[0061] In practical applications, not only can trapezoidal correction be used to correct target images, but other correction techniques can also be used, such as perspective transformation and affine transformation. No specific limitations are made for these techniques.

[0062] In the embodiments of this specification, by performing correction processing on the target image, image distortion caused by shooting angle shift, lens optical characteristics, or changes in the posture of the shooting device can be effectively corrected. This is beneficial to improving the accuracy of determining the position and size information of the bounding boxes corresponding to each word in the target image, as well as the recognition accuracy of each word.

[0063] As one implementation method, optionally, in the embodiments of this specification, before inputting the plurality of sub-images into the recognition model to obtain the word recognition results output by the recognition model for each of the sub-images, the method may further include: Each sub-image is preprocessed to obtain preprocessed sub-images; the preprocessing includes noise reduction and contrast enhancement. Correspondingly, the step of inputting the plurality of sub-images into the recognition model to obtain the word recognition results for each sub-image output by the recognition model may specifically include: The preprocessed sub-images are input into the recognition model to obtain the word recognition results for each sub-image output by the recognition model.

[0064] In practical applications, cropping a target image can result in multiple sub-images. These sub-images may suffer from noise interference, low contrast, and blurred details due to factors such as shooting environment (e.g., insufficient light, reflection, shadows), equipment performance (e.g., camera sensor noise, lens blur), and shooting operation (e.g., hand tremors, inaccurate focus). Directly inputting these sub-images into a recognition model can lead to difficulties in extracting effective features, resulting in decreased recognition accuracy. Therefore, before processing the sub-images using the recognition model, preprocessing can be performed on each sub-image. Through denoising and contrast enhancement in preprocessing, interference information in the sub-images can be eliminated or suppressed, and the difference between the target (word text) and the background can be strengthened. This makes the text features of the sub-images clearer and easier for the model to capture, providing high-quality input data for the recognition model and ultimately ensuring the accuracy and stability of the recognition results.

[0065] In the embodiments of this specification, denoising processing can refer to identifying, suppressing, or removing interference signals (i.e., noise, such as salt-and-pepper noise, Gaussian noise, random noise caused by ambient light, etc.) in a sub-image that are unrelated to the target information (word text) using a specific algorithm. This reduces the impact of noise on subsequent recognition tasks while preserving the detailed features of the text. Common denoising methods include mean filtering, median filtering, Gaussian filtering, and bilateral filtering. For example, median filtering can effectively remove salt-and-pepper noise, while bilateral filtering can maintain the clarity of text edges while denoising, ensuring that the processed sub-image is clean without losing key recognition information.

[0066] In the embodiments of this specification, contrast enhancement processing can refer to a preprocessing technique that adjusts the pixel grayscale value distribution of a sub-image through algorithms to increase the grayscale difference between the target (word text) and the background, making the text area more prominent and the details clearer. When the original sub-image has similar grayscale values ​​to the background due to insufficient light or improper exposure (such as light gray text superimposed on a gray-white background), the model has difficulty distinguishing between the target and the background. Contrast enhancement can increase the brightness of text pixels or decrease the brightness of background pixels, strengthening the boundary distinction between the two (for example, darkening the dark text and brightening the bright background). Common methods can include histogram equalization, adaptive histogram equalization, gamma correction, etc. Among them, adaptive histogram equalization can make precise adjustments to local areas of the sub-image, avoiding the loss of details caused by overall adjustment, and is suitable for processing sub-images with uneven lighting.

[0067] As one implementation, optionally, in this embodiment of the specification, the position information of the bounding box may include the x-axis coordinate information and the y-axis coordinate information of the center point of the bounding box; the size information of the bounding box may include the length information and the width information of the bounding box; the origin of the coordinates corresponding to the position information of the bounding box is the upper left corner of the target image.

[0068] In the embodiments of this specification, during the process of determining the position and size information of the bounding boxes of each word in the English text content of the target image, the text detection model can determine the top left corner of the target image as the origin and establish an xy coordinate system, with the horizontal axis as the x-axis and the vertical axis as the y-axis. After determining the coordinate system, the text detection model can determine the bounding boxes of each word in the English text content of the target image, and then determine the position of the center point of each bounding box in the coordinate system, thereby determining the (x, y) coordinate information of the center point of each bounding box. In addition, the text detection model can also determine the length and width information corresponding to each bounding box, obtain the size information of each bounding box, and output it. For example, the text detection model outputs the position information of the bounding box A of a certain word as (1.2, 3.1), and the size information of bounding box A as: length: 0.4, width: 0.2.

[0069] As an alternative implementation, in the embodiments of this specification, the text detection model may include at least one of the following: a text detection model based on a connected preselection box network, an efficient and accurate scene text detection pipeline model, a differentiable binarization network model, and a unified network model for fast targeted text detection and recognition.

[0070] In the embodiments of this specification, the concepts of the text detection model based on the pre-selection box network, the efficient and accurate scene text detection pipeline model, the differentiable binarization network model, and the unified network model for fast targeted text detection and recognition can be found in the glossary section of this specification, and will not be repeated here.

[0071] As one implementation method, optionally, in the embodiments of this specification, the method may further include: Obtain a training sample set; a training sample in the training sample set includes a training image and a corresponding text label; the training image contains handwritten English words; the text label corresponding to the training image contains the text corresponding to the handwritten English words in the training image. The TROCR model is trained using the training data to obtain the recognition model.

[0072] In the embodiments of this specification, the training sample set may contain a preset number of training samples. The preset number can be set and adjusted according to the model training requirements. This application does not specifically limit the number of training samples in the training sample set.

[0073] In the embodiments of this specification, each training sample may include a training image and a corresponding text label. Each training image may contain a handwritten English word, and the text label corresponding to the training image may contain the text corresponding to the handwritten English word in the training image. For example, if training image A contains the handwritten word "apple", then the text label corresponding to training image A will contain the text of the word "apple".

[0074] In practical applications, each training image can also contain multiple handwritten English words. If a training image contains only one handwritten English word, then the training image can be an image obtained by taking a picture of a single handwritten word, or it can be a sub-image containing only one word obtained by cropping a picture containing multiple handwritten words. If a training image contains multiple handwritten English words, then the training image can be an image obtained by taking a picture of multiple handwritten words, or it can be a sub-image containing multiple words obtained by cropping a picture containing multiple handwritten words. There are no specific limitations on this.

[0075] In practical applications, the handwritten English words in the training images can be images that include real-world scenarios such as illegible handwriting, blurry images, and cluttered backgrounds. By training the TROCR model using training data containing these training images and their corresponding text labels, the TROCR model can be trained in a supervised manner. This improves the TROCR model's ability to recognize handwritten words in real-world scenarios such as illegible handwriting, blurry images, and cluttered backgrounds, which is beneficial for improving the recognition accuracy of the recognition model for handwritten English words in sub-images.

[0076] As one implementation method, optionally, in the embodiments of this specification, the step of arranging the word recognition results of each sub-image based on the position and size information of the bounding boxes corresponding to each word to obtain the recognition result of the English text in the target image may specifically include: Based on the position and size information of the bounding boxes corresponding to each word, the x-axis and y-axis coordinates of the top left corner of each bounding box are determined; Based on the x-axis and y-axis coordinates of the top-left corner of each of the bounding boxes and the intersection-union ratio between the bounding boxes, multiple sets of bounding boxes in the same row are determined; each bounding box in the set of bounding boxes in the same row belongs to the bounding boxes of words in the same row. Based on the y-axis coordinates of the top-left corner of each of the aforementioned set of bounding boxes, the layout order of each of the aforementioned set of bounding boxes on the y-axis is determined. Based on the x-axis coordinates of the top-left corner of each bounding box in each set of peer bounding boxes, the word order on the x-axis within each bounding box in each set of peer bounding boxes is determined. Based on the y-axis arrangement order of each set of peer bounding boxes and the x-axis arrangement order of words within each bounding box of each set of peer bounding boxes, the word recognition results of each sub-image are arranged to obtain the recognition result of the English text in the target image.

[0077] In practical applications, if the bounding box position information output by the text detection model includes the x-coordinates and y-coordinates of the top-left corner of each bounding box, then multiple sets of bounding boxes in the same row can be determined directly based on these coordinates and the intersection-over-union (IoU) ratio between the bounding boxes. If the bounding box position information output by the text detection model does not include the x-coordinates and y-coordinates of the top-left corner of each bounding box, then the x-coordinates and y-coordinates of the top-left corner of each bounding box can be determined first based on the position and size information of the bounding boxes corresponding to each word. Then, multiple sets of bounding boxes in the same row can be determined based on these coordinates and the IoU ratio between the bounding boxes. For example: In a coordinate system with the top left corner of the target image as the origin, the right direction as the positive x-axis, and the bottom direction as the positive y-axis, the center point coordinates of the bounding box A output by the text detection model are (3.0, 3.0). The length of bounding box A is 2 and the width is 1. Then, the x-axis coordinate of the top left corner of bounding box A is 3 - 2 / 2 = 2; the y-axis coordinate of the top left corner of bounding box A is 3 - 1 / 2 = 2.5.

[0078] In this embodiment, each bounding box in each set of bounding boxes in the same row is a bounding box belonging to the same word in the same row, determined based on the x-axis and y-axis coordinates of the top-left corner of each bounding box and the intersection-union ratio (IUGR) between the bounding boxes. Specifically, taking a coordinate system with the top-left corner of the target image as the origin, the right direction as the positive x-axis, and the down direction as the positive y-axis as the positive y-axis as the positive x-axis, the determination of multiple sets of bounding boxes in the same row based on the x-axis and y-axis coordinates of the top-left corner of each bounding box and the IUGR between the bounding boxes may include the following steps S1 to S7: Step S1: Based on the x-axis coordinates of the upper left corner of each bounding box, sort the bounding boxes in ascending order of their x-axis coordinates to obtain the sorting result.

[0079] Step S2: Determine the first bounding box in the sorting results as the target bounding box.

[0080] Step S3: Add the bounding boxes that meet the preset conditions in the sorting results that are located after the target bounding box to the candidate box list of the target bounding box; the preset conditions include that the intersection-union ratio of the bounding box and the target bounding box is greater than a first threshold, or that the intersection-union ratio of the bounding box and the target bounding box in the y-axis direction is greater than a second threshold and the difference in distance between the top left corners of the two boxes on the x-axis is less than a third threshold.

[0081] Step S4: Determine the bounding box with the smallest Euclidean distance from the target bounding box in the candidate box list as the bounding box that is in the same row as the target bounding box and adjacent to it.

[0082] Step S5: Add the target bounding box and the bounding boxes that are in the same row and adjacent to the target bounding box to the first set of row bounding boxes.

[0083] Step S6: Re-determine the bounding boxes that are in the same row and adjacent to the target bounding box as the target bounding box, and execute the above steps S1 to S5 to obtain new bounding boxes in the same row and add them to the first set of bounding boxes in the same row, until no new bounding boxes in the same row and adjacent to the target bounding box can be determined, and obtain each bounding box in the first set of bounding boxes in the same row.

[0084] Step S7: Delete each bounding box in the first row bounding box set from the sorting result, and perform steps S1 to S6 above for the remaining bounding boxes in the sorting result until each candidate box in the sorting result is added to the corresponding row bounding box set, thus obtaining multiple row bounding box sets.

[0085] In practical applications, considering that in a coordinate system with the top left corner of the target image as the origin, the right direction as the positive x-axis, and the bottom direction as the positive y-axis, the x-axis coordinates of the bounding boxes corresponding to the first English word in each line of the English text in the target image are all relatively small, therefore, by step S1: sorting the bounding boxes according to their x-axis coordinates from smallest to largest based on the top left corner of each bounding box, and then by step S2: determining the bounding box that ranks first in the sorting result as the target bounding box, the obtained target bounding box is the bounding box corresponding to the first English word in a certain line.

[0086] In this embodiment of the specification, in step S3 above, the preset condition can be a preset condition used to filter out one or more bounding boxes that are in the same row as the target bounding box and are adjacent to the target bounding box. Specifically, the preset condition may include: the intersection-union ratio (IU) between the bounding box and the target bounding box is greater than a first threshold, or the IU of the bounding box and the target bounding box in the y-axis direction is greater than a second threshold and the difference in distance between the top left corners of the two boxes on the x-axis is less than a third threshold. The IU between two bounding boxes is used to measure the degree of overlap between the two bounding boxes; the larger the value (maximum 1), the higher the degree of overlap between the two bounding boxes. The first threshold can be set and adjusted according to actual needs and is not specifically limited thereto. When the IU of a bounding box with the target bounding box is greater than the first threshold, the bounding box can be added to the candidate box list of the target bounding box. The IU between two bounding boxes in the y-axis direction is used to measure the degree of overlap between the two bounding boxes in the y-axis direction; the larger the value (maximum 1), the greater the probability that the two bounding boxes are in the same row. The second threshold can be set and adjusted according to actual needs, and the third threshold can also be set and adjusted according to actual needs, without specific limitations.

[0087] In practical applications, if the intersection-over-union (IoU) ratio between a bounding box and a target bounding box along the y-axis is greater than a second threshold, then the bounding box is determined to be on the same row as the target bounding box. If the IoU ratio between a bounding box and a target bounding box along the y-axis is greater than the second threshold and the difference in distance between the top-left corners of the two boxes along the x-axis is less than a third threshold, then the bounding box is determined to be on the same row as the target bounding box and is adjacent to the target bounding box. Therefore, when the IoU ratio between a bounding box and a target bounding box along the y-axis is greater than the second threshold and the difference in distance between the top-left corners of the two boxes along the x-axis is less than the third threshold, the bounding box can be added to the candidate box list of the target bounding box.

[0088] In this embodiment of the specification, in step S3, the bounding boxes that meet the preset conditions in each bounding box after the target bounding box in the sorting result are added to the candidate box list of the target bounding box. In fact, this involves traversing each bounding box after the target bounding box, selecting one or more bounding boxes that are in the same row as the target bounding box and adjacent to the target bounding box, and adding them to the candidate box list of the target bounding box.

[0089] In practical applications, the candidate bounding box list of the target bounding box may contain multiple bounding boxes that are in the same row and adjacent to the target bounding box. In this case, step S4 can be used to determine a bounding box that is in the same row and adjacent to the target bounding box from the multiple candidate bounding boxes in this candidate box list. This bounding box is located to the right of the target bounding box. Further, step S5 can be used to add the determined bounding box that is in the same row and adjacent to the right of the target bounding box, along with the target bounding box, to the first set of bounding boxes in the same row. Each bounding box in this first set of bounding boxes in the same row is a bounding box that is in the same row as the target bounding box.

[0090] In practical applications, after identifying a bounding box that is in the same row and adjacent to the target bounding box, this bounding box can be re-identified as the target bounding box. Steps S1 to S5 are then executed to obtain new bounding boxes in the same row and adjacent to it, which are added to the first set of bounding boxes in the same row. This process continues until no new bounding boxes in the same row and adjacent to it can be identified, resulting in the set of bounding boxes in the first set of bounding boxes in the same row. For example, initially, the target bounding box is bounding box A. After step S4, bounding box B is identified as being in the same row and adjacent to bounding box A. After adding bounding boxes A and B to the first set of bounding boxes in the same row, bounding box B can be identified as the target bounding box. Steps S1 to S5 are then executed again to identify bounding box C as being in the same row and adjacent to bounding box B. After adding bounding box C to the first set of bounding boxes in the same row, bounding box C can be identified as the target bounding box. Steps S1 to S5 are then executed again to identify bounding boxes in the same row and adjacent to bounding box C as edge boxes. Boundary box D: After adding boundary box D to the first row of boundary boxes set, boundary box D is determined as the target boundary box. Steps S1 to S5 are executed again. If none of the boundary boxes meet the preset conditions during the traversal in step S3, then the boundary box that is in the same row and adjacent to boundary box D cannot be determined. In this case, it can be considered that the selection of each boundary box in the first row of boundary boxes is completed. The resulting first row of boundary boxes set includes boundary box A, boundary box B, boundary box C and boundary box D (each boundary box corresponds to a word, so this row can be considered to include four words).

[0091] In the embodiments of this specification, steps S1 to S6 are the process of determining a set of bounding boxes in the same row corresponding to a certain row. If the English text in the target image includes multiple rows, step S7 needs to be executed to determine the set of bounding boxes in the same row corresponding to each row.

[0092] As one implementation method, optionally, in the embodiments of this specification, determining the layout order of each set of bounding boxes on the y-axis based on the y-axis coordinates of the upper left corner of each bounding box in each set of bounding boxes may specifically include: For any set of bounding boxes in the same row, determine the highest and lowest y-axis coordinate values ​​of each bounding box contained in the set of bounding boxes in the same row. The average of the highest and lowest y-axis coordinates is determined as the y-axis reference coordinate of the set of bounding boxes in the same row; Based on the y-axis reference coordinates of each of the aforementioned peer bounding box sets, the layout order of each of the peer bounding box sets on the y-axis is determined.

[0093] In the embodiments of this specification, after determining the sets of bounding boxes in each row, it is also necessary to determine which set of bounding boxes corresponds to the first row, which set corresponds to the second row, and so on. Therefore, during the layout process, it is necessary to determine the layout order of each set of bounding boxes in each row on the y-axis based on the y-axis coordinates of the top-left corner of each bounding box in each set of bounding boxes.

[0094] In practical applications, for any set of bounding boxes in the same row, the average of the highest and lowest y-axis coordinates of the top-left corner points of each bounding box in the set is determined as the y-axis reference coordinate of the set. This can reduce the impact of the highest or lowest y-axis coordinates on the y-axis layout. Layout based on the y-axis reference coordinates is beneficial to improving the layout accuracy of each set of bounding boxes in the same row on the y-axis.

[0095] In practical applications, after determining the y-axis reference coordinates of each set of bounding boxes in the same row, the layout order of the bounding boxes within each row can be determined from left to right based on the x-axis coordinates of the top-left corner of each bounding box in the set of bounding boxes in the same row. This ultimately determines which row each word within a bounding box belongs to and its layout order within that row. By performing the above layout steps based on the position and size information of the bounding boxes corresponding to each word to layout the word recognition results of each sub-image, the layout order of each word in the user's handwritten English text in the target image can be accurately restored, which helps improve the recognition accuracy of English text in the target image.

[0096] As one implementation method, optionally, in this embodiment of the specification, after arranging the word recognition results of each sub-image based on the position and size information of the bounding boxes corresponding to each word to obtain the recognition result of the English text in the target image, it may further include: The recognition result of the English text in the target image is input into the layout correction model to obtain the layout correction result of the layout correction model for the recognition result; the layout correction model is used to perform semantic recognition on the recognition result after the initial layout, and correct the layout order of the recognition result based on the semantic recognition result.

[0097] In practical applications, the initial layout of word recognition results for each sub-image based on the position and size information of the bounding boxes corresponding to each word may contain layout errors. Therefore, the initial layout results can be input into a layout correction model. This model can perform semantic recognition on the initial layout results. If semantic inconsistencies exist, it can correct the word layout based on semantics, making the word layout more fluent and reasonable. This helps improve the layout accuracy of English text recognition results for target images.

[0098] As one implementation method, optionally, in this embodiment of the specification, after arranging the word recognition results of each sub-image based on the position and size information of the bounding boxes corresponding to each word to obtain the recognition result of the English text in the target image, it may further include: The recognition result of the English text in the target image is input into the scoring model to obtain the score of the English text and / or the modification suggestions for the English text; the scoring model is used to score the user's handwritten English text or to make modification suggestions for the English text.

[0099] In practical applications, the scoring model can be a large language model. The recognition results of the English text in the target image are input into the scoring model, and preset prompts can also be input. These prompts guide the large language model to score the input English text or provide suggestions for improvement. For example, if the English text in the target image is a handwritten English essay, this manual can be used... Figure 2 After the method identifies the content of this English essay, the content can be input into the large language model along with the prompt "Please rate this English essay out of 100 points." The large language model can then output a score for the essay. Alternatively, the prompt "Please provide suggestions for revising this English essay" can be input, and the large language model can output suggested revisions.

[0100] In practical applications, the scoring model can also be a neural network model trained specifically on a sample dataset. A training sample in the dataset can contain information such as essay requirements, the user-written English essay text, essay scores, and suggestions for improvement. Using these training samples to conduct supervised training on the neural network model allows it to be trained to score English essays or provide suggestions for improvement, thus obtaining the scoring model. This scoring model can then be used to apply the knowledge to this manual. Figure 2 The method identifies English text and scores it or provides suggestions for improvement.

[0101] In practical applications, after determining the recognition results of the English and Chinese text in the target image, the recognition results can also be scored or corrected manually. The application server can also provide essay scores and / or revision suggestions to the application client. After receiving the essay scores and / or revision suggestions, the application client can display the essay scores and / or revision suggestions to the user.

[0102] Corresponding to the above method embodiments, this specification also provides an embodiment of an apparatus for recognizing image content. Figure 3 This is a schematic diagram of the structure of an image content recognition device provided in one embodiment of this specification. Figure 3 As shown, the device may include: The acquisition module 302 is used to acquire a target image including handwritten English text by the user; The first input module 304 is used to input the target image into the text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model; The cropping module 306 is used to crop the target image based on the position and size information of the bounding boxes corresponding to each word, to obtain multiple cropped sub-images; each sub-image contains one word; The second input module 308 is used to input the plurality of sub-images into the recognition model to obtain the word recognition results for each of the sub-images output by the recognition model; the recognition model includes the TROCR model; The layout module 310 is used to layout the word recognition results of each sub-image based on the position and size information of the bounding boxes corresponding to each word, so as to obtain the recognition result of the English text in the target image.

[0103] Optionally, the device may further include: An image correction module is used to input the target image into an image correction model to obtain a corrected target image output by the image correction model; the image correction model is used to perform trapezoidal correction on the target image to correct the tilt or distortion of the target image.

[0104] Correspondingly, the first input module 304 may specifically include: The input submodule is used to input the corrected target image into the text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model.

[0105] Optionally, the device may further include: The sub-image preprocessing module is used to preprocess each sub-image to obtain preprocessed sub-images; the preprocessing includes noise reduction and contrast enhancement.

[0106] Correspondingly, the second input module 308 may specifically include: The input submodule is used to input the preprocessed sub-images into the recognition model to obtain the word recognition results for each sub-image output by the recognition model.

[0107] Optionally, the position information of the bounding box may include the x-axis coordinate information and the y-axis coordinate information of the center point of the bounding box; the size information of the bounding box may include the length information and the width information of the bounding box; the origin of the coordinates corresponding to the position information of the bounding box may be the upper left corner of the target image.

[0108] Optionally, the text detection model may include at least one of the following: a text detection model based on a connected preselection box network, an efficient and accurate scene text detection pipeline model, a differentiable binarization network model, and a unified network model for fast targeted text detection and recognition.

[0109] Optionally, the device may further include: A training sample set acquisition module is used to acquire a training sample set; a training sample in the training sample set includes a training image and a corresponding text label; the training image contains handwritten English words; the text label corresponding to the training image contains the text corresponding to the handwritten English words in the training image. The model training module is used to train the TROCR model using the training data to obtain the recognition model.

[0110] Optionally, the typesetting module 310 may specifically include: The first determining submodule is used to determine the x-axis coordinates and y-axis coordinates of the upper left corner of each bounding box based on the position and size information of the bounding box corresponding to each word. The second determining submodule is used to determine multiple sets of parallel bounding boxes based on the x-axis and y-axis coordinates of the top-left corner of each of the bounding boxes and the intersection-union ratio between the bounding boxes; each bounding box in the set of parallel bounding boxes belongs to the bounding boxes of words in the same line; The third determining submodule is used to determine the layout order of each set of bounding boxes on the y-axis based on the y-axis coordinates of the upper left corner of each bounding box in each set of bounding boxes in the same row. The fourth determination submodule is used to determine the word order on the x-axis within each bounding box in each set of the same row bounding boxes based on the x-axis coordinate of the top left corner of each bounding box in each set of the same row bounding boxes. The layout submodule is used to layout the word recognition results of each sub-image based on the layout order of each set of peer bounding boxes on the y-axis and the layout order of the words in each bounding box of each set of peer bounding boxes on the x-axis, so as to obtain the recognition result of the English text in the target image.

[0111] Optionally, the second determining submodule may specifically include: The sorting unit is used to sort the bounding boxes in ascending order of their x-axis coordinates based on the x-axis coordinates of the upper left corner of each bounding box, and obtain the sorting result. The first determining unit is used to determine the first bounding box in the sorting result as the target bounding box; The first adding unit is used to add the bounding boxes that meet the preset conditions in the sorting results that are located after the target bounding box to the candidate box list of the target bounding box; the preset conditions include the intersection-union ratio of the bounding box and the target bounding box being greater than a first threshold, or the intersection-union ratio of the bounding box and the target bounding box in the y-axis direction being greater than a second threshold and the difference in distance between the top left corners of the two boxes on the x-axis being less than a third threshold. The second determining unit is used to determine the bounding box with the smallest Euclidean distance from the target bounding box in the candidate box list as the bounding box that is in the same row and adjacent to the target bounding box. The second adding unit is used to add the target bounding box and the bounding boxes that are in the same row and adjacent to the target bounding box to the first set of row bounding boxes; The third determining unit is used to re-determine the bounding boxes that are in the same row and adjacent to the target bounding box as the target bounding box, perform the above steps, and add the new bounding boxes in the same row and adjacent to the first set of bounding boxes in the same row until no new bounding boxes in the same row and adjacent to the target bounding box can be determined, and obtain each bounding box in the first set of bounding boxes in the same row. The fourth determining unit is used to delete each bounding box in the first row bounding box set from the sorting result, and perform the above steps on the remaining bounding boxes in the sorting result until each candidate box in the sorting result is added to the corresponding row bounding box set, thus obtaining multiple row bounding box sets.

[0112] Optionally, the third determining submodule may specifically include: The first determining unit is used to determine the highest and lowest y-axis coordinate values ​​of each bounding box contained in any set of bounding boxes in the same row. The second determining unit is used to determine the average of the highest value of the y-axis coordinate and the lowest value of the y-axis coordinate as the y-axis reference coordinate of the set of bounding boxes in the same row; The third determining unit is used to determine the layout order of each set of bounding boxes on the y-axis based on the y-axis reference coordinates of each set of bounding boxes in the same row.

[0113] Optionally, the device may further include: The scoring module is used to input the recognition result of the English text in the target image into the scoring model to obtain the score of the English text and / or the modification suggestions for the English text; the scoring model is used to score the user's handwritten English text or to make modification suggestions for the English text.

[0114] The above is an illustrative scheme of an image content recognition device according to this embodiment. It should be noted that the technical solution of this image content recognition device and the technical solution of the image content recognition method described above belong to the same concept. For details not described in detail in the technical solution of the image content recognition device, please refer to the description of the technical solution of the image content recognition method described above.

[0115] Figure 4 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0116] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0117] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0118] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 400 can also be a mobile or stationary server.

[0119] The processor 420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the method for recognizing image content described above.

[0120] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described method for recognizing image content belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described method for recognizing image content.

[0121] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method for recognizing image content described above.

[0122] The above is an illustrative embodiment of a computer-readable storage medium. It should be noted that the technical solution of this storage medium and the technical solution of the above-described method for recognizing image content belong to the same concept. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the above-described method for recognizing image content.

[0123] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0124] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0125] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0126] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0127] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for recognizing image content, characterized in that, The method includes: Acquire a target image that includes handwritten English text by the user; The target image is input into the text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model. Based on the position and size information of the bounding boxes corresponding to each word, the target image is cropped to obtain multiple cropped sub-images; each sub-image contains one word. The multiple sub-images are input into the recognition model to obtain the word recognition results for each sub-image output by the recognition model; the recognition model includes the TROCR model; Based on the position and size information of the bounding boxes corresponding to each word, the word recognition results of each sub-image are arranged to obtain the recognition result of the English text in the target image.

2. The method according to claim 1, characterized in that, Before inputting the target image into the text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model, the method further includes: The target image is input into the image correction model to obtain the corrected target image output by the image correction model; the image correction model is used to perform trapezoidal correction on the target image to correct the tilt or distortion of the target image; The step of inputting the target image into a text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model specifically includes: The corrected target image is input into the text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model.

3. The method according to claim 1, characterized in that, Before inputting the plurality of sub-images into the recognition model to obtain the word recognition results for each of the sub-images output by the recognition model, the method further includes: Each sub-image is preprocessed to obtain preprocessed sub-images; the preprocessing includes noise reduction and contrast enhancement. The step of inputting the multiple sub-images into the recognition model to obtain the word recognition results for each sub-image output by the recognition model specifically includes: The preprocessed sub-images are input into the recognition model to obtain the word recognition results for each sub-image output by the recognition model.

4. The method according to claim 1, characterized in that, The position information of the bounding box includes the x-axis coordinate information and the y-axis coordinate information of the center point of the bounding box; the size information of the bounding box includes the length information and the width information of the bounding box; the origin of the coordinate system corresponding to the position information of the bounding box is the upper left corner of the target image.

5. The method according to claim 1, characterized in that, The text detection model includes at least one of the following: a text detection model based on a connected preselection box network, an efficient and accurate scene text detection pipeline model, a differentiable binarization network model, and a unified network model for fast targeted text detection and recognition.

6. The method according to claim 1, characterized in that, The method further includes: Obtain a training sample set; a training sample in the training sample set includes a training image and a corresponding text label; the training image contains handwritten English words; the text label corresponding to the training image contains the text corresponding to the handwritten English words in the training image. The TROCR model is trained using the training data to obtain the recognition model.

7. The method according to claim 1, characterized in that, The step of arranging the word recognition results of each sub-image based on the position and size information of the bounding boxes corresponding to each word to obtain the recognition result of the English text in the target image specifically includes: Based on the position and size information of the bounding boxes corresponding to each word, the x-axis and y-axis coordinates of the top left corner of each bounding box are determined; Based on the x-axis and y-axis coordinates of the top-left corner of each of the bounding boxes and the intersection-union ratio between the bounding boxes, multiple sets of bounding boxes in the same row are determined; each bounding box in the set of bounding boxes in the same row belongs to the bounding boxes of words in the same row. Based on the y-axis coordinates of the top-left corner of each of the aforementioned set of bounding boxes, the layout order of each of the aforementioned set of bounding boxes on the y-axis is determined. Based on the x-axis coordinates of the top-left corner of each bounding box in each set of peer bounding boxes, the word order on the x-axis within each bounding box in each set of peer bounding boxes is determined. Based on the y-axis arrangement order of each set of peer bounding boxes and the x-axis arrangement order of words within each bounding box of each set of peer bounding boxes, the word recognition results of each sub-image are arranged to obtain the recognition result of the English text in the target image.

8. The method according to claim 7, characterized in that, The step of determining the layout order of each set of bounding boxes on the y-axis based on the y-axis coordinates of the top-left corner of each bounding box in each set of bounding boxes in the same row specifically includes: For any set of bounding boxes in the same row, determine the highest and lowest y-axis coordinate values ​​of each bounding box contained in the set of bounding boxes in the same row. The average of the highest and lowest y-axis coordinates is determined as the y-axis reference coordinate of the set of bounding boxes in the same row; Based on the y-axis reference coordinates of each of the aforementioned peer bounding box sets, the layout order of each of the peer bounding box sets on the y-axis is determined.

9. The method according to claim 1, characterized in that, After arranging the word recognition results of each sub-image based on the position and size information of the bounding boxes corresponding to each word to obtain the recognition result of the English text in the target image, the method further includes: The recognition result of the English text in the target image is input into the scoring model to obtain the score of the English text and / or the modification suggestions for the English text; the scoring model is used to score the user's handwritten English text or to make modification suggestions for the English text.

10. An apparatus for recognizing image content, characterized in that, The device includes: The acquisition module is used to acquire target images, including handwritten English text by the user; The first input module is used to input the target image into the text detection model to obtain the position and size information of the bounding boxes of each word in the English text content output by the text detection model; The cropping module is used to crop the target image based on the position and size information of the bounding boxes corresponding to each word, to obtain multiple cropped sub-images; each sub-image contains one word; The second input module is used to input the multiple sub-images into the recognition model to obtain the word recognition results for each sub-image output by the recognition model; the recognition model includes the TROCR model. The layout module is used to layout the word recognition results of each sub-image based on the position and size information of the bounding boxes corresponding to each word, so as to obtain the recognition result of the English text in the target image.

11. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method for recognizing image content as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method for recognizing image content as described in any one of claims 1 to 9.

Images