Image processing method, model training method and character recognition method

Abstract

The embodiment of the invention provides an image processing method, a model training method and a character recognition method, and the image processing method comprises the steps: determining a target image and determining processing prompt information corresponding to the target image, the target image is an image determined from a target image processing task, and the processing prompt information corresponds to the target image. The target image processing task is any one of at least two image processing tasks; and inputting the target image and the processing prompt information into a target image processing model to obtain an image processing result of the target image, the target image processing model being obtained by performing model training on an initial image processing model by using initial training sub-data and target training sub-data, the initial image processing model is obtained by performing model training on a to-be-trained image processing model by using the initial training sub-data, and the target training sub-data and the initial training sub-data are determined from training data according to task types of the at least two image processing tasks.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of artificial intelligence technology, and in particular to an image processing method; one or more embodiments of this specification also relate to a model training method, a character recognition method, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the continuous development of artificial intelligence technology, it has also been introduced into the field of image processing technology. By using trained neural network models to process images, image processing efficiency has been improved.

[0003] However, current neural network models have relatively limited capabilities, often only able to perform one type of image processing task. That is to say, when processing images, the neural network model only handles images for a specific image processing task, and cannot flexibly process images for multiple image processing tasks. Therefore, how to perform multiple image processing tasks through a single neural network model has become an urgent technical problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of this specification provide an image processing method. One or more embodiments of this specification simultaneously relate to a model training method, a character recognition method, an image processing apparatus, a model training apparatus, a character recognition apparatus, two other character recognition methods, a computing device, a computer-readable storage medium, and a computer program product, to address the technical shortcomings of existing neural network models in flexibly processing images for various image processing tasks.

[0005] According to a first aspect of the embodiments of this specification, an image processing method is provided, comprising:

[0006] Determine a target image and determine the processing prompt information corresponding to the target image, wherein the target image is an image determined from a target image processing task, and the target image processing task is any one of at least two image processing tasks;

[0007] The target image and the processing prompt information are input into the target image processing model to obtain the image processing result of the target image. The target image processing model is obtained by training an initial image processing model using initial training sub-data and target training sub-data. The initial image processing model is obtained by training a training image processing model using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two image processing tasks.

[0008] According to a second aspect of the embodiments of this specification, an image processing apparatus is provided, comprising:

[0009] The image determination module is configured to determine a target image and determine the processing prompt information corresponding to the target image, wherein the target image is an image determined from a target image processing task, and the target image processing task is any one of at least two image processing tasks;

[0010] An image processing module is configured to input the target image and the processing prompt information into a target image processing model to obtain the image processing result of the target image. The target image processing model is obtained by training an initial image processing model using initial training sub-data and target training sub-data. The initial image processing model is obtained by training a training image processing model using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two image processing tasks.

[0011] According to a third aspect of the embodiments of this specification, a model training method is provided, comprising:

[0012] Determine the image processing model to be trained and the training data corresponding to at least two image processing tasks, wherein the training data includes sample images corresponding to at least two image processing tasks, as well as sample labels and sample processing prompts corresponding to the sample images;

[0013] Based on the task types of the at least two image processing tasks, determine the initial training sub-data and the target training sub-data from the training data;

[0014] The initial image processing model is trained using the initial training sub-data to obtain the initial image processing model.

[0015] The initial image processing model is trained using the initial training sub-data and the target training sub-data to obtain a target image processing model, wherein the target image processing model is used to perform the at least two image processing tasks.

[0016] According to a fourth aspect of the embodiments of this specification, a model training apparatus is provided, comprising:

[0017] The first training data determination module is configured to determine the image processing model to be trained and the training data corresponding to at least two image processing tasks, wherein the training data includes sample images corresponding to at least two image processing tasks, as well as sample labels and sample processing prompts corresponding to the sample images;

[0018] The second training data determination module is configured to determine initial training sub-data and target training sub-data from the training data based on the task types of the at least two image processing tasks.

[0019] The first model training module is configured to train the image processing model to be trained using the initial training sub-data to obtain the initial image processing model.

[0020] The second model training module is configured to train the initial image processing model using the initial training sub-data and the target training sub-data to obtain a target image processing model, wherein the target image processing model is used to perform the at least two image processing tasks.

[0021] According to a fifth aspect of the embodiments of this specification, a character recognition method is provided, comprising:

[0022] Determine the target text image and the corresponding processing prompt information for the target text image, wherein the target text image is an image determined from a target text recognition task, and the target text recognition task is any one of at least two text recognition tasks;

[0023] The target text image and the processing prompt information are input into the target text recognition model to obtain the text information contained in the target text image. The target text recognition model is obtained by training an initial text recognition model using initial training sub-data and target training sub-data. The initial text recognition model is obtained by training a text recognition model to be trained using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two text recognition tasks.

[0024] According to a sixth aspect of the embodiments of this specification, a character recognition device is provided, comprising:

[0025] The image determination module is configured to determine a target text image and determine the processing prompt information corresponding to the target text image, wherein the target text image is an image determined from a target text recognition task, and the target text recognition task is any one of at least two text recognition tasks;

[0026] An image processing module is configured to input the target text image and the processing prompt information into a target text recognition model to obtain the text information contained in the target text image. The target text recognition model is obtained by training an initial text recognition model using initial training sub-data and target training sub-data. The initial text recognition model is obtained by training a text recognition model to be trained using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two text recognition tasks.

[0027] According to a seventh aspect of the embodiments of this specification, a text recognition method is provided, applied to a cloud-based device, comprising:

[0028] The receiving end device sends a target text image and determines the processing prompt information corresponding to the target text image, wherein the target text image is an image determined from a target text recognition task, and the target text recognition task is any one of at least two text recognition tasks;

[0029] The target text image and the processing prompt information are input into the target text recognition model to obtain the text information contained in the target text image. The target text recognition model is obtained by training an initial text recognition model using initial training sub-data and target training sub-data. The initial text recognition model is obtained by training a text recognition model to be trained using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two text recognition tasks.

[0030] According to an eighth aspect of the embodiments of this specification, a character recognition method is provided, comprising:

[0031] The target text image is determined from the received target document, and the corresponding processing prompt information is determined. The target text image is an image determined from a target text recognition task, and the target text recognition task is any one of at least two text recognition tasks.

[0032] The target text image and the processing prompt information are input into the target text recognition model to obtain the text information contained in the target text image. The target text recognition model is obtained by training an initial text recognition model using initial training sub-data and target training sub-data. The initial text recognition model is obtained by training a text recognition model to be trained using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two text recognition tasks.

[0033] According to a ninth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0034] Memory and processor;

[0035] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of any of the above methods.

[0036] According to a tenth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0037] According to an eleventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0038] The image processing method provided in one or more embodiments of this specification offers a method for training an initial image processing model using initial training sub-data and target training sub-data to obtain a target image processing model. The target training sub-data and the initial training sub-data are determined from the training data based on the task types of at least two image processing tasks. Therefore, during image processing, a target image and corresponding processing prompts can be determined from any one of the at least two image processing tasks. The target image and processing prompts are then input into the target image processing model to obtain the image processing result. This enables the execution of any one of at least two image processing tasks using a single target image processing model, avoiding the limitation of neural network models and enabling flexible processing of images for various image processing tasks. Attached Figure Description

[0039] Figure 1 This is a schematic diagram illustrating the application of an image processing method provided in one embodiment of this specification;

[0040] Figure 2 This is a flowchart illustrating an image processing method provided in one embodiment of this specification;

[0041] Figure 3 This is a flowchart illustrating the processing procedure of an image processing method provided in one embodiment of this specification.

[0042] Figure 4 This is a flowchart illustrating a model training method provided in one embodiment of this specification;

[0043] Figure 5 This is a flowchart of a text recognition method provided in one embodiment of this specification;

[0044] Figure 6 This is a flowchart of another character recognition method provided in one embodiment of this specification;

[0045] Figure 7 This is a flowchart of another character recognition method provided in one embodiment of this specification;

[0046] Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0047] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0048] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0049] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0050] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0051] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundational model (Foundation Model 1). It is pre-trained on a large-scale unlabeled corpus to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and Multi-modal Pre-training Models (MLMs).

[0052] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0053] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0054] Text recognition: which can be understood as character recognition, refers to the technology of automatically recognizing text content in images or videos.

[0055] Transformer: A deep learning model used for natural language processing tasks such as machine translation and text generation.

[0056] SceneTextSpottingbenchmarks: A set of benchmark tests used to evaluate the performance of scene text detection algorithms.

[0057] Generalization ability: refers to the model's ability to perform on unseen data, i.e., the model's adaptability.

[0058] Large Language Models (MLLMs) are large language models that can not only process text data, but also process and understand multiple different types of data (or "modalities"), such as images, audio, and video.

[0059] STR (Scene Text Recognition): Scene text recognition refers to the technology of identifying text in natural scene images. These natural scenes can include street scenes, shop signs, billboards, etc.

[0060] HTR (Handwriting Text Recognition): Handwriting text recognition refers to the technology of detecting and recognizing text from images of handwritten text. This task can be applied to a variety of handwritten texts, such as historical documents, notes, letters, and forms. HTR systems need to handle various handwriting styles, different writing tools (such as pens and pencils), and variations and irregularities in handwriting.

[0061] MER (Mathematical Expression Recognition): Mathematical expression recognition refers to the technique of detecting and recognizing mathematical formulas and expressions in images.

[0062] STS (Scene Text Spotting): Scene text localization and recognition is a comprehensive task that combines text detection and text recognition, that is, locating text in natural scene images and recognizing its content simultaneously. Unlike text detection or text recognition alone, STS systems directly generate recognition results for each text region in the image, and typically employ an end-to-end training approach.

[0063] With the continuous development of artificial intelligence technology, it has also been introduced into the field of image processing technology. By using trained neural network models to process images, the efficiency of image processing has been improved. However, the current capabilities of neural network models are relatively limited, and they can often only perform one type of image processing task.

[0064] For example, the task of reading text from images, whether in natural scenes or documents, has significantly evolved to support a wide range of applications, from archiving historical documents to real-time translation services. While traditionally categorized under the field of Optical Character Recognition (OCR), this task transcends the conventional boundaries of OCR, encompassing a broader concept of text reading. Advances in this field have witnessed the emergence of specialized models for different subdomains of text reading, such as scene text recognition, handwriting recognition, and formula recognition. While these specialized models excel in their specific domains, they perform poorly when dealing with unpredictable text in natural scenes.

[0065] Traditional text reading systems typically consist of two distinct stages: text detection followed by text recognition. Overall performance is highly dependent on the accuracy of the initial text detection stage, and errors can have a cascading effect. Furthermore, the emergence of Multimodal Large Language Models (MLLMs) has contributed a new dimension to the field by providing models with broad text reading capabilities, although there are trade-offs in computational efficiency and specific accuracy; for example, this specification provides five approaches.

[0066] The first approach is an optical character recognition (OCR) system, which is limited to recognizing text in scanned images such as documents. Its disadvantages include limitations, accuracy dependence, and lack of multimodal processing.

[0067] The limitations of traditional OCR systems include: Significant performance degradation in natural scene images (such as street views and billboards), and difficulty handling complex backgrounds and various text styles. Accuracy dependence: The overall performance of an OCR system is highly dependent on the accuracy of initial text detection; detection errors can have a cascading effect, leading to recognition errors. Lack of multimodal processing: The system typically cannot handle modalities beyond images, such as audio or video data, limiting its application scenarios.

[0068] The second approach utilizes a system that employs SceneTextRecognition (STR) models, which focus on recognizing text in natural scene images and typically use convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

[0069] Their drawbacks lie in two aspects: task limitations and complex backgrounds. Task limitations: these models perform well in specific tasks (such as text recognition in natural scenes) but struggle to generalize to other types of text tasks, such as handwritten text recognition or formula recognition. Complex backgrounds: these models still face challenges in text recognition against complex backgrounds, especially when the text is partially occluded or deformed.

[0070] The third approach utilizes systems based on Handwritten Text Recognition (HTR) models, which are specifically designed for recognizing handwritten text; these typically employ Long Short-Term Memory (LSTM) networks or convolutional neural networks for processing.

[0071] Its drawbacks include: preprocessing requirements and style diversity. Style diversity: the diversity and irregularity of handwritten text make these models perform inconsistently when dealing with different writing styles. Preprocessing requirements: these models typically require complex preprocessing steps, such as text normalization and denoising, to ensure recognition accuracy, which increases implementation complexity.

[0072] The fourth approach is the Mathematical Expression Recognition (MER) model, which focuses on recognizing mathematical expressions from images and converting them into LaTeX format.

[0073] Their drawbacks include: spatial resolution difficulty and single-task processing limitations. Specifically, the two-dimensional structure of mathematical expressions complicates the analysis of spatial relationships between symbols, making these models unstable when handling complex formulas. Furthermore, these models can only process mathematical expressions and are difficult to generalize to other text recognition tasks.

[0074] The fifth approach is multimodal large language models (MLLMs), which can process data in multiple modalities such as text and images, providing extensive text reading capabilities.

[0075] Their disadvantages include low computational efficiency and limited accuracy. Specifically, these models are typically very large, requiring significant computational resources, resulting in slow deployment and inference speeds. Furthermore, despite their diversity and adaptability, their accuracy in specific domains is often lower than that of dedicated models.

[0076] In summary, the aforementioned technical solutions typically face the following challenges: First, they are highly task-specific and difficult to generalize to various text recognition tasks. Second, they rely on complex preprocessing and specific detection stages, making them susceptible to error propagation. Finally, they have limited processing capabilities for multimodal data and incur significant computational resource overhead.

[0077] Based on this, an image processing method is provided in this specification. One or more embodiments of this specification also relate to a model training method, a character recognition method, an image processing apparatus, a model training apparatus, a character recognition apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0078] See Figure 1 , Figure 1 This diagram illustrates an application illustration of an image processing method according to an embodiment of this specification, based on... Figure 1 It can be seen that users can send various types of images and corresponding prompts to the server 104 through the terminal 102. After receiving the images and prompts, the server 104 will input the images and prompts into the trained text recognition model for text recognition, thereby reading the text information in any form of image and outputting the text information contained in the image. After the server 104 determines the text information contained in the image using the text recognition model, it will send it to the terminal 102 to display to the user, thereby realizing the ability to accurately interpret and recognize the text in various input images using the text recognition model.

[0079] See Figure 2 , Figure 2 A flowchart of an image processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0080] Step 202: Determine the target image and the corresponding processing prompt information for the target image, wherein the target image is an image determined from a target image processing task, and the target image processing task is any one of at least two image processing tasks.

[0081] The target image can be understood as a text image containing text information; that is, the target image can be a target text image. For example, the target image can be a full view of a natural scene, a full view of a document, a handwritten text image, a cropped text image, or a cropped formula image. The target image can be an image from any image processing task. Alternatively, the target image can be understood as a text image containing a target object; for example, an image containing target objects such as vehicles, pedestrians, or animals. The target image can also be a video frame from a video.

[0082] The processing prompts can be understood as information used to control the target image processing model to process the target image, or to control the type and direction of the content generated by the target image processing model. For example, the processing prompts can be the task category of the target image, the writing type of the target image, the output granularity of the model, and / or the location information of the text to be recognized in the target image, etc., without specific limitations. The processing prompts can be understood as prompts.

[0083] The image processing task can be a variety of image processing tasks; the image processing task can be a full-image task, which is a task for processing the entire image (i.e., the complete image); for example, the full-image task can be a task for recognizing text from a full image of a natural scene or a full image of a document; or, a task for recognizing target objects (such as objects, organisms, etc.) from a full image of a natural scene.

[0084] The image processing task can be an image cropping task, which is used to process cropped images (i.e., partial images acquired from a complete image); for example, an image cropping task can be a task to identify text from cropped text images and cropped formula images; or, a task to identify target objects (such as objects, organisms, etc.) from cropped images, etc.

[0085] The image processing task can be a task that processes artificially created images. For example, the task can be a task that recognizes text or formulas from a handwritten text image, or a task that recognizes a target object from a hand-drawn image.

[0086] In one or more embodiments provided in this specification, the image processing method can achieve task unification. Specifically, in addressing the various challenges of reading text from images, this image processing method introduces a unified framework that encompasses various text interpretation scenarios that transcend the boundaries of traditional OCR. Based on the source and presentation method of the image, the text reading task is divided into five categories.

[0087] Specifically, this method can define the categories as five types: full-view natural scene, full-view document, handwritten text image, cropped text, and cropped formula.

[0088] Natural scenes and full-image documents often contain various heterogeneous text styles and layouts, thus requiring a more general recognition method. For these images, three different recognition scenarios are introduced, each including corresponding processing prompts:

[0089] RAT (RecognizeAllText): Recognizes all text or target objects in the entire image without requiring specific location cues.

[0090] PPR (Point Prompt Recognition): Identifies text or target objects within a specific area around a point cue, allowing for precise identification of target text or target objects.

[0091] BPR (BoxPromptRecognition): Identifies text or target objects within a user-defined box area, and can be used to extract text or target objects from a specific region of interest.

[0092] For cropped images, such as cropped text and cropped formulas, these tasks involve directly interpreting the text or mathematical expressions in the image without needing cues about the image's location. By identifying all text or formulas within the entire image, the recognition process is simplified.

[0093] Furthermore, for the above categories, this solution also introduces a processing prompt indicating the output granularity of the identification, to meet the level of detail required by different text reading applications; this output granularity is divided into two levels:

[0094] Word-level: At this level, the model focuses on identifying words as the atomic units of text.

[0095] Line-level: At this level, the model recognizes an entire line of text, which may contain multiple words or even sentences.

[0096] Regarding the granularity mentioned above, for natural scenes and full-image document tasks, granularity is a crucial parameter determining the level of output refinement. However, for text cropping and formula tasks, the granularity is implicitly provided by the cropping process and therefore requires no further explanation. Through this structured classification, this method demonstrates a versatile ability to interpret text from various sources and formats, bridging the gap between dedicated text reading models and real-world text recognition challenges.

[0097] In one or more embodiments provided in this specification, this method can be applied to a server to receive the target image and processing prompts sent by the client, thereby responding to the image processing request initiated by the user through the client and satisfying the user's image processing needs. The specific implementation is as follows:

[0098] The determination of the target image and the corresponding processing prompt information include:

[0099] The client receives the target image and the corresponding processing prompt information sent by the client, wherein the target image and the processing prompt information are sent by the client when the user performs an image upload operation on the client.

[0100] After inputting the target image and the processing prompt information into the target image processing model to obtain the image processing result of the target image, the method further includes:

[0101] The image processing results are sent to the client for display.

[0102] Specifically, users can perform an image upload operation on the client to provide the target image and processing prompt information to the client, and the client responds to the image upload operation by sending the target image and processing prompt information to the server to which this method is applied;

[0103] After receiving the target image and processing prompts, the server uses a target image processing model to process the target image and processing prompts to obtain the image processing result. Then, the server sends the image processing result to the client, so that the client can display the image processing result to the user, thereby satisfying the user's image processing needs.

[0104] In one or more embodiments provided in this specification, the processing prompt information can be provided by the client. However, in the case of incomplete processing prompt information, how to ensure that the target image processing model is used to efficiently process the target image using the processing prompt information becomes a problem that needs to be solved. The specific solution is as follows.

[0105] The process prompt information corresponding to the target image is determined as follows:

[0106] Receive the processing prompt information to be detected corresponding to the target image, and determine the parameter type of the processing prompt parameters contained in the processing prompt information to be detected, wherein the processing prompt information to be detected contains at least one type of processing prompt parameter;

[0107] If the parameter type is determined to be consistent with the preset type format, the processing prompt information to be detected is determined to be the processing prompt information corresponding to the target image;

[0108] If the parameter type is determined to be inconsistent with the preset type format, a reference processing prompt is determined for the target image from the processing prompt template based on the task type prompt parameter contained in the processing prompt information to be detected, and the reference processing prompt and the processing prompt information to be detected are determined as the processing prompt information corresponding to the target image.

[0109] The "processing prompts" can be understood as user-provided prompts that require processing.

[0110] Processing prompt parameters can be understood as parameters used to control the target image processing model to process the target image, or to control the type and direction of the content generated by the target image processing model. For example, processing prompt parameters can be any one or more of the following data: the task category of the target image, the writing type of the target image, the output granularity of the model, or the location information of the text to be recognized in the target image. The processing prompt information consists of one or more types of processing prompt parameters.

[0111] The preset type format can be understood as a pre-built format used to detect the processing prompt information to be detected. This preset type format is used to detect whether the processing prompt information to be detected is complete. If the parameter type is consistent with the preset type format, it is determined that the processing prompt information to be detected is complete; otherwise, it is determined that the processing prompt information to be detected is incomplete. For example, the preset type format can be a format template built based on four prompt information types: task category, writing type, output granularity, and the location information of the text to be recognized. Based on this, when the received processing prompt information to be detected only contains the processing prompt parameter of the task category parameter type, it is determined that the parameter type is inconsistent with the prompt information type contained in the preset type format, and therefore the processing prompt information to be detected is incomplete.

[0112] The processing prompt information template can be understood as a pre-set processing prompt information, used to determine the corresponding complete processing prompt information for the target image when the user cannot provide processing prompt information, cannot provide complete processing prompt information, or does not need to provide complete processing prompt information; the reference processing prompt information can be used to supplement the missing or unnecessary information in the processing prompt information to be detected provided by the user.

[0113] Taking the application of the image processing method provided in this specification in scenarios of reading various forms of text as an example, the image processing method is explained. The target image is a text image containing text information, and the processing prompts include prompts such as task category, writing type, output granularity, and / or the location information of the text to be recognized.

[0114] Based on this, the image processing method provided in this specification can receive processing prompts for text images, and detect whether the processing prompts contain complete processing prompts such as task category, writing type, output granularity, and location information of the text to be recognized. If so, the processing prompts are used as prompts to guide the model to process the text images.

[0115] Conversely, if the processing prompts only include task category, handwriting type, output granularity, and the location of the text to be recognized, and some of these prompts, such as only including the task type, can be used to determine the corresponding handwriting type, output granularity, and location of the text to be recognized from the preset processing prompt template (i.e., reference processing prompts). This information, along with the task type, will serve as a prompt to guide the model in processing the text image (i.e., processing prompts).

[0116] It should be noted that this method can automatically complete the processing prompts that do not require client-provided input. For example, for tasks involving complete handwritten images, full natural scene images, and / or full document images, granularity is an important parameter that determines the level of output refinement, and therefore needs to be provided. However, for tasks involving cropping handwritten text images, cropping text, and / or cropping formulas, the granularity is implicitly provided by the cropping process, so no additional explanation is needed. The word-level granularity is used as the preset prompt information for tasks involving cropping text images, cropping formula images, and / or cropping handwritten text images.

[0117] In the above embodiments, during the process of determining the processing prompt information, the processing prompt information to be detected corresponding to the target image is detected, and if the information parameter type is inconsistent with the preset type format, reference processing prompt information is determined for the target object from the processing prompt information template, and the reference processing prompt information and the processing prompt information to be detected are determined as the processing prompt information corresponding to the target image; thereby avoiding the problem of incomplete processing prompt information leading to low model processing efficiency.

[0118] In one or more embodiments provided in this specification, to ensure the consistency of input data, the image size or format of the target image can be adjusted for different image processing tasks, thereby ensuring that the model can perform image processing efficiently and thus ensuring the accuracy of the image processing results. Based on this, determining the target image includes:

[0119] Determine the target image format corresponding to the target image processing task;

[0120] Based on the target image format, the image format of the received image to be adjusted is adjusted to obtain the target image.

[0121] The target image format can be a preset image size and a preset image data format (e.g., JPG, PNG, etc.). Different image processing tasks can correspond to different image formats.

[0122] Following the previous example, for full-image tasks such as Scene Text Localization (STS), this method adjusts the long side of the image to 1024 while maintaining the aspect ratio. For RAT scene recognition, this method uses bounding box cues covering the entire image for evaluation. For PPR scenes, the center point of each polygon annotation is used; for BPR scenes, true quad bounding boxes are used. For cropped image tasks, such as Scene Text Recognition (STR), Handwritten Text Recognition (HTR), and Mathematical Expression Recognition (MER), which only involve RAT scenes, this method adjusts the long side of the image to 768 while maintaining the aspect ratio.

[0123] Step 204: Input the target image and the processing prompt information into the target image processing model to obtain the image processing result of the target image. The target image processing model is obtained by training an initial image processing model using initial training sub-data and target training sub-data. The initial image processing model is obtained by training a training image processing model using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two image processing tasks.

[0124] The target image processing model can be understood as a model used for image processing. For example, the target image processing model can be a target text recognition model used to recognize text information contained in a target image; or, the target image processing model can be a target object recognition model used to recognize target objects contained in a target image. Correspondingly, the image processing result can be text information contained in the target image, or target objects contained in the target image.

[0125] It should be noted that the target training sub-data and the initial training sub-data correspond to different types of image processing tasks; the target training sub-data and the initial training sub-data correspond to different loss functions; the target training sub-data corresponds to the target loss function, and the initial training sub-data corresponds to the initial loss function.

[0126] During the training of the initial image processing model using the initial training sub-data and the target training sub-data, the initial loss function can be determined based on the initial training sub-data, and the target loss function can be determined based on the target training sub-data. The model parameters of the initial image processing model can then be adjusted using the initial loss function and the target loss function to obtain the target image processing model.

[0127] In one or more embodiments provided in this specification, to effectively address the various challenges posed by different text reading scenarios, the training data can be divided into two main subsets based on the image type of the sample images. The first subset (i.e., the initial training sub-data) is geared towards full-image tasks, such as images of complete handwritten text, full natural scenes, and / or full documents; while the second subset (i.e., the target training sub-data) is geared towards cropped image tasks, such as cropped text images, cropped formula images, and / or cropped handwritten text images. These subsets differ not only in image size but also in the use of their respective optimized independent data loaders. Furthermore, the model is trained to distinguish between two main writing types: printed and handwritten. These inherently different text types require unique recognition strategies, and the target text recognition model (i.e., the target image processing model) seamlessly adapts to these needs.

[0128] In one or more embodiments provided in this specification, the target image is a target text image, the target image processing model is a target text recognition model, and the at least two image processing tasks are at least two text recognition tasks.

[0129] The step of inputting the target image and the processing prompt information into the target image processing model to obtain the image processing result of the target image includes:

[0130] The target text image and the processing prompt information are input into the target text recognition model to obtain the text information contained in the target text image. This enables the execution of any one of the at least two text recognition tasks through a single target text recognition model, avoiding the problem of the limited capabilities of neural network models and enabling flexible processing of images for multiple text recognition tasks.

[0131] In one or more embodiments provided in this specification, inputting the target image and the processing prompt information into a target image processing model to obtain the image processing result of the target image includes:

[0132] The target image and the processing prompt information are input into the target image processing model, wherein the target image processing model includes an image encoding unit, an information encoding unit, and an encoding processing unit;

[0133] The target image is encoded using the image encoding unit to obtain the target image code corresponding to the target image.

[0134] The information encoding unit is used to encode the processing prompt information to obtain the processing prompt information encoding corresponding to the processing prompt information.

[0135] Using the encoding processing unit, the target image encoding is performed according to the processing prompt information encoding to identify the result, thereby obtaining the image processing result of the target image.

[0136] The image encoding unit can be understood as a network layer or sub-model used to encode the target image; for example, the image encoding unit can be an image encoder. The information encoding unit can be understood as a network layer or sub-model used to encode the processing prompt information; for example, the information encoding unit can be a prompt information encoder. The encoding processing unit can be understood as a network layer or sub-model used to decode the encoding of the processing prompt information and the encoding of the target image; for example, the encoding processing unit can be a recognition decoder.

[0137] Following the previous example, the image processing model (target text recognition model) architecture of this scheme is an encoding and decoding framework inspired by the principles of the SAM model, and introduces a cue encoder to encode various cues. The entire architecture combines these components to form a system designed to robustly interpret text from various image formats.

[0138] The model in this method includes a cue encoder, which is designed to process various cue information, informing the model of the task category, writing type, output granularity, and location information of the text to be recognized. The embedding for each category is defined through location encoding and learned embeddings. For example, the embedding for each task category is calculated as follows:

[0139] Enhccidingc7tcgor v @@PE+EtMc,o,,

[0140] in The learned embeddings represent four explicit task categories (full image of a natural scene, full image of a document, cropped text, and cropped formula) plus an additional embedding to handle cases where image categories are not specified; this setup enhances the model's flexibility, enabling it to perform robustly during inference for images or tasks with unknown categories. Similarly, handwriting type and granularity cues include an unspecified category to maximize the model's adaptability across various text recognition scenarios. The point and box cue embeddings are generated in a similar manner, but for the box representation, a quadrilateral form with four points (top left, top right, bottom right, bottom left) is used to provide more precise location.

[0141] By employing a unified encoder-decoder architecture, this model can handle various text reading tasks, including Natural Scene Text Recognition (STR), Handwritten Text Recognition (HTR), and Mathematical Expression Recognition (MER). Unlike dedicated models that can only handle specific types of text recognition tasks, this target text recognition model can coordinate multiple tasks within a single architecture, achieving high-precision and efficient text recognition. Furthermore, by integrating a Prompt Encoder, the target text recognition model can accurately recognize text based on different task categories, handwriting types, and output granularity. The Prompt Encoder allows users to specify the area of ​​text recognition and the level of output refinement, improving user interactivity and accuracy while avoiding complex preprocessing steps.

[0142] The target image processing model also includes an image encoder. During the image encoding process, multi-scale feature extraction and enhancement can be achieved. Specifically, a pre-trained Swin-BTransformer and Feature Pyramid Network (FPN) are used to extract and enhance text features in the target image at multiple scales to obtain the enhanced target image encoding. This method constructs rich text representations at different scales and resolutions, significantly improving recognition performance in complex scenes.

[0143] In one or more embodiments provided in this specification, the encoding processing unit is an image decoding unit;

[0144] The step of using the encoding processing unit to perform result recognition on the target image encoding according to the processing prompt information encoding, and obtaining the image processing result of the target image, includes:

[0145] Using the image decoding unit, the target image is decoded according to the processing prompt information encoding to obtain the image processing result of the target image.

[0146] Following the previous example, inspired by the Transformer architecture, the recognition decoder in this method is an autoregressive module that generates the output text sequence. It combines visual features from the image encoder and cue embeddings from the cue encoder to generate the final text recognition result. This decoder (i.e., the image decoding unit) contains Transformer layers with multiple attention heads; for example, it contains six Transformer layers with eight attention heads each, all randomly initialized.

[0147] For a multi-head attention layer (Transformer layer) with N attention heads, each input vector (i.e., the target image encoding and the processing cue information encoding) is split into N smaller vectors, each of which undergoes an independent linear transformation into a different subspace. Then, the query, key, and value vectors in each subspace are processed using a dot product attention mechanism to calculate attention weights, and the value vectors are then weighted and summed based on these weights. Finally, the outputs of the N subspaces are concatenated and transformed into the final output vector (i.e., the image processing result) through an additional linear layer.

[0148] This method achieves efficient single-pass decoding by employing a recognition decoder. Specifically, the target text recognition model uses an autoregressive recognition decoder, combining visual features and cue embeddings to generate the final text recognition result. This method significantly improves inference speed through an efficient single-pass decoding process.

[0149] In one or more embodiments provided in this specification, the following are also included:

[0150] Upon receiving an unknown image, the corresponding unknown processing prompt information is determined from the processing prompt information template. This unknown image is not an image determined from the target image processing task.

[0151] The unknown image and the unknown processing prompt information are input into the target image processing model to obtain the image processing result of the unknown image.

[0152] Here, an unknown image can be understood as an image that is not determined in at least two image processing tasks, and the unknown image may not contain text information or target objects; the target image processing model in the image processing method provided in this specification can learn additional embeddings during training to handle the case of unspecified image (i.e., unknown image) categories; this setting enhances the flexibility of the model, enabling the model to perform robustly for images or tasks with unknown categories during inference.

[0153] For the processing steps of the unknown image and the unknown processing prompt information, please refer to the processing steps of the target image and the processing prompt information described above, which will not be repeated here.

[0154] In one or more embodiments provided in this specification, before inputting the target image and the processing prompt information into the target image processing model to obtain the image processing result of the target image, the method further includes:

[0155] The training data corresponding to the image processing model to be trained and the at least two image processing tasks are determined, wherein the training data includes sample images corresponding to the at least two image processing tasks, as well as sample labels and sample processing prompts corresponding to the sample images;

[0156] Based on the task types of the at least two image processing tasks, the initial training sub-data and the target training sub-data are determined from the training data;

[0157] The initial image processing model is obtained by training the image processing model to be trained using the initial training sub-data.

[0158] The initial image processing model is trained using the initial training sub-data and the target training sub-data to obtain the target image processing model, wherein the target image processing model is used to perform the at least two image processing tasks.

[0159] For an explanation of the above model training process, please refer to the corresponding or relevant content in the following model training methods, which will not be elaborated on here.

[0160] The image processing method provided in one or more embodiments of this specification offers a method for training an initial image processing model using initial training sub-data and target training sub-data to obtain a target image processing model. The target training sub-data and the initial training sub-data are determined from the training data based on the task types of at least two image processing tasks. Therefore, during image processing, a target image and corresponding processing prompts can be determined from any one of the at least two image processing tasks. The target image and processing prompts are then input into the target image processing model to obtain the image processing result. This enables the execution of any one of at least two image processing tasks using a single target image processing model, avoiding the limitation of neural network models and enabling flexible processing of images for various image processing tasks.

[0161] The following is in conjunction with the appendix Figure 3 Taking the application of the image processing method provided in this specification in scenarios involving reading various forms of text as an example, the image processing method will be further explained. Figure 3 A flowchart illustrating the processing procedure of an image processing method provided in one embodiment of this specification is shown.

[0162] The target text recognition model architecture in this method is an encoding and decoding framework, and it introduces a cue encoder to encode various cues, based on... Figure 3As can be seen, the entire architecture combines these components to form a system designed to robustly interpret text from various image forms. The application process of this target text recognition model architecture specifically includes the following steps one through three.

[0163] Step 1: Obtain the text image for text recognition and the prompt information for that text image.

[0164] The text image can be a full image of a natural scene, a full image of a document, a handwritten text image, a cropped text image, or a cropped formula image, etc.

[0165] Different image resizing strategies are employed during text recognition for different types of text images. Specifically:

[0166] For full-image tasks such as Scene-to-Text (STS), the long side of the image can be adjusted to 1024 while maintaining the aspect ratio.

[0167] For image cropping tasks, such as scene text recognition (STR), handwritten text recognition (HTR), and mathematical expression recognition (MER), involving only RAT scenarios, the long side of the image can be adjusted to 768 while maintaining the aspect ratio of the image.

[0168] The prompt information can include the task category, writing type, output granularity, and location information of the text to be recognized for the text image.

[0169] It should be noted that, in addressing the challenge of reading text from various types of images, the target text recognition model introduces a unified framework that encompasses a wide range of text interpretation scenarios that transcend the boundaries of traditional OCR. Based on the source and presentation of the image, this method categorizes text reading tasks into four main types: full-image natural scenes, full-image documents, cropped text, and cropped formulas.

[0170] Since full-view images of natural scenes and full-view images of documents typically contain various heterogeneous text styles and layouts, a more general recognition method is needed. For these images, this method introduces three different recognition scenarios to determine the corresponding text recognition region (i.e., the location information of the text to be recognized) in the image, namely:

[0171] RAT (RecognizeAllText): Recognizes all text in the entire image without requiring specific location cues. In other words, for RAT recognition scenarios, box cues covering the entire image can be used for evaluation.

[0172] PPR (Point Prompt Recognition): Recognizes text within a specific area around a point prompt, allowing for precise recognition of target text. In other words, for PPR scenarios, the center point of each polygon annotation can be used.

[0173] BPR (BoxPromptRecognition): Recognizes text within a user-defined bounding box. It can be used to extract text from a specific region of interest. In other words, for BPR scenarios, a real quadrilateral bounding box is used.

[0174] For cropped text images and formula text images, these tasks involve directly interpreting the text or mathematical expressions in the image without location cues, and instead directly recognizing the text in the entire image, thus simplifying the recognition process.

[0175] Furthermore, this method introduces a granularity in the recognition output to meet the level of detail required by different text reading applications. This granularity is divided into two levels:

[0176] Word-level: At this level, the model focuses on identifying words as the atomic units of text.

[0177] Line-level: At this level, the model recognizes an entire line of text, which may contain multiple words or even sentences.

[0178] For full-image tasks of natural scenes and full-image tasks of documents, granularity is an important parameter that determines the level of refinement of the output. However, for cropped text images and formula text images, the granularity is implicitly given by the cropping process, so no further explanation is needed.

[0179] Step 2: Input the text image and prompt information into the model (i.e., the target text recognition model), and use the image encoder and prompt encoder in the model to encode the text image and prompt information.

[0180] Specifically, the full image of the natural scene, the full image of the document, the cropped text image, or the cropped formula image are input into the image encoder for encoding processing to obtain the corresponding image code.

[0181] The prompt message is input into the prompt encoder for encoding processing, thereby obtaining the corresponding prompt message code.

[0182] It should be noted that this prompt encoder is designed to process the encoding of various prompts, thereby informing the model, during the text recognition process, of information such as the task category, handwriting type, output granularity, and location information of the text to be recognized. This method defines the embedding for each category through positional encoding (i.e., PE in the formula below) and the learned embeddings.

[0183] For example, the embedding calculation for task categories is as follows:

[0184] EuibcddingcMc,o,.i@@PE+ft.7tcgoiv

[0185] in It represents four explicit task categories (full natural scene, full document, cropped text, and cropped formula) and an additional learned embedding to handle cases where the image category is not specified; this setting enhances the model's flexibility, enabling it to perform robustly for images with unknown categories during inference.

[0186] Similarly, writing type and granularity hints include an unspecified category to maximize the model's adaptability across various text recognition scenarios. Hints based on points and boxes are generated in a similar manner; for box hints, this scheme uses a quadrilateral form with four points (top left, top right, bottom right, bottom left) to provide more precise location.

[0187] Step 3: Input the image encoding and prompt information encoding into the recognition decoder in the model to obtain the text information contained in the text image.

[0188] Specifically, inspired by the Transformer architecture, the recognition decoder in this scheme is an autoregressive module that generates the output text sequence. It combines visual features from the image encoder and cue embeddings from the cue encoder to generate the final text recognition result; and the recognition decoder contains six Transformer layers with eight heads each, all randomly initialized.

[0189] Before applying inference, the target text recognition model in this method needs to be trained using training data. In order for the trained target text recognition model to perform robust text reading in various scenarios, this solution prepares a dataset and conducts a two-stage training process. Specifically, the training process of the target text recognition model architecture includes the following steps one to four.

[0190] Step 1: Prepare training data.

[0191] Specifically, this approach divides the training data into two subsets. To effectively address the various challenges posed by different text reading scenarios, this method divides the training data into two main subsets to accommodate the different dimensions of full-image and cropped image tasks. The first subset is geared towards full-image tasks, such as full-image views of natural scenes and full-image views of documents, while the second subset is geared towards cropped image tasks, such as cropping text and cropping formulas.

[0192] These subsets differ not only in image size but also in the use of their own optimized, independent data loaders.

[0193] In addition, full-size images of natural scenes and documents are resized to have a long side of 1024 pixels. For cropped images that include text fragments and formulas, the long side can be resized to 768 pixels.

[0194] Step 2: Pre-train the model.

[0195] First, the initial phase focuses on pre-training using full-map data, which involves complex large-scale recognition challenges;

[0196] For RAT scenarios, the dot tooltip can be set to [0,0], and the box tooltip can be defined as the full image size;

[0197] For PPR, the point (i.e., the text position) can be sampled uniformly within the text bounding box;

[0198] For BPR, the true bounding box is used after adding noise (10% of the box size, but up to 20 pixels).

[0199] Secondly, during each forward propagation, multiple points and boxes are selected from a single image, up to eight cues, so that the model can learn the contextual relationships of text entities in the image more comprehensively.

[0200] When both word and line comments are present, the granularity is randomly selected to guide the selection of the corresponding true value.

[0201] Finally, data augmentation was performed using a batch size of 2, combined with instance-aware random cropping, rotation between -90 and 90 degrees, random scaling, and color jitter.

[0202] In addition, the pre-training used the AdamW optimizer with an initial learning rate of 5e^{-4} and a total of 1,000k steps. The first 5k steps used a warm-up plan, and then the learning rate was linearly decayed to zero.

[0203] Step 3: Jointly train the model.

[0204] Following the pre-training phase, this approach introduces cropped image data for training. A separate data loader is maintained for each subset, with batches alternating between full-image and cropped image data to ensure a diverse learning experience.

[0205] The data preparation and training techniques for the full image subset are consistent with those in the pre-training phase. For cropped image subsets, there is no need to recognize text granularity because these images are pre-cropped to contain single words or lines of text, which is suitable for RAT scenarios. Therefore, point hints can be set to [0,0], and box hints can be designed to cover the entire image size; where point hints and box hints can be understood as the points and boxes selected in the above images.

[0206] For cropped images, a larger batch size of 16 is used, and common text image enhancement methods such as perspective and affine transformations, blurring, noise, and rotation are employed to simulate various environmental conditions.

[0207] The joint training continues to use the AdamW optimizer for a total of 500k steps with an initial learning rate of 3e^{-4}, the same learning rate schedule as the pre-training phase.

[0208] Step 4: During the two-stage training process, the model parameters are adjusted using the loss function.

[0209] Specifically, the loss function of the model in this scheme is a combination of four independent loss components, each corresponding to an independent recognition task. The full graph subset involves three types of loss: one for the RAT task. Used for PPR tasks and used for BPR tasks In contrast, cropping a subset of images has only a single loss associated with the RAT task. Related. Each of these losses has an equal weight, set to 1 for simplicity.

[0210] Each of these losses is calculated using the standard cross-entropy loss, which compares the prediction to the text's true label.

[0211] Based on the above steps, the image processing method in one or more embodiments of this specification provides a general expert model for reading various forms of text, namely, a target character recognition model. This general expert model provides a comprehensive solution for diverse text reading scenarios, capable of handling various text reading tasks, serving as a multifunctional and unified text reading solution. This model embodies the combination of high-precision dedicated text reading models and the broad adaptability of multimodal methods. Similarly, this target character recognition model combines the specificity of dedicated models with the multifunctionality of general models, enabling the model to cope with various heterogeneous text reading tasks. In multiple text reading scenarios, it surpasses dedicated text reading models and large multimodal models (MLLMs), establishing a new benchmark. Furthermore, the target character recognition model has high interactivity; users can specify the text recognition region and select the granularity of the output, thereby enhancing usability and accuracy.

[0212] During inference, the target text recognition model employs a simplified approach, generating predictions based on the input scene category and specific cues provided. This enables the model to accurately interpret and recognize text in a variety of input images without directly using detection or segmentation models.

[0213] The advantage of this target text recognition model lies in its unified architecture, which combines the high accuracy of specialized models with the broad adaptability of general-purpose models, thus providing a more comprehensive and efficient text reading solution. This integrated capability enables the target text recognition model to excel in various practical application scenarios, specifically including the following five technical effects.

[0214] 1. High-precision recognition: In various text reading tasks such as natural scene text, handwritten text, and mathematical expressions, the target text recognition model shows good recognition accuracy, surpassing existing dedicated models and multimodal large-scale language models.

[0215] 2. Efficient Inference: Thanks to the efficient single-decoding process and optimized training strategy, the target text recognition model is significantly faster than existing MLLMs and dedicated models in terms of inference speed, enabling real-time text recognition applications.

[0216] 3. Multi-task adaptability: Through a unified architecture and prompting encoder, the target text recognition model can handle multiple text reading tasks in a single model, filling the technical gap in unified multi-task processing and significantly improving the model's versatility and practicality.

[0217] 4. Enhanced user interactivity: The introduction of the prompt encoder allows users to adjust the text recognition area and the level of output refinement according to specific needs, greatly improving the flexibility and applicability of the model.

[0218] 5. Strong generalization ability: Through a multi-stage training process and diverse datasets, the target text recognition model can perform well in a wide range of text reading scenarios, including handling text recognition tasks with various writing styles and complex backgrounds.

[0219] Based on the above, this solution proposes a general-purpose recognition model with strong versatility. A single model can perform text recognition in various scenarios, including whole images in natural scenes, documents, image matting, formulas, etc. The model supports multi-granularity recognition, outputting both word and line granularities—something many previous expert and large-scale models could not achieve. The model is highly flexible; users can interact with it via prompts, such as specifying the region in an image to be recognized via point-prompt or box-prompt, or specifying the output granularity via granularity-prompt. The recognition results are independent of detection and can be well integrated with detection and segmentation models. Furthermore, the model exhibits superior performance and high recognition accuracy, achieving or even surpassing state-of-the-art (SOTA) performance on various academic datasets. Compared to large multimodal models, it is more specialized, more accurate, and smaller.

[0220] For the specific experimental results of this scheme and related schemes, please refer to the following content.

[0221] Firstly, the capabilities of our proposed model and state-of-the-art (SOTA) models in various text reading tasks (including MLLMs) were compared. As shown in Table 1, SOTA-specific models (i.e., SOTA Methods) only excel in specific text reading tasks. In contrast, MLLMs and the target character recognition model (Platipus) proposed in our scheme, as general-purpose models, can compete in a wide range of text reading tasks.

[0222] Table 1

[0223]

[0224] The second aspect: comparison with Scene Text Localization (STS).

[0225] To further demonstrate performance on the full natural scene map, MLLMs and previous state-of-the-art (SOTA) scene text localization models were evaluated on the scene text localization benchmark. The comparison results are shown in Table 2. For the comparison of different prompts, the bounding box prompts (BPR) and point prompts (PPR) in the target text recognition model outperform the full map prompts (RAT) when explicitly indicating the location of text. Compared to dedicated models on STS, which are typically fine-tuned on specific datasets and support only one granularity, the target text recognition model is able to read text with both word-level and line-level granularity. Furthermore, the target text recognition model outperforms the SOTA on the four benchmarks in the PPR and BPR settings, and its performance in the RAT scene is comparable to that of dedicated models on STS. When compared to the general-purpose model GPT-4V, the target text recognition model significantly outperforms it even in the RAT setting.

[0226] Table 2

[0227]

[0228] Third aspect: Comparison with scene text recognition (STR).

[0229] The performance of our proposed target character recognition model on STR is compared with that of state-of-the-art (SOTA) models and MLLMs. Results from six benchmarks are shown in the table. All results were obtained without any dictionary-based post-processing. It is worth noting that, to obtain results from GPT-4V, our method uses the prompt "Please only output the text content in the scene text image; no additional description or formatting is required" and filters out text irrelevant to the description. As shown in Table 3, although the proposed target character recognition model is a unified four-task model, it achieves state-of-the-art (SOTA) results on fine-grained tasks such as STR, surpassing general models (e.g., GPT-4V and Monkey) and STR-specific models (e.g., ABINet, MGP-STR, and PARSeq).

[0230] Table 3

[0231]

[0232] Fourth aspect: Comparison with handwritten text recognition (HTR).

[0233] To verify the adaptability of the target character recognition model to different types of text, such as handwritten text, it was evaluated on the HTR benchmark and compared with previous state-of-the-art methods and MLLMs. The results of the LMM model were generated by using the prompt "Please output only the text content in the handwritten text image, without any additional description or formatting" and filtering out text irrelevant to the description. As shown in Table 4, GPT-4V showed low accuracy on the three Latin HTR benchmarks, including English and French text, indicating poor adaptability to handwritten text. In contrast, the proposed target character recognition model set a new state-of-the-art on the three benchmarks, achieving performance improvements of 8.2%, 0.1%, and 1.2% on datasets such as IAM, CVL, and RIMES compared to previous state-of-the-art methods. These results demonstrate that, as a unified model, the target character recognition model adapts well to different types of text.

[0234] Table 4

[0235]

[0236] Fifth aspect: Mathematical Expression Recognition (MER) comparison.

[0237] This approach also compares the performance of the target text recognition model with MLLMs and open-source formula recognition methods on the MER benchmark LaTeX-OCR. As shown in Table 5, GPT-4V performs poorly in structured text such as formulas. The proposed target text recognition model outperforms LaTeX-OCR, improving CER (Character Error Rate) and WER (Word Error Rate) by 1.5% and 1.8%, respectively.

[0238] Table 5

[0239]

[0240] Sixth aspect: Efficiency (FPS) comparison.

[0241] Table 6 below presents a comparison of inference speeds (evaluated on a single V100 GPU). It can be observed that the target text recognition model exhibits superior inference speed, surpassing MLLMs (e.g., mPLUG-Owl2) and outperforming dedicated models on STS, HTR, and MER tasks. In particular, the target text recognition model runs even 3 times faster than SPTS, primarily due to its efficient single-pass decoding process.

[0242] Table 6

[0243]

[0244] See Figure 4 , Figure 4 A flowchart of a model training method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0245] Step 402: Determine the image processing model to be trained and the training data corresponding to at least two image processing tasks, wherein the training data includes sample images corresponding to at least two image processing tasks, as well as sample labels and sample processing prompts corresponding to the sample images.

[0246] Here, the sample label can be understood as the image processing result serving as the sample label. For example, the sample label can be real text contained in the image or real objects contained in the image. The sample processing prompt information can be understood as the processing prompt information serving as the training sample.

[0247] Step 404: Determine the initial training sub-data and the target training sub-data from the training data according to the task types of the at least two image processing tasks.

[0248] In one or more embodiments provided in this specification, determining the initial training sub-data and the target training sub-data from the training data based on the task types of the at least two image processing tasks includes:

[0249] Based on the task types of the at least two image processing tasks, the at least two image processing tasks are divided into complete image processing tasks corresponding to complete image types and cropped image processing tasks corresponding to cropped image types.

[0250] From the training data, determine the complete sample image corresponding to the complete image processing task, and use the complete sample image, the sample label and sample processing prompt information corresponding to the complete sample image as the initial training sub-data;

[0251] From the training data, the sample cropped image corresponding to the image cropping processing task is determined, and the sample cropped image, along with the sample label and sample processing prompt information corresponding to the sample cropped image, is used as the target training sub-data.

[0252] Taking the application of the model training method provided in this manual in scenarios involving reading various forms of text as an example, this method is explained. To train the target character recognition model of this method to perform robust text reading in various scenarios, a dataset was prepared and a two-stage training process was conducted. To effectively address the various challenges posed by different text reading scenarios, the training data was divided into two main subsets. The first subset (i.e., the initial training sub-data) is geared towards full-image tasks, such as full-image samples of complete image types like natural scene full images, full-image documents, and / or complete handwritten images. The second subset is geared towards cropped image tasks, such as cropped text images, cropped formula images, and / or cropped handwritten images. These subsets not only differ in image size but also use their own optimized independent data loaders.

[0253] Furthermore, the image sizes are adjusted during the process of dividing the training data into two subsets to obtain adjusted sample images to adapt to the different dimensions of the full-image and cropped image tasks; for example, full-size images of natural scenes and documents are resized to have a long side of 1024 pixels. For cropped images, including text fragments and formulas, the long side is resized to 768 pixels.

[0254] In the above embodiments, comprehensive dataset and multi-task training were implemented. The Worms comprehensive text reading dataset was constructed, containing four main categories of data: full-image natural scenes, full-image documents, cropped text, and cropping formulas. Through a two-stage training process, including pre-training on full-image data and joint training on cropped image data, the target text recognition model performed excellently in various text reading scenarios.

[0255] Step 406: Use the initial training sub-data to train the image processing model to be trained, and obtain the initial image processing model.

[0256] In one or more embodiments provided in this specification, training the image processing model to be trained using the initial training sub-data to obtain the initial image processing model includes:

[0257] Determine the initial sample images included in the initial training sub-data, as well as the initial sample labels and initial sample processing prompts corresponding to the initial sample images;

[0258] The initial sample image and the initial sample processing prompt information are input into the image processing model to be trained to obtain the initial sample image processing result;

[0259] Based on the initial sample image processing results and the initial sample labels, a loss function is calculated, and the model parameters of the image processing model to be trained are adjusted based on the loss function to obtain the initial image processing model.

[0260] Following the previous example, the initial phase focuses on pre-training using full-image data, which addresses the complex challenge of large-scale recognition. For the RAT scenario, point cues are set to [0,0], and bounding box cues are defined as the full image size. For PPR, points are uniformly sampled within text bounding boxes, while for BPR, ground truth bounding boxes are used after adding noise (10% of the box size, but at most 20 pixels). Furthermore, during each forward propagation, multiple points and boxes, up to eight cues, are selected from a single image to allow the model to learn the contextual relationships of text entities in the image more comprehensively. When both word and line annotations are present, the granularity is randomly selected to guide the selection of the corresponding ground truth. This method uses a batch size of 2 and incorporates instance-aware random cropping, rotations between -90 and 90 degrees, random scaling, and color jitter for data augmentation. Pre-training uses the AdamW optimizer with an initial learning rate of 5e^{-4} for a total of 1,000k steps. The first 5k steps use a warm-up plan, followed by a linear decay of the learning rate to zero.

[0261] Furthermore, the model's loss function is a combination of four independent loss components, each corresponding to an independent recognition task. The full graph subset involves three types of losses: those used for the RAT task. Used for PPR tasks and used for BPR tasks In contrast, cropping a subset of images has only a single loss associated with the RAT task. Related. Each of these losses has an equal weight, set to 1 for simplicity.

[0262]

[0263] Each of these losses is calculated using the standard cross-entropy loss, which compares the predicted probability to the true label of the text.

[0264] In the initial stage, the following can be adopted: These three loss functions are used to tune the model parameters.

[0265] Step 408: Use the initial training sub-data and the target training sub-data to train the initial image processing model to obtain the target image processing model, wherein the target image processing model is used to perform the at least two image processing tasks.

[0266] In one or more embodiments provided in this specification, training the initial image processing model using the initial training sub-data and the target training sub-data to obtain the target image processing model includes:

[0267] The initial sample images contained in the initial training sub-data are determined, as well as the initial sample labels and initial sample processing prompts corresponding to the initial sample images. The target sample images contained in the target training sub-data are determined, as well as the target sample labels and target sample processing prompts corresponding to the target sample images.

[0268] The initial sample image and the initial sample processing prompt information are input into the initial image processing model to obtain the initial sample image processing result;

[0269] Based on the initial sample image processing results and the initial sample labels, an initial loss function is calculated, and the initial loss function is used to adjust the model parameters of the initial image processing model to obtain the adjusted image processing model.

[0270] The target sample image and the target sample processing prompt information are input into the adjusted image processing model to obtain the target sample image processing result.

[0271] Based on the target sample image processing results and the target sample labels, a target loss function is calculated, and based on the target loss function, the model parameters of the adjusted image processing model are adjusted to obtain the target image processing model.

[0272] Following the previous example, after the pre-training phase, cropped image data is introduced for training. A separate data loader is maintained for each subset, and batches alternate between full-image and cropped image data to ensure a diverse learning experience. During the alternating training process, the corresponding loss function is calculated when training with either full-image or cropped image data, and this loss function is used to adjust the model parameters.

[0273] Data preparation and training techniques for the full image subset remained consistent with the pre-training phase. For the cropped image subset, no text recognition granularity was required because these images were pre-cropped, containing only single words or lines of text. Only for RAT scenarios were used, with dot hints set to [0,0] and box hints designed to cover the entire image size. For cropped images, a larger batch size of 16 was used, and common text image augmentation methods such as perspective and affine transformations, blurring, noise, and rotation were employed to simulate various environmental conditions. Joint training continued using the AdamW optimizer for a total of 500k steps with an initial learning rate of 3e^{-4}, the same learning rate schedule as the pre-training phase.

[0274] During the joint training phase, the following methods can be adopted: These four loss functions are used to adjust the model parameters.

[0275] In the above embodiments, multimodal processing is achieved through two-stage model training. Although MLLMs perform well in processing multimodal data, they suffer from low computational efficiency and limited accuracy. The target text recognition model, through a unified architecture and efficient training strategy, achieves a dual improvement in accuracy and efficiency, and is highly adaptable to various text formats.

[0276] The model training method provided in one or more embodiments of this specification offers a method for training an initial image processing model using initial training sub-data and target training sub-data to obtain a target image processing model. The target training sub-data and the initial training sub-data are determined from the training data based on the task types of at least two image processing tasks. Therefore, during image processing, a target image and corresponding processing prompts can be determined from any one of the at least two image processing tasks. The target image and processing prompts are then input into the target image processing model to obtain the image processing result. This enables the execution of any one of at least two image processing tasks using a single target image processing model, avoiding the limitation of neural network models and enabling flexible processing of images for various image processing tasks.

[0277] See Figure 5 , Figure 5A flowchart of a character recognition method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0278] Step 502: Determine the target text image and the corresponding processing prompt information for the target text image, wherein the target text image is an image determined from a target text recognition task, and the target text recognition task is any one of at least two text recognition tasks.

[0279] Step 504: Input the target text image and the processing prompt information into the target text recognition model to obtain the text information contained in the target text image. The target text recognition model is obtained by training an initial text recognition model using initial training sub-data and target training sub-data. The initial text recognition model is obtained by training a text recognition model to be trained using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two text recognition tasks.

[0280] The text recognition method provided in one or more embodiments of this specification offers a method for training an initial text recognition model using initial training sub-data and target training sub-data to obtain a target text recognition model. The target training sub-data and the initial training sub-data are determined from the training data based on the task types of at least two text recognition tasks. Therefore, during the text recognition process of a target text image, the target text image and the corresponding processing prompt information can be determined from any one of the at least two text recognition tasks. The target text image and the processing prompt information are then input into the target text recognition model to obtain the text information contained in the target text image. This achieves the execution of any one of at least two text recognition tasks using a single target text recognition model, avoiding the problem of limited capabilities of neural network models and enabling flexible processing of images for multiple text recognition tasks.

[0281] The above is an illustrative scheme of a text recognition method according to this embodiment. It should be noted that the technical solution of this text recognition method and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the text recognition method, please refer to the description of the technical solution of the image processing method described above.

[0282] See Figure 6 , Figure 6 A flowchart of another character recognition method according to an embodiment of this specification is shown. This character recognition method is applied to a cloud-side device and specifically includes the following steps.

[0283] Step 602: Receive the target text image sent by the receiving end device and determine the processing prompt information corresponding to the target text image, wherein the target text image is an image determined from the target text recognition task, and the target text recognition task is any one of at least two text recognition tasks.

[0284] Step 604: Input the target text image and the processing prompt information into the target text recognition model to obtain the text information contained in the target text image. The target text recognition model is obtained by training an initial text recognition model using initial training sub-data and target training sub-data. The initial text recognition model is obtained by training a text recognition model to be trained using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two text recognition tasks.

[0285] After obtaining the text information contained in the target text image, the text information can be sent to the terminal device.

[0286] In one or more embodiments provided in this specification, the cloud-side device can be a central cloud device in a distributed architecture or an edge cloud device in a distributed architecture. The cloud-side device can also be a cloud-side device with a cloud desktop system or cloud desktop software installed and deployed. For example, the cloud-side device can be a cloud server, cloud host, etc. The end-side device can be understood as any terminal that interacts with the cloud-side device; the terminal can be a laptop, desktop computer, tablet computer, smart device, server, etc.

[0287] This specification provides one or more embodiments of a text recognition method applied to cloud-side devices. It offers a method for training an initial text recognition model using initial training sub-data and target training sub-data to obtain a target text recognition model. The target training sub-data and the initial training sub-data are determined from the training data based on the task types of at least two text recognition tasks. Based on this, during the text recognition process of a target text image sent by an end-side device, the method can receive information from the end-side device regarding the target text image determined from any one of the at least two text recognition tasks, as well as the corresponding processing prompts. The target text image and processing prompts are then input into the target text recognition model to obtain the text information contained in the target text image. This achieves the execution of any one of at least two text recognition tasks using a single target text recognition model, avoiding the limitation of neural network models and enabling flexible processing of images for multiple text recognition tasks.

[0288] See Figure 7 , Figure 7A flowchart of another character recognition method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0289] Step 702: Determine the target text image from the received target document, and determine the processing prompt information corresponding to the target text image, wherein the target text image is an image determined from a target text recognition task, and the target text recognition task is any one of at least two text recognition tasks.

[0290] Step 704: Input the target text image and the processing prompt information into the target text recognition model to obtain the text information contained in the target text image. The target text recognition model is obtained by training an initial text recognition model using initial training sub-data and target training sub-data. The initial text recognition model is obtained by training a text recognition model to be trained using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two text recognition tasks.

[0291] The target document can be understood as document data such as papers and books, or the target document can be a webpage; the target document can contain text information, or the target document has a target text image containing text information.

[0292] Determining the target text image from the received target document can be understood as performing image conversion processing on the received target document to obtain the target text image corresponding to the target document. For example, the document data can be converted into image data by cropping, screenshotting, saving as an image, taking a picture, etc., thereby obtaining the target document image.

[0293] Alternatively, determining the target text image from the received target document can also be understood as obtaining an image containing text information in the target document and using that image as the target text image; or, cropping the formulas in the target document to obtain the target text image.

[0294] In one or more embodiments provided in this specification, after obtaining the text information contained in the target text image, the text information can be used as input data for a large language model; the large language model can perform language reasoning tasks based on the input data; for example, the target text image is an image of a thesis document, the text information is the thesis data, the thesis data is used as input data to the large language model for reasoning, thereby reasoning out the central idea of ​​the thesis document.

[0295] This specification provides a text recognition method through one or more embodiments. It offers a method for training an initial text recognition model using initial training sub-data and target training sub-data to obtain a target text recognition model. The target training sub-data and the initial training sub-data are determined from the training data according to the task types of at least two text recognition tasks. Based on this, in the process of recognizing target text images from target documents, the target text image and the corresponding processing prompt information can be determined from any one of the at least two text recognition tasks. The target text image and the processing prompt information are then input into the target text recognition model to obtain the text information contained in the target text image. This achieves the execution of any one of at least two text recognition tasks using a single target text recognition model, avoiding the problem of limited capabilities of neural network models and enabling flexible processing of images for multiple text recognition tasks.

[0296] Corresponding to the above method embodiments, this specification also provides an image processing apparatus embodiment, which includes:

[0297] The image determination module is configured to determine a target image and determine the processing prompt information corresponding to the target image, wherein the target image is an image determined from a target image processing task, and the target image processing task is any one of at least two image processing tasks;

[0298] An image processing module is configured to input the target image and the processing prompt information into a target image processing model to obtain the image processing result of the target image. The target image processing model is obtained by training an initial image processing model using initial training sub-data and target training sub-data. The initial image processing model is obtained by training a training image processing model using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two image processing tasks.

[0299] Optionally, the image processing module is further configured to:

[0300] The target image and the processing prompt information are input into the target image processing model, wherein the target image processing model includes an image encoding unit, an information encoding unit, and an encoding processing unit;

[0301] The target image is encoded using the image encoding unit to obtain the target image code corresponding to the target image.

[0302] The information encoding unit is used to encode the processing prompt information to obtain the processing prompt information encoding corresponding to the processing prompt information.

[0303] Using the encoding processing unit, the target image encoding is performed according to the processing prompt information encoding to identify the result, thereby obtaining the image processing result of the target image.

[0304] Optionally, the encoding processing unit is an image decoding unit;

[0305] The image processing module is further configured to:

[0306] Using the image decoding unit, the target image is decoded according to the processing prompt information encoding to obtain the image processing result of the target image.

[0307] Optionally, the image determination module is further configured to:

[0308] The client receives the target image and the corresponding processing prompt information sent by the client, wherein the target image and the processing prompt information are sent by the client when the user performs an image upload operation on the client.

[0309] After inputting the target image and the processing prompt information into the target image processing model to obtain the image processing result of the target image, the method further includes:

[0310] The image processing results are sent to the client for display.

[0311] Optionally, the image determination module is further configured to:

[0312] Upon receiving an unknown image, the corresponding unknown processing prompt information is determined from the processing prompt information template. This unknown image is not an image determined from the target image processing task.

[0313] The unknown image and the unknown processing prompt information are input into the target image processing model to obtain the image processing result of the unknown image.

[0314] Optionally, the image determination module is further configured to:

[0315] Receive the processing prompt information to be detected corresponding to the target image, and determine the parameter type of the processing prompt parameters contained in the processing prompt information to be detected, wherein the processing prompt information to be detected contains at least one type of processing prompt parameter;

[0316] If the parameter type is determined to be consistent with the preset type format, the processing prompt information to be detected is determined to be the processing prompt information corresponding to the target image;

[0317] If the parameter type is determined to be inconsistent with the preset type format, a reference processing prompt is determined for the target image from the processing prompt template based on the task type prompt parameter contained in the processing prompt information to be detected, and the reference processing prompt and the processing prompt information to be detected are determined as the processing prompt information corresponding to the target image.

[0318] Optionally, the device further includes a model training module configured to:

[0319] The training data corresponding to the image processing model to be trained and the at least two image processing tasks are determined, wherein the training data includes sample images corresponding to the at least two image processing tasks, as well as sample labels and sample processing prompts corresponding to the sample images;

[0320] Based on the task types of the at least two image processing tasks, the initial training sub-data and the target training sub-data are determined from the training data;

[0321] The initial image processing model is obtained by training the image processing model to be trained using the initial training sub-data.

[0322] The initial image processing model is trained using the initial training sub-data and the target training sub-data to obtain the target image processing model, wherein the target image processing model is used to perform the at least two image processing tasks.

[0323] Optionally, the target image is a target text image, the target image processing model is a target text recognition model, and the at least two image processing tasks are at least two text recognition tasks;

[0324] The image processing module is configured as follows:

[0325] The target text image and the processing prompt information are input into the target text recognition model to obtain the text information contained in the target text image.

[0326] The image processing apparatus provided in one or more embodiments of this specification provides a method for training an initial image processing model using initial training sub-data and target training sub-data to obtain a target image processing model. The target training sub-data and the initial training sub-data are determined from training data based on the task types of at least two image processing tasks. Therefore, during image processing, a target image and corresponding processing prompts can be determined from any one of the at least two image processing tasks. The target image and processing prompts are then input into the target image processing model to obtain the image processing result of the target image. This enables the execution of any one of at least two image processing tasks using a single target image processing model, avoiding the limitation of neural network models and enabling flexible processing of images for various image processing tasks.

[0327] The above is an illustrative scheme of an image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0328] Corresponding to the above method embodiments, this specification also provides embodiments of a model training apparatus, which includes:

[0329] The first training data determination module is configured to determine the image processing model to be trained and the training data corresponding to at least two image processing tasks, wherein the training data includes sample images corresponding to at least two image processing tasks, as well as sample labels and sample processing prompts corresponding to the sample images;

[0330] The second training data determination module is configured to determine initial training sub-data and target training sub-data from the training data based on the task types of the at least two image processing tasks.

[0331] The first model training module is configured to train the image processing model to be trained using the initial training sub-data to obtain the initial image processing model.

[0332] The second model training module is configured to train the initial image processing model using the initial training sub-data and the target training sub-data to obtain a target image processing model, wherein the target image processing model is used to perform the at least two image processing tasks.

[0333] Optionally, the second training data determination module is configured to:

[0334] Based on the task types of the at least two image processing tasks, the at least two image processing tasks are divided into complete image processing tasks corresponding to complete image types and cropped image processing tasks corresponding to cropped image types.

[0335] From the training data, determine the complete sample image corresponding to the complete image processing task, and use the complete sample image, the sample label and sample processing prompt information corresponding to the complete sample image as the initial training sub-data;

[0336] From the training data, the sample cropped image corresponding to the image cropping processing task is determined, and the sample cropped image, along with the sample label and sample processing prompt information corresponding to the sample cropped image, is used as the target training sub-data.

[0337] Optionally, the first model training module is configured as follows:

[0338] Determine the initial sample images included in the initial training sub-data, as well as the initial sample labels and initial sample processing prompts corresponding to the initial sample images;

[0339] The initial sample image and the initial sample processing prompt information are input into the image processing model to be trained to obtain the initial sample image processing result;

[0340] Based on the initial sample image processing results and the initial sample labels, a loss function is calculated, and the model parameters of the image processing model to be trained are adjusted based on the loss function to obtain the initial image processing model.

[0341] Optionally, the second model training module is configured as follows:

[0342] The initial sample images contained in the initial training sub-data are determined, as well as the initial sample labels and initial sample processing prompts corresponding to the initial sample images. The target sample images contained in the target training sub-data are determined, as well as the target sample labels and target sample processing prompts corresponding to the target sample images.

[0343] The initial sample image and the initial sample processing prompt information are input into the initial image processing model to obtain the initial sample image processing result;

[0344] Based on the initial sample image processing results and the initial sample labels, an initial loss function is calculated, and the initial loss function is used to adjust the model parameters of the initial image processing model to obtain the adjusted image processing model.

[0345] The target sample image and the target sample processing prompt information are input into the adjusted image processing model to obtain the target sample image processing result.

[0346] Based on the target sample image processing results and the target sample labels, a target loss function is calculated, and based on the target loss function, the model parameters of the adjusted image processing model are adjusted to obtain the target image processing model.

[0347] The model training apparatus provided in one or more embodiments of this specification provides a method for training an initial image processing model using initial training sub-data and target training sub-data to obtain a target image processing model. The target training sub-data and the initial training sub-data are determined from the training data according to the task types of at least two image processing tasks. Based on this, during image processing, a target image and corresponding processing prompts can be determined from any one of the at least two image processing tasks. The target image and processing prompts are then input into the target image processing model to obtain the image processing result of the target image. This enables the execution of any one of at least two image processing tasks using a single target image processing model, avoiding the problem of limited capabilities of neural network models and achieving flexible image processing for various image processing tasks.

[0348] The above is an illustrative scheme of a model training device according to this embodiment. It should be noted that the technical solution of this model training device and the technical solution of the model training method described above belong to the same concept. For details not described in detail in the technical solution of the model training device, please refer to the description of the technical solution of the model training method described above.

[0349] Corresponding to the above method embodiments, this specification also provides an embodiment of a character recognition device, which includes:

[0350] The image determination module is configured to determine a target text image and determine the processing prompt information corresponding to the target text image, wherein the target text image is an image determined from a target text recognition task, and the target text recognition task is any one of at least two text recognition tasks;

[0351] An image processing module is configured to input the target text image and the processing prompt information into a target text recognition model to obtain the text information contained in the target text image. The target text recognition model is obtained by training an initial text recognition model using initial training sub-data and target training sub-data. The initial text recognition model is obtained by training a text recognition model to be trained using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two text recognition tasks.

[0352] The text recognition device provided in one or more embodiments of this specification provides a method for training an initial text recognition model using initial training sub-data and target training sub-data to obtain a target text recognition model. The target training sub-data and the initial training sub-data are determined from the training data according to the task types of at least two text recognition tasks. Based on this, during the text recognition process of a target text image, the target text image and the corresponding processing prompt information can be determined from any one of the at least two text recognition tasks. The target text image and the processing prompt information are then input into the target text recognition model to obtain the text information contained in the target text image. This enables the execution of any one of the at least two text recognition tasks through a single target text recognition model, avoiding the problem of the limited capabilities of neural network models and enabling flexible processing of images for multiple text recognition tasks.

[0353] The above is an illustrative scheme of a character recognition device according to this embodiment. It should be noted that the technical solution of this character recognition device and the technical solution of the above-described character recognition method belong to the same concept. For details not described in detail in the technical solution of the character recognition device, please refer to the description of the technical solution of the above-described character recognition method.

[0354] Figure 8 A structural block diagram of a computing device 800 according to one embodiment of this specification is shown. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0355] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of such networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. Access device 840 may include one or more of any type of wired or wireless network interface (e.g., network interface card (NIC)), such as IEEE 802.11 Wireless Local Area Network (WLAN) interface, Wi-MAX (Worldwide Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, and Near Field Communication (NFC).

[0356] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0357] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 can also be a mobile or stationary server.

[0358] The processor 820 is configured to execute computer-executable instructions that, when executed by the processor, implement the steps of any of the methods described above.

[0359] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to any one method embodiment, so the description is relatively simple; relevant parts can be referred to in the description of any one method embodiment.

[0360] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0361] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are relatively simple in description because they are substantially similar to any method embodiment; relevant parts can be referred to in the description of any method embodiment.

[0362] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0363] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solution of any of the above methods, and any details not described in detail in the technical solution of the computer program product can be referred to the description of the technical solution of any of the above methods.

[0364] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0365] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0366] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0367] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0368] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: Determine a target image and determine the processing prompt information corresponding to the target image, wherein the target image is an image determined from a target image processing task, and the target image processing task is any one of at least two image processing tasks; The target image and the processing prompt information are input into the target image processing model to obtain the image processing result of the target image. The target image processing model is obtained by training an initial image processing model using initial training sub-data and target training sub-data. The initial image processing model is obtained by training a training image processing model using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two image processing tasks.

2. The image processing method according to claim 1, wherein inputting the target image and the processing prompt information into the target image processing model to obtain the image processing result of the target image includes: The target image and the processing prompt information are input into the target image processing model, wherein the target image processing model includes an image encoding unit, an information encoding unit, and an encoding processing unit; The target image is encoded using the image encoding unit to obtain the target image code corresponding to the target image. The information encoding unit is used to encode the processing prompt information to obtain the processing prompt information encoding corresponding to the processing prompt information. Using the encoding processing unit, the target image encoding is performed according to the processing prompt information encoding to identify the result, thereby obtaining the image processing result of the target image.

3. The image processing method according to claim 2, wherein the encoding processing unit is an image decoding unit; The step of using the encoding processing unit to perform result recognition on the target image encoding according to the processing prompt information encoding, and obtaining the image processing result of the target image, includes: Using the image decoding unit, the target image is decoded according to the processing prompt information encoding to obtain the image processing result of the target image.

4. The image processing method according to claim 1, wherein determining the target image and determining the processing prompt information corresponding to the target image includes: The client receives the target image and the corresponding processing prompt information sent by the client, wherein the target image and the processing prompt information are sent by the client when the user performs an image upload operation on the client. After inputting the target image and the processing prompt information into the target image processing model to obtain the image processing result of the target image, the method further includes: The image processing results are sent to the client for display.

5. The image processing method according to claim 1, further comprising: Upon receiving an unknown image, the corresponding unknown processing prompt information is determined from the processing prompt information template. This unknown image is not an image determined from the target image processing task. The unknown image and the unknown processing prompt information are input into the target image processing model to obtain the image processing result of the unknown image.

6. The image processing method according to claim 1 or 4, wherein determining the processing prompt information corresponding to the target image includes: Receive the processing prompt information to be detected corresponding to the target image, and determine the parameter type of the processing prompt parameters contained in the processing prompt information to be detected, wherein the processing prompt information to be detected contains at least one type of processing prompt parameter; If the parameter type is determined to be consistent with the preset type format, the processing prompt information to be detected is determined to be the processing prompt information corresponding to the target image; If the parameter type is determined to be inconsistent with the preset type format, a reference processing prompt is determined for the target image from the processing prompt template based on the task type prompt parameter contained in the processing prompt information to be detected, and the reference processing prompt and the processing prompt information to be detected are determined as the processing prompt information corresponding to the target image.

7. The image processing method according to any one of claims 1 to 5, wherein the target image is a target text image, the target image processing model is a target text recognition model, and the at least two image processing tasks are at least two text recognition tasks; The step of inputting the target image and the processing prompt information into the target image processing model to obtain the image processing result of the target image includes: The target text image and the processing prompt information are input into the target text recognition model to obtain the text information contained in the target text image.

8. A model training method, comprising: Determine the image processing model to be trained and the training data corresponding to at least two image processing tasks, wherein the training data includes sample images corresponding to at least two image processing tasks, as well as sample labels and sample processing prompts corresponding to the sample images; Based on the task types of the at least two image processing tasks, determine the initial training sub-data and the target training sub-data from the training data; The initial image processing model is trained using the initial training sub-data to obtain the initial image processing model. The initial image processing model is trained using the initial training sub-data and the target training sub-data to obtain a target image processing model, wherein the target image processing model is used to perform the at least two image processing tasks.

9. The model training method according to claim 8, wherein determining the initial training sub-data and the target training sub-data from the training data based on the task types of the at least two image processing tasks includes: Based on the task types of the at least two image processing tasks, the at least two image processing tasks are divided into complete image processing tasks corresponding to complete image types and cropped image processing tasks corresponding to cropped image types. From the training data, determine the complete sample image corresponding to the complete image processing task, and use the complete sample image, the sample label and sample processing prompt information corresponding to the complete sample image as the initial training sub-data; From the training data, the sample cropped image corresponding to the image cropping processing task is determined, and the sample cropped image, along with the sample label and sample processing prompt information corresponding to the sample cropped image, is used as the target training sub-data.

10. The model training method according to claim 8, wherein the step of training the image processing model to be trained using the initial training sub-data to obtain the initial image processing model comprises: Determine the initial sample images included in the initial training sub-data, as well as the initial sample labels and initial sample processing prompts corresponding to the initial sample images; The initial sample image and the initial sample processing prompt information are input into the image processing model to be trained to obtain the initial sample image processing result; Based on the initial sample image processing results and the initial sample labels, a loss function is calculated, and the model parameters of the image processing model to be trained are adjusted based on the loss function to obtain the initial image processing model.

11. The model training method according to claim 8, wherein training the initial image processing model using the initial training sub-data and the target training sub-data to obtain the target image processing model comprises: The initial sample images contained in the initial training sub-data are determined, as well as the initial sample labels and initial sample processing prompts corresponding to the initial sample images. The target sample images contained in the target training sub-data are determined, as well as the target sample labels and target sample processing prompts corresponding to the target sample images. The initial sample image and the initial sample processing prompt information are input into the initial image processing model to obtain the initial sample image processing result; Based on the initial sample image processing results and the initial sample labels, an initial loss function is calculated, and the initial loss function is used to adjust the model parameters of the initial image processing model to obtain the adjusted image processing model. The target sample image and the target sample processing prompt information are input into the adjusted image processing model to obtain the target sample image processing result. Based on the target sample image processing results and the target sample labels, a target loss function is calculated, and based on the target loss function, the model parameters of the adjusted image processing model are adjusted to obtain the target image processing model.

12. A character recognition method, comprising: Determine the target text image and the corresponding processing prompt information for the target text image, wherein the target text image is an image determined from a target text recognition task, and the target text recognition task is any one of at least two text recognition tasks; The target text image and the processing prompt information are input into the target text recognition model to obtain the text information contained in the target text image. The target text recognition model is obtained by training an initial text recognition model using initial training sub-data and target training sub-data. The initial text recognition model is obtained by training a text recognition model to be trained using the initial training sub-data. The target training sub-data and the initial training data are determined from the training data according to the task types of the at least two text recognition tasks.

13. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images