Ocr-based task processing method, device, medium and product

Abstract

The embodiment of the application relates to an information technology task processing method, equipment, medium and product based on OCR. The method comprises the following steps: determining image features according to a document image and a visual encoder; mapping the image features to a text space of a decoder through a connector; determining text features according to a text instruction issued for the document image; and fusing the image features and the text features according to the decoder to determine an OCR recognition result. The technical problem of relatively independent modules in a traditional OCR framework, accuracy reduction caused by connection and error accumulation between the modules, and insufficient generalization ability when processing multiple complex tasks can be solved.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a task processing method, device, medium and product based on OCR. Background Technology

[0002] OCR is an abbreviation for Optical Character Recognition, which refers to a technology that converts text in an image into an editable text format. Specifically, OCR technology scans documents, photos, or other images containing text, extracts, analyzes, and recognizes the features of the characters in the image, converts them into character codes that a computer can understand and process, and ultimately generates editable text content for subsequent storage, editing, and other operations.

[0003] In related technologies, the implementation process of traditional OCR models requires multiple independent steps. First, image cropping is performed, dividing a large image containing text into smaller slices. Then, for each slice, character segmentation is performed, breaking down continuous text into individual characters for subsequent processing. After character segmentation, features of each character need to be extracted, such as stroke thickness and direction, to provide crucial information for character recognition. Finally, the extracted features are input into a classifier, which identifies the character by matching it with predefined character categories. Taking the mainstream OCR technology PaddleOCR as an example, it typically employs a modular pipeline design. This design mainly consists of multiple independent modules such as image layout detection, region cropping, and character recognition. Through the collaborative work of these modules, PaddleOCR can progressively handle complex OCR tasks.

[0004] However, the inventors have discovered at least the following technical problems in the related technology:

[0005] In a modular OCR framework, each module operates relatively independently. When each module strives for its own performance improvement, collaboration becomes difficult, leading to local optima and hindering overall optimization, resulting in poor overall performance. Furthermore, module independence limits the framework's generalization ability, especially when facing multilingual and multi-symbol recognition tasks. Each module struggles to adapt to different features, requiring separate expert models for each language or symbol, increasing system complexity and resource consumption. Moreover, module independence makes errors from earlier steps easily propagate to later steps. For example, if image preprocessing fails to accurately extract text regions, subsequent character segmentation and recognition modules will process based on erroneous data, severely impacting recognition accuracy. Summary of the Invention

[0006] One objective of this application is to provide an OCR-based task processing method, device, medium, and product, at least to solve the technical problem in related technologies where the various modules in the OCR framework are relatively independent, difficult to coordinate, and difficult to achieve overall optimization when facing complex tasks.

[0007] To achieve the above objectives, some embodiments of this application provide the following aspects:

[0008] In a first aspect, some embodiments of this application also provide an OCR-based task processing method, the method comprising: determining image features based on a document image and a visual encoder; mapping the image features to the text space of a decoder through a connector; determining text features based on text instructions issued for the document image; and fusing the image features and the text features according to the decoder to determine the OCR recognition result.

[0009] Secondly, some embodiments of this application also provide an electronic device, the electronic device comprising: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method described above.

[0010] Thirdly, some embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the method described above.

[0011] Fourthly, some embodiments of this application also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0012] Compared with related technologies, the solution provided in this application allows the encoder, connector, and decoder to jointly form a multimodal visual language model. The visual encoder converts document images into image features and simultaneously determines text features based on text instructions. The connector maps image features to the decoder's text space, enabling the visual encoder and decoder to work collaboratively. The decoder performs OCR recognition based on image and text features and outputs the recognition result. Because the multimodal visual language model of this application can be trained end-to-end, it eliminates the redundant expert modules in traditional OCR, integrating the visual encoder, connector, and decoder into a unified model, greatly simplifying the system architecture. This reduces module connection problems, avoids error accumulation, and achieves global collaboration. Moreover, the model directly uses raw multimodal data as input, processes it internally, and directly outputs the result, eliminating the need for manual design of intermediate steps and feature engineering. In summary, compared with traditional OCR, the model of this application has significant advantages. The simplified architecture, reduced module connections and error accumulation create favorable conditions for global collaboration, thereby improving the model's generalization ability and overall performance in complex scenarios. The model's recognition accuracy has been improved, and its compatibility and generalization ability in handling multiple tasks have been significantly enhanced, demonstrating excellent performance in different scenarios. Attached Figure Description

[0013] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0014] Figure 1 This is an exemplary flowchart of an OCR-based task processing method provided according to some embodiments of this application;

[0015] Figure 2 This is an exemplary schematic diagram of an OCR-based task processing method provided according to some embodiments of this application;

[0016] Figure 3 This is an exemplary structural diagram of an electronic device provided according to some embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The following terms are used in this document.

[0019] The flatten operation, also known as the flattening operation, primarily transforms multi-dimensional data structures into one-dimensional arrays.

[0020] A token is a feature representation of an image. Each token may correspond to a specific region or a specific visual feature in the image.

[0021] Linear layer MLP is a key component in multilayer perceptron, typically consisting of one or more fully connected layers (linear layers).

[0022] ViT-Det, Vision Transformer for Object Detection, is an object detection model developed based on the Vision Transformer (ViT) architecture.

[0023] A prompt, often translated as "hint," "prompt," or "prompt word," is a text instruction used to guide a model to generate specific outputs. It can be a question, a statement, or a task description.

[0024] Global attention, in Transformers, allows elements at each position to interact with all other positions in the sequence, thus capturing global dependencies in long sequences. When processing images, it allows the model to consider information from the entire image, but it is computationally expensive, especially for high-resolution images.

[0025] Local attention mechanisms restrict each location to interacting only with elements within a local region. This reduces computational cost, improves model efficiency, and also helps capture local features and details in the image to some extent.

[0026] First Embodiment

[0027] The first embodiment of this application relates to an OCR-based task processing method. The method is applied to a multimodal visual language model, which includes a visual encoder, a connector, and a decoder; as shown... Figure 1 As shown, the method may include the following steps:

[0028] Step S101: Determine image features based on the document image and the visual encoder;

[0029] Step S102: The image features are mapped to the text space of the decoder through the connector;

[0030] Step S103: Determine text features based on the text instructions issued by the decoder for the document image;

[0031] Step S104: Based on the decoder, the image features and the text features are fused to determine the OCR recognition result.

[0032] Reference Figure 1 , Figure 2 As shown, the visual encoder, connector, and decoder form a multimodal visual language model; the following sections will provide a detailed explanation of each of the above steps.

[0033] Specifically, in step S101, the document image refers to document materials existing in image form. These document materials can include formulas, tables, musical notes, chemical formulas, charts, and text in different languages. The document image serves as the input image and is the object processed by the visual encoder. The visual encoder refers to a neural network module used to analyze, extract, and encode visual information. It is responsible for receiving and processing the information in the document image, converting the input document image into a feature representation that the decoder can understand and process. The image features can include visual information such as image texture, edges, and character outlines, used to reflect the appearance of the text.

[0034] Specifically, in step S102, the connector links the visual encoder and the decoder, facilitating information interaction and fusion between different modalities (such as visual and text). The connector maps the image features determined by the visual encoder to the text space of the decoder, ensuring seamless conversion between visual and text features, thereby improving the overall cross-modal collaborative capability of the multimodal visual language model.

[0035] For example, the visual encoder may specifically be a high-resolution visual encoder, capable of supporting document images with a base resolution of 1024×1024.

[0036] Specifically, in step S103, the text instruction is a command issued to the document image, indicating the processing requirements for the document image. After a series of processing steps, the text instruction is converted into text encoding that the computer can understand and process. Further processing and analysis of the text encoding yields text features. Text features may include elements such as the semantics, syntactic structure, and position of the text within the document.

[0037] For example, a text instruction given to this document image might be "Identify the company names in the work experience section of the resume and bold the font." First, the positional characteristics of the "work experience" section within the document image must be determined. This involves locating this content by analyzing the text layout, paragraph spacing, and the coordinate positions of specific keywords (such as "work experience" or "work expertise") within the image. For instance, if the words "work experience" are found to be located in the area between 200-300 pixels horizontally and 400-450 pixels vertically, then this area can be used as a starting point to search downwards for related content. Next, the textual characteristics of the company names must be determined. Company names are generally nouns and are usually prominent information in the work experience section. Further identification can be achieved through features such as font, font size, and color. For example, if text in the work experience section is found to be in SimSun font, 12 points in size, black in color, and occupying a single line, then using a language model or pre-defined rules, this text can be identified as the company name, such as "XX Technology Co., Ltd." These characteristics regarding font, font size, color, position, and language attributes are the textual features determined for this text instruction. Based on these textual features, subsequent processing can be performed on the relevant content in the document image, such as displaying the company name in bold according to instructions.

[0038] For example, relevant personnel can input document images via electronic devices such as mobile phones and computers, and simultaneously input text commands for the document images.

[0039] Specifically, for step S104, a large language model can be used as the decoder; the large language model can be, but is not limited to, the Qwen-0.5B large language model.

[0040] The decoder receives image and text features and fuses them. This fusion process allows visual and textual information to complement each other, enabling the multimodal visual language model to gain a more comprehensive and in-depth understanding of the document image content. Next, the decoder generates a text sequence based on the fused features, completing the conversion from image to text, thus obtaining the OCR recognition result—that is, accurately converting the text in the document image into editable text.

[0041] Here is an example illustrating the OCR-based task processing method of this embodiment:

[0042] 1. Users upload a document image via electronic device and simultaneously enter text commands, such as "OCR:".

[0043] 2. The electronic device performs corresponding encoding operations on the received document images and text instructions respectively.

[0044] 3. The electronic device uses the features output by the visual encoder as a conditional prior, and the decoder generates a response sequence autoregressively based on the features.

[0045] 4. The electronic device will output the generated answer to the user interface and display it to the user.

[0046] After completing this task, the process returns to the first step, waiting for the user to input new document images and text commands for the next round of processing.

[0047] Understandably, traditional OCR models typically require multiple distinct modules. Each module must be designed, trained, and optimized independently before being integrated. However, in scenarios with uneven lighting, varied fonts, and complex backgrounds, these modules struggle to collaborate effectively at a global level, making it difficult for the model to achieve optimal overall recognition results. Furthermore, the multi-module design makes it difficult to adapt to different features. For instance, recognizing Chinese, English, and mathematical symbols often necessitates introducing specific expert models for each language or symbol, requiring the separate construction of corresponding models. This significantly increases task complexity and creates technical issues related to poor compatibility between different modules.

[0048] The inventors discovered that the root cause of the aforementioned problems lies in the fact that existing OCR systems do not employ end-to-end training. Simply put, an end-to-end trained model refers to a direct mapping from input to output. The model can automatically learn the complete transformation from the original input to the final output, without requiring manual design of intermediate processing steps or feature engineering. In this application, the end-to-end trained model refers to a multimodal visual language model. This model directly takes raw multimodal data such as images and text as input, and after a series of internal processing steps, directly outputs the final recognition, understanding, or generation results.

[0049] It is easy to see that, compared with related technologies, the solution provided in this application allows the encoder, connector, and decoder to jointly form a multimodal visual language model. The visual encoder converts document images into image features and simultaneously determines text features based on text instructions. The connector maps image features to the decoder's text space, enabling the visual encoder and decoder to work collaboratively. The decoder completes OCR recognition based on image and text features and outputs the recognition result. Since the multimodal visual language model of this application can be trained end-to-end, it eliminates the redundant expert modules in traditional OCR, integrating the visual encoder, connector, and decoder into a unified model, greatly simplifying the system architecture. This reduces module connection problems, avoids error accumulation, and achieves global collaboration. Moreover, the model directly uses raw multimodal data as input, processes it internally, and directly outputs the result, eliminating the need for manual design of intermediate steps and feature engineering. In summary, compared with traditional OCR, the model of this application has significant advantages. Simplified architecture, reduced module connections and error accumulation create favorable conditions for global collaboration, thereby improving the model's generalization ability and overall performance in complex scenarios. The model's recognition accuracy has been improved, and its compatibility and generalization ability in handling multiple tasks have been significantly enhanced, demonstrating excellent performance in different scenarios.

[0050] Second Embodiment

[0051] The second embodiment of this application relates to a task processing method based on [the first embodiment]. The second embodiment is an improvement upon the first embodiment, specifically in that it provides a concrete implementation of a visual encoder.

[0052] Optionally, in some embodiments, the visual encoder is specifically a visual encoder generated based on ViT-Det, which includes a global attention mechanism and a local attention mechanism.

[0053] The inventors discovered that in related technologies, if the document image is a high-resolution image, the Transformer visual encoder layer in the visual encoder uses a global attention mechanism to process segmented image patches. High-resolution images contain a large amount of pixel information; using a global attention mechanism requires calculating the attention weights of each location with all other locations, resulting in a quadratic increase in computational complexity. In this embodiment, some layers use a local attention mechanism instead of a global attention mechanism. In these layers, the local attention mechanism only focuses on the local region surrounding each location. This cleverly limits the attention range, significantly reducing computational complexity and enabling the model to run more efficiently with limited computational resources, thus improving training and inference speed.

[0054] By employing local attention mechanisms in some layers, the model's input image range is expanded. It can handle document images with a basic resolution (typically 448×448) as well as those with high resolutions (e.g., 1024×1024). For example, with high-resolution satellite images, a global attention mechanism might require processing relationships between millions of pixels, while with local attention, each pixel only needs to consider relationships with a few hundred surrounding pixels, significantly reducing computational cost.

[0055] Specifically, assume the input image (i.e., the document image) has a resolution of 1024×1024. After a series of processing steps, its feature dimension is obtained as 64×64×256. Next, this feature is processed through two convolutional layers with a stride of 2, resulting in a feature map of dimension 16×16×1024. Finally, a flattening operation (converting a multidimensional array into a one-dimensional array) is performed on this feature map, ultimately forming an image token of size 256×1024.

[0056] Optionally, in some embodiments, the visual encoder has 12 layers, each layer including an attention mechanism and a feedforward network; wherein, layers 2, 5, 8, and 11 use a global attention mechanism, and the other layers except layers 2, 5, 8, and 11 use a local attention mechanism.

[0057] One improvement of this scheme is that some layers use a local attention mechanism instead of a global attention mechanism. The advantage of doing so is that since the 12 layers of the visual encoder contain all parameters, reducing the overhead caused by using a global attention mechanism for all layers can reduce the amount of GPU memory used, improve computational efficiency, and better adapt to detection tasks.

[0058] Specifically, a global attention mechanism is used in layers 2, 5, 8, and 11 to fully utilize its ability to capture global information, while a local attention mechanism is used in other layers. This hybrid approach balances computational cost and feature extraction effectiveness, enabling the global attention mechanism to acquire key global information while the local attention mechanism can effectively handle local details of high-resolution images, thereby reducing overall computational cost.

[0059] It is not difficult to see that this application provides a specific implementation of a visual encoder. Because the use of a local attention mechanism can reduce computational load and memory consumption, the speed of model training and inference is significantly improved. With the help of the local attention mechanism, the model can process high-resolution document images more efficiently while maintaining a certain level of performance, thereby meeting real-time requirements.

[0060] Third Embodiment

[0061] The third embodiment of this application relates to an OCR-based task processing method. The third embodiment is an improvement on the first embodiment, specifically in that: in this embodiment, a linear layer MLP with a dimension of 1024×1024 is provided as a connector.

[0062] Optionally, in some embodiments, the connector is specifically a linear layer MLP with dimensions of 1024×1024.

[0063] The phrase "dimension 1024×1024" indicates the parameter scale of the linear layer MLP. The two 1024s here have two meanings: the first 1024 represents the input feature dimension, meaning the data input to this linear layer has 1024 features; the second 1024 represents the output feature dimension, meaning that after transformation by this linear layer, the data will be mapped to a 1024-dimensional feature space.

[0064] For example, if there is a data vector with 1024 features, inputting it into a linear MLP with dimensions 1024×1024 will result in a new vector with the same 1024 features after linear transformation. This new vector can then be used as input for subsequent model processing. In this embodiment, the linear MLP acts as a "connector," responsible for transforming the preceding data features and connecting them to subsequent processing stages.

[0065] It should be noted that this embodiment can also be an improvement based on the second embodiment.

[0066] It is not difficult to see that in this embodiment, a linear layer MLP with a dimension of 1024×1024 is provided as a connector. The function of this linear layer MLP is to map the image features output by the visual encoder to a text space suitable for the decoder to process. Through this design, seamless conversion between visual features and text features can be ensured, thereby improving the overall cross-modal collaborative capability of the model and enabling it to better fuse and process information from different modalities.

[0067] Fourth embodiment

[0068] The fourth embodiment of this application relates to a task processing method based on OCR. The fourth embodiment is an improvement upon the first embodiment, specifically in that it provides a concrete implementation of text instructions.

[0069] Optionally, in some embodiments, the text instruction includes at least one of the following: a first text instruction, a second text instruction, a third text instruction, and a fourth text instruction;

[0070] The first text instruction is used to recognize ordinary text;

[0071] The second text instruction is used to recognize formatted text;

[0072] The third text instruction is used to recognize text in ultra-high resolution images;

[0073] The fourth text instruction is used to identify text in documents with multiple pages.

[0074] Specifically, this embodiment provides at least four text prompts for controlling different OCR functions. The usage and applicable scenarios of these four text prompts are explained below:

[0075] For example, the first text instruction is used to recognize plain text. Plain text refers to plain text content without special formatting or layout, such as conventionally presented plain text or simple paragraphs. When processing document images containing this type of plain text, the user can input "OCR:" (assuming this is the form of the first text instruction) on the interface. After receiving the first text instruction, the system loaded on the electronic device can recognize the plain text in the document image and convert it into editable text for subsequent processing. For example, a scanned meeting record document with no special formatting or layout can be recognized and processed using the aforementioned first text instruction.

[0076] For example, the second text command is used to recognize formatted text. The formatted text can include rich formatting and layout information. For example, fonts may have different styles such as SimSun and KaiTi, font sizes can be large or small, text colors can be set, and bolding is also possible; in terms of layout, there are paragraph indentation (such as a first-line indent of 2 characters) and alignment methods (such as left alignment, center alignment, and right alignment). When a user needs to process a document image containing this type of formatted text, they can enter "OCR with format" (assuming this is the input form of the second text command) in the corresponding operation interface. After receiving the command, the system will recognize the formatted text in the document image. It can not only recognize the text content, but also retain the original font, font size, color, bolding, and other formatting features, as well as paragraph indentation, alignment, and other layout information, making it convenient to view, edit, or further analyze the document according to its original style. For example, a scanned image of a promotional poster with different fonts and font sizes, some text bolded, and paragraphs with indentation and center alignment can be completely obtained using the "OCR with format" command.

[0077] For example, the third text command is used to recognize text in ultra-high resolution images. Ultra-high resolution image text can include scanned high-resolution document images, large-format poster images, high-resolution drawings, etc., where the text and annotations require high precision. When a user needs to process such ultra-high resolution image documents, they enter "OCR with format upon the patch reference:" in the corresponding operation interface (assuming this is the input form of the third text command). For example, consider an ultra-high resolution scanned high-resolution image of an ancient book with densely packed text in a unique format. After entering this command, the system will recognize the text in the image, accurately extracting the text content and, based on local features in the image, restoring the text format as much as possible, such as text size, color, and font information, facilitating subsequent research, organization, and digital preservation of the ancient book content.

[0078] For example, the fourth text command is used to recognize text in multi-page documents, which can include multi-page PDF files, scanned multi-page documents, or books. When a user needs to process images of such multi-page documents, they enter "OCR with format across multi pages:" in the corresponding interface (assuming this is the input format for the fourth text command). For instance, given a scanned multi-page academic book, after the user enters this command, the system will recognize the text on each page sequentially. It can accurately extract the text content of each page and preserve the original text formatting, such as font, font size, color, and paragraph layout. This allows for the complete presentation of the book's original content and style during subsequent use, facilitating electronic reading, data organization, or knowledge retrieval.

[0079] It is not difficult to see that in the embodiments of this application, the text instructions include at least one of the following: a first text instruction, a second text instruction, a third text instruction, and a fourth text instruction. Considering that different types of document images have different characteristics and varying recognition difficulties, for example, formatted text requires not only text recognition but also additional processing of font, font size, color, paragraph layout, and other formatting information; multi-page text may involve special elements such as page number order, footers, and headers. Therefore, using specialized text instructions to process different types of document images can avoid content confusion and omissions, and also facilitates optimization based on the specific type of document image, thereby improving processing efficiency and meeting diverse needs.

[0080] Fifth Embodiment

[0081] The fifth embodiment of this application relates to an OCR-based task processing method. The fifth embodiment is an improvement upon the first embodiment, specifically in that: in this embodiment, the visual encoder, the connector, and the decoder form a multimodal visual language model, and a training method for the multimodal visual language model is provided.

[0082] Optionally, in some embodiments, the training method of the multimodal visual language model may include the following steps:

[0083] Step S201: Connect the visual encoder to a preset language model for pre-training to determine the target visual encoder;

[0084] Step S202: Connect the target visual encoder to the decoder, and train the multimodal visual language model based on the parameters of the target visual encoder and the decoder.

[0085] For example, the preset language model can be, but is not limited to, the OPT-125M language model, qwen1.5B, qwen7B, llama8B, etc. The decoder can be, but is not limited to, the Qwen-0.5B large language model.

[0086] For example, step S201 can be used to preheat the high-resolution visual encoder. Specifically, to improve training efficiency and incorporate OCR-related visual knowledge into the multimodal visual language model, in this embodiment, the visual encoder can be pre-trained on the OPT-125M language model. The dataset used during pre-training can include: document-information-dense data and OCR data from natural scenes. Using such a dataset for pre-training enables the visual encoder to possess certain basic capabilities before formally participating in the overall model training, thereby playing a better role in subsequent training stages.

[0087] Specifically, step S202 completes the end-to-end training of the overall model. Specifically, after warming up the high-resolution visual encoder, the warmed-up target visual encoder is connected to the decoder (taking the Qwen-0.5B large language model as an example) to construct a preliminary multimodal visual language model. Then, all parameters of this preliminary multimodal visual language model are released for comprehensive training to obtain the final multimodal visual language model. In this multimodal visual language model, the visual encoder is responsible for processing image data and extracting image feature information, while the Qwen-0.5B large language model acts as a decoder, receiving features from the visual encoder and combining them with its own language knowledge and abilities to perform subsequent operations such as text generation and question answering. This achieves the fusion of image and language information, enabling the model to process multimodal input data.

[0088] In this process of fully training the initial multimodal visual language model with all parameters released, all parameters include those of the visual encoder and decoder. This means that not only are the decoder parameters trained and updated, but also all parameters of the visual encoder are updated. Those skilled in the art will understand that some multimodal models in related technologies typically employ a "freeze the visual encoder" strategy, fixing the visual encoder parameters and not updating them during training. However, in this embodiment, by adjusting all parameters of the visual encoder and decoder, the multimodal visual language model can more accurately optimize image feature extraction and language generation according to specific task requirements, thereby improving the model's performance in multimodal tasks (such as image-based question answering and image description generation).

[0089] For example, the data used to train the multimodal visual language model is mixed data from multiple tasks. This mixed data may include, but is not limited to: OCR data of ordinary text, Markdown format data, formula data, table data, musical note recognition data, chemical formula recognition data, and data extracted from charts and graphs. By inputting such diverse data, the model's generalization ability can be significantly improved, enhancing its recognition performance for different types of symbols, thereby better addressing complex and diverse real-world application scenarios.

[0090] Optionally, in some embodiments, after obtaining the multimodal visual language model, the method may further include step S301: freezing the parameters of the target visual encoder and adjusting the parameters of the connector and the decoder within a preset range so that the multimodal visual language model conforms to a preset application scenario.

[0091] Specifically, the multimodal visual language model obtained through the above training already possesses a relatively powerful ability to recognize diverse optical symbols. However, in practical applications, if customized functions are required, the parameters of the visual encoder can be frozen, and the parameters of the linear layer MLP and the large language model can be fine-tuned instead, allowing the multimodal visual language model to adapt to the needs of specific tasks. For example, in scenarios such as sliding window cropping recognition of ultra-high resolution images, accurate recognition and processing can be achieved by fine-tuning the multimodal visual language model, allowing the model to better serve different downstream application scenarios.

[0092] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0093] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0094] Sixth Embodiment

[0095] Some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0096] The electronic device includes: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the steps of the methods provided in any one or more of the above embodiments. Figure 3 An exemplary structural diagram of the electronic device is disclosed. For example... Figure 3As shown, the electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0097] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103, and output device 1104 may be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0098] Input device 1103 can receive input numerical or character information and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0099] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0100] In this embodiment, a computer-readable medium stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0101] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0102] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0103] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0104] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0105] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, or similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that works with a processor to perform the various steps or functions.

[0107] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0108] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0109] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used only for distinguishing descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0110] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be considered exemplary and non-limiting.

Claims

1. An OCR-based task processing method, characterized by, The method is applied to a multi-modal visual language model, the multi-modal visual language model is obtained through end-to-end training, the multi-modal visual language model comprises a visual encoder, a connector and a decoder, and the method comprises the following steps: determining image features according to a document image and the visual encoder; mapping the image features to a text space of the decoder through the connector; wherein the connector is a linear layer MLP with a dimension of 1024x1024, used for fusion processing of information from different modalities; determining text features according to the decoder for a text instruction issued for the document image; wherein the text instruction comprises at least one of the following: a first text instruction, a second text instruction, a third text instruction and a fourth text instruction; the first text instruction is used for recognizing ordinary text; the second text instruction is used for recognizing text with format; the third text instruction is used for recognizing text of an ultra-high resolution image; and the fourth text instruction is used for recognizing text of a multi-page document; fusing the image features and the text features according to the decoder, so as to generate a text sequence through mutual complementation of visual information and text information, and determining an OCR recognition result. The training method of the multi-modal visual language model comprises the following steps: pre-training the visual encoder by connecting the visual encoder to a preset language model, to determine a target visual encoder; and connecting the target visual encoder to the decoder, and training the target visual encoder and the decoder according to parameters of the target visual encoder, to obtain the multi-modal visual language model.

2. The method of claim 1, wherein, The visual encoder is a visual encoder generated based on a ViT-Det, and the visual encoder comprises a global attention mechanism and a local attention mechanism.

3. The method of claim 2, wherein, The visual encoder has 12 layers, each layer comprising an attention mechanism and a feedforward network; wherein the 2nd layer, the 5th layer, the 8th layer and the 11th layer use the global attention mechanism, and the other layers use the local attention mechanism.

4. The method of claim 1, wherein, After obtaining the multi-modal visual language model, the method further comprises the following steps: freezing parameters of the target visual encoder, and adjusting parameters of the connector and parameters of the decoder within a preset range, so that the multi-modal visual language model meets a preset application scenario.

5. The method of claim 1, wherein, The preset language model comprises one of the following: an OPT-125M language model, qwen1.5B, qwen7B and llama8B.

6. An electronic device, comprising: The electronic device comprises: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method of any one of claims 1 to 5.

7. A computer readable medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the steps of the method of any one of claims 1 to 5.

8. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the steps of the method of any one of claims 1 to 5.

Images