Document image processing method and device

Abstract

The embodiment of the invention provides a document image processing method and device.The document image processing method comprises the steps that in the analysis process of a document image, on one hand, feature extraction is conducted on image blocks in the document image to obtain image features; on the other hand, attention calculation is carried out based on the virtual text features and the image features, attention image features are calculated according to obtained attention weights and the image features, and on the other hand, feature extraction is carried out on analysis task texts of the document images to obtain text features; the text features and the attention image features obtained in the two aspects are spliced to obtain spliced features, the spliced features are input into a large model, a processing module of the large model carries out image analysis processing on the attention image features according to the text features to obtain an image analysis result, and therefore analysis of the document image is achieved on the basis of multi-modal feature splicing.

Description

[0001] This patent application is a divisional application of the Chinese patent application with application number: CN202410961302.9, application date: July 17, 2024, and invention name “Document Image Processing Method and Device”. Technical Field

[0002] This document relates to the field of data processing technology, and in particular to a method and device for processing document images. Background Art

[0003] With the continuous development and promotion of computer technology and network technology, artificial intelligence-related technologies have become popular. Various artificial intelligence models represented by large language models (LLM) have been rapidly applied in related fields. Among them, an important area of ​​large language models is multimodal data processing. For example, it can simultaneously process multimodal data such as text, images, audio, etc. to obtain processing results. However, with the popularization of large language models, there is an increasing demand for processing diversity and flexibility of large language models. Summary of the invention

[0004] One or more embodiments of the present specification provide a method for processing a document image, comprising: extracting features from image blocks in a document image to obtain image features. Performing attention calculation on virtual text features generated based on the feature space of a large model and the image features, and calculating attention image features based on the obtained attention weights and the image features. Performing feature extraction on the parsing task text of the document image to obtain text features, and splicing the text features with the attention image features to obtain splicing features. Inputting the splicing features into the processing module of the large model to perform image parsing processing on the attention image features according to the text features to obtain an output image parsing result.

[0005] One or more embodiments of the present specification provide a document image processing device, including: a feature extraction module, configured to perform feature extraction on image blocks in a document image to obtain image features. An attention calculation module, configured to perform attention calculation on virtual text features generated based on the feature space of a large model and the image features, and calculate attention image features based on the obtained attention weights and the image features. A feature splicing module, configured to perform feature extraction on the parsing task text of the document image to obtain text features, and splice the text features with the attention image features to obtain splicing features. A parsing processing module, configured to input the splicing features into the processing module of the large model to perform image parsing processing on the attention image features according to the text features to obtain an output image parsing result.

[0006] One or more embodiments of the present specification provide a document image processing device, comprising: a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor: performs feature extraction on image blocks in the document image to obtain image features. Performs attention calculation on virtual text features generated based on the feature space of a large model and the image features, and calculates attention image features based on the obtained attention weights and the image features. Performs feature extraction on the parsing task text of the document image to obtain text features, and splices the text features with the attention image features to obtain spliced ​​features. The spliced ​​features are input into the processing module of the large model to perform image parsing processing on the attention image features according to the text features to obtain an output image parsing result.

[0007] One or more embodiments of the present specification provide a computer-readable storage medium for storing computer-executable instructions, which implement the following process when executed: extract features from image blocks in a document image to obtain image features. Perform attention calculation on virtual text features generated based on the feature space of a large model and the image features, and calculate attention image features based on the obtained attention weights and the image features. Perform feature extraction on the parsing task text of the document image to obtain text features, and splice the text features with the attention image features to obtain spliced ​​features. Input the spliced ​​features into the processing module of the large model to perform image parsing processing on the attention image features according to the text features to obtain an output image parsing result. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate one or more embodiments of the present specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor. Figure 1 A schematic diagram of an implementation environment of a document image processing method provided by one or more embodiments of this specification; Figure 2 A processing flow chart of a document image processing method provided in one or more embodiments of this specification; Figure 3 A processing flow chart of a method for processing a document image applied to a document material parsing scenario provided by one or more embodiments of this specification; Figure 4A schematic diagram of an embodiment of a document image processing device provided by one or more embodiments of this specification; Figure 5 A schematic diagram of the structure of a document image processing device provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0009] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0010] The document image processing method provided in one or more embodiments of this specification can be applied to the implementation environment of the large model. Figure 1 , the implementation environment at least includes: a large model consisting of an image encoder 101, a text encoder 102, a splicing module 103 and a processing module 104; The image encoder 101 is used to encode the text image in image dimension to obtain image features; The text encoder 102 is used to encode the input parsing task text in terms of text dimension to obtain text features; The splicing module 103 is used to perform attention calculation on the virtual text features and image features generated based on the feature space of the large model, and calculate the attention image features according to the obtained attention weights and image features, further perform feature extraction on the parsing task text of the document image to obtain text features, and splice the text features with the attention image features to obtain splicing features; The processing module 104 is used to perform image parsing processing on the attention image features contained in the splicing features according to the parsing task determined by the text features contained in the splicing features to obtain image parsing results, thereby realizing multimodal processing of text images and parsing task texts through a large model.

[0011] One or more embodiments of a document image processing method provided in this specification are as follows: Reference Figure 2 The document image processing method provided in this embodiment specifically includes steps S202 to S208.

[0012] Step S202: extract features from image blocks in the document image to obtain image features.

[0013] In specific implementation, during the parsing process of the document image, the image features of the text image are extracted by extracting features from feature blocks in the document image. Specifically, the image features can be obtained by extracting features from image blocks in the document image through an image encoder of a large model.

[0014] A big model refers to a pre-trained large language model. The big language model can adopt a Transformer architecture with a large number of parameters, such as a multimodal big model that can process and integrate multiple types of data (such as text, images, etc.), a conversational interaction big model in the field of conversational interaction, or a generative big model in the field of content generation.

[0015] In this embodiment, in the process of extracting features from image blocks in a text image, a sliding window method may be used to segment the document image to obtain image blocks with overlapping image regions, so as to enhance the contextual connection of image features. In an optional implementation manner provided in this embodiment, feature extraction is performed on image blocks in a document image to obtain image features, including: The document image is segmented by using a sliding window to obtain the image blocks; there are overlapping areas between the image blocks of the same document image; The image block is input into the image encoder of the large model to extract image features to obtain the image features.

[0016] The document image may be an image of a document material to be analyzed, such as a document image of a report document; the number of text images may be one or more. If there are multiple images, feature extraction is performed for each document image.

[0017] For example, in the scenario of parsing the document image of a physical examination report, for each document image of the physical examination report, a sliding window method is used to segment each document image of the physical examination report to obtain multiple image blocks of each document image, and the size of the overlapping area between each image block and the adjacent image block can be 1 / 2 of the image block size; after obtaining the multiple image blocks of each document image of the physical examination report, the multiple image blocks of each document image are input into the image encoder of the large model for image feature extraction to obtain image feature vectors of the multiple image blocks of each document image.

[0018] In practical applications, there may be different image resolutions in multiple document images collected for the same document material. In order to improve the efficiency of image feature extraction, the document image can also be converted to a resolution based on a preset standard image resolution, so as to convert the resolution of the text image to a preset standard image resolution. Specifically, in an optional implementation provided by this embodiment, before the document image is segmented using a sliding window to obtain image blocks, the resolution of the document image is adjusted according to a preset resolution, and the resolution of the adjusted document image is the same as the preset resolution. The preset resolution refers to a preset standard image resolution for image feature extraction.

[0019] Step S204, performing attention calculation on the virtual text features generated based on the feature space of the large model and the image features, and calculating the attention image features based on the obtained attention weights and the image features.

[0020] The virtual text features described in this embodiment refer to virtual text features generated in advance based on the feature space of the large model itself, and specifically can be virtual text features generated randomly based on the feature space. Here, the virtual text features are generated based on the feature space of the large model itself, so the large model can fully understand the virtual text features. Here, attention calculation is performed based on the virtual text features and image features, with the purpose of mapping the image features to the virtual text features with the help of the virtual text features, that is, mapping the image features to the feature space of the large model itself, so that the image features can be understood by the large model.

[0021] In specific implementation, in the process of performing attention calculation on the virtual text features and image features generated based on the feature space of the large model, a cross-attention mechanism can be used to perform cross-attention calculation on the virtual text features and image features, so as to map the image features to the feature space of the large model. In an optional implementation provided in this embodiment, the attention calculation on the virtual text features and image features generated based on the feature space of the large model includes: performing cross-attention calculation on the virtual text features as queries and the image features as keys and values, and using the obtained cross-attention weights as the attention weights.

[0022] Specifically, in the process of performing cross attention calculation using virtual text features as queries and image features as keys and values, the following optional implementation methods are used to perform cross attention calculation: Using the virtual text feature as a query and the image feature as a key and a value, calculating the similarity between the text element included in the virtual text feature and the image element included in the image feature to obtain a similarity matrix; The similarity matrix is ​​normalized to obtain the cross attention weight.

[0023] For example, based on the virtual text feature vector randomly generated in the feature space of the large model and the image feature vectors of multiple image blocks of each document image, in the process of cross-attention calculation, the virtual text feature vector is used as the query (Query) and the image feature vector is used as the key (Key) and the value (Value), and the similarity between the text vector element at each position in the virtual text feature vector and the image vector element at each position in the image feature vector is calculated to obtain a similarity matrix, and the similarity matrix is ​​normalized to obtain a cross-attention weight matrix representing the similarity between the text vector elements of the text feature vector and the image vector elements of the image feature vector.

[0024] Based on the attention weights obtained by performing attention calculation on the virtual text features and image features generated based on the feature space of the large model, the attention image features are calculated based on the attention weights and image features to obtain the image features influenced by the attention weights, that is, the image features mapped to the feature space of the large model. Specifically, in an optional implementation provided by this embodiment, the attention image features are calculated based on the obtained attention weights and image features, including: weighted processing of the image features using the attention weights to obtain the attention image features.

[0025] Step S206, extracting features of the parsing task text of the document image to obtain text features, and concatenating the text features with the attention image features to obtain concatenated features.

[0026] The document image parsing task text described in this embodiment refers to a text composed of task description words for performing corresponding task processing on the document image, such as the parsing task text for parsing the document image.

[0027] In a specific implementation, in the process of extracting features from the parsing task text of the document image to obtain text features, the large model can be used to extract features from the parsing task text, thereby mapping the parsing task text to the feature space of the large model. Specifically, in an optional implementation provided by this embodiment, extracting features from the parsing task text of the document image to obtain text features includes: inputting the parsing task text into the text encoder of the large model to extract features and obtain the text features.

[0028] In the above, the attention image features mapped to the feature space of the large model are obtained through attention calculation, and the parsing task text is mapped to the feature space of the large model by performing feature extraction on the parsing task text through the large model. Based on this, the text features and the attention image features are spliced ​​to obtain the splicing features, so as to obtain the splicing features mapped to the feature space of the large model, that is, to obtain the whole of the attention image features and text features that can be understood by the large model. Based on the splicing features, the large model can understand that the task to be processed currently is the parsing task, and can understand that the object of the current parsing task is the attention image features.

[0029] Step S208, inputting the splicing features into the processing module of the large model to perform image analysis processing on the attention image features according to the text features to obtain an output image analysis result.

[0030] As described above, on the basis of obtaining the attention image features and text features mapped to the feature space of the large model, by inputting the splicing features into the processing module of the large model, the processing module of the large model understands that the current task to be processed is the parsing task through the text features contained in the splicing features, and understands that the object of the current parsing task is the attention image features through the attention image features contained in the splicing features. Then, the processing module of the large model performs image parsing on the attention image features according to the text features to obtain the output image parsing results.

[0031] In an optional implementation manner provided by this embodiment, performing image parsing processing on the attention image feature according to the text feature includes: A parsing task type of obtaining the attention image feature by performing semantic recognition based on the text feature; Perform image analysis processing on the attention image features according to the analysis task type to obtain the image analysis result.

[0032] It should be noted that the document image processing method provided in this embodiment can also be implemented by a large model, that is, the document image processing method provided in this embodiment is applied to the large model. In this case, the input of the large model includes the document image and the parsing task text, and the output includes the image parsing result. Specifically, the large model includes an image encoder, a text encoder, a splicing module and the processing module; Optionally, the process of extracting features from image blocks in the document image to obtain image features is implemented by an image encoder; Optionally, the process of extracting features from the parsing task text of the document image to obtain text features is implemented by a text encoder; Optionally, the virtual text features generated based on the feature space of the large model are used to perform attention calculation with the image features, and the attention image features are calculated based on the obtained attention weights and image features, and the text features and the attention image features are spliced ​​to obtain the spliced ​​features, which are implemented through a splicing module.

[0033] In addition, the large model may also be provided with an image segmentation module to adjust the resolution of the document image and segment the document image into image blocks. Optionally, the image segmentation module is used to adjust the resolution of the document image according to a preset resolution, and use a sliding window to segment the adjusted document image to obtain image blocks.

[0034] The following describes the document image processing process in combination with the model architecture of the large model.

[0035] The input of the large model includes document images and parsing task text; On the one hand, for the document image of the input large model, the resolution of the document image is adjusted according to the preset resolution through the image segmentation module, and the adjusted document image is segmented using a sliding window to obtain image blocks of the document image, and the obtained image blocks are input into the image encoder; The image encoder extracts image features from the input image blocks to obtain image features of the input image blocks, and inputs the obtained image features into the splicing module; On the other hand, for the parsing task text input into the large model, the text encoder is used to extract text features from the parsing task text, and the extracted text features are input into the splicing module; After receiving the image features input by the image encoder, the splicing module performs cross-attention calculation on the virtual text features and image features generated based on the feature space of the large model, and calculates the attention image features based on the obtained cross-attention weights and image features. Further, the attention image features are spliced ​​with the text features input by the text encoder to obtain the splicing features, thereby realizing the multimodal feature fusion of text features and image features on the basis of mapping the document image and the parsing task text to the same feature space (i.e., the feature space of the large model); Finally, after the splicing model inputs the splicing features obtained by multimodal feature fusion into the processing module, the processing model understands the text features contained in the splicing features and understands that the current task to be processed is the parsing task, and understands that the object of the current parsing task is the attention image feature through the attention image features contained in the splicing features. The processing module performs image parsing on the attention image features according to the text features, and finally outputs the image parsing results, thereby realizing end-to-end document image processing.

[0036] In summary, the document image processing method provided in the present embodiment, in the process of parsing the text image, on the one hand, performs feature extraction on the image blocks in the document image to obtain image features, and performs attention calculation on the virtual text features and image features generated based on the feature space of the large model, and calculates the attention image features according to the obtained attention weights and the image features, thereby mapping the image features to the feature space of the large model with the help of the virtual text features, and on the other hand, performs feature extraction on the parsing task text of the document image to obtain text features, and on the basis of obtaining the text features and the attention image features, splices the text features with the attention image features to obtain splicing features, thereby realizing multimodal feature fusion of the text features and the image features in the feature space of the large model, and thereafter, inputs the splicing features into the processing module of the large model to perform image parsing processing on the attention image features according to the text features, and obtains the output image parsing results, thereby realizing multimodal feature fusion of the text features and the image features in the feature space of the large model, and performing image parsing processing on the attention image features contained in the splicing features according to the feature text features contained in the splicing features, thereby improving the processing efficiency and accuracy of the document image parsing processing.

[0037] The following takes the application of a document image processing method provided by this embodiment in a document material parsing scenario as an example, combined with Figure 3 , the document image processing method provided in this embodiment is further described, see Figure 3 , a document image processing method applied to a document material parsing scenario specifically includes the following steps.

[0038] Step S302: adjusting the resolution of the document image of the document material according to the preset resolution, wherein the resolution of the adjusted document image is the same as the preset resolution.

[0039] Step S304: segment the adjusted document image using a sliding window to obtain image blocks of the document image.

[0040] Step S306, extracting image features from the image block using the image encoder of the large model to obtain image features.

[0041] Step S308, through the stitching module of the large model, the attention calculation is performed on the virtual text features and image features generated based on the feature space of the large model, and the attention image features are calculated according to the obtained attention weights and image features.

[0042] Step S310, extracting features of the parsing task text of the document image through the text encoder of the large model to obtain text features.

[0043] Step S312, the text features and the attention image features are spliced ​​together through the splicing module of the large model to obtain splicing features.

[0044] Step S314, performing image analysis processing on the attention image features contained in the splicing features according to the text features contained in the splicing features through the processing module of the large model to obtain an image analysis result.

[0045] It should be noted that any one of steps S302 to S314 or any combination of multiple steps can be replaced by the corresponding technical means provided in the above steps S202 to S208 according to the needs of implementation and deployment, and will not be described one by one here.

[0046] An embodiment of a document image processing device provided in this specification is as follows: In the above-mentioned embodiment, a method for processing a document image is provided, and correspondingly, a device for processing a document image is also provided, which will be described below in conjunction with the accompanying drawings.

[0047] Reference Figure 4 , which shows a schematic diagram of an embodiment of a document image processing device provided in this embodiment.

[0048] Since the device embodiment corresponds to the method embodiment, the description is relatively simple, and the relevant parts can refer to the corresponding description of the method embodiment provided above. The device embodiment described below is only illustrative.

[0049] This embodiment provides a document image processing device, the device comprising: The feature extraction module 402 is configured to extract features from image blocks in the document image to obtain image features; An attention calculation module 404 is configured to perform attention calculation on the virtual text features generated based on the feature space of the large model and the image features, and calculate an attention image feature based on the obtained attention weight and the image features; A feature splicing module 406 is configured to extract features of the parsing task text of the document image to obtain text features, and splice the text features with the attention image features to obtain splicing features; The parsing processing module 408 is configured to input the splicing features into the processing module of the large model to perform image parsing processing on the attention image features according to the text features to obtain an output image parsing result.

[0050] An embodiment of a document image processing device provided in this specification is as follows: Corresponding to the document image processing method described above, based on the same technical concept, one or more embodiments of this specification further provide a document image processing device, which is used to execute the document image processing method provided above. Figure 5 A schematic diagram of the structure of a document image processing device provided in one or more embodiments of this specification.

[0051] This embodiment provides a document image processing device, including: like Figure 5 As shown, the document image processing device may have relatively large differences due to different configurations or performances, and may include one or more processors 501 and memory 502, and the memory 502 may store one or more storage applications or data. Among them, the memory 502 may be a temporary storage or a permanent storage. The application stored in the memory 502 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the document image processing device. Furthermore, the processor 501 may be configured to communicate with the memory 502, and execute a series of computer executable instructions in the memory 502 on the document image processing device. The document image processing device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input / output interfaces 505, one or more keyboards 506, etc.

[0052] In a specific embodiment, the document image processing device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the document image processing device, and the one or more programs are configured to be executed by one or more processors, including computer executable instructions for performing the following: Extract features from image blocks in the document image to obtain image features; Performing attention calculation on the virtual text features generated based on the feature space of the large model and the image features, and calculating the attention image features based on the obtained attention weights and the image features; Extracting features of the parsed task text of the document image to obtain text features, and splicing the text features with the attention image features to obtain spliced ​​features; The splicing features are input into the processing module of the large model to perform image analysis processing on the attention image features according to the text features to obtain an output image analysis result.

[0053] An embodiment of a computer-readable storage medium provided in this specification is as follows: Corresponding to the document image processing method described above, based on the same technical concept, one or more embodiments of this specification also provide a computer-readable storage medium.

[0054] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, and the computer-executable instructions implement the following process when executed: Extract features from image blocks in the document image to obtain image features; Performing attention calculation on the virtual text features generated based on the feature space of the large model and the image features, and calculating the attention image features based on the obtained attention weights and the image features; Extracting features of the parsed task text of the document image to obtain text features, and splicing the text features with the attention image features to obtain spliced ​​features; The splicing features are input into the processing module of the large model to perform image analysis processing on the attention image features according to the text features to obtain an output image analysis result.

[0055] It should be noted that the embodiment of a computer-readable storage medium in this specification and the embodiment of a document image processing method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the aforementioned corresponding method, and the repeated parts will not be repeated.

[0056] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. For example, the device embodiment, equipment embodiment and computer-readable storage medium embodiment are similar to the method embodiment, so the description is relatively simple. To read the relevant contents in the device embodiment, equipment embodiment and computer-readable storage medium embodiment, please refer to the partial description of the method embodiment.

[0057] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0058] In the 1930s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to circuit structures such as diodes, transistors, switches, etc.) or software improvements (improvements to method flows). However, with the development of technology, many of today's method flow improvements can be regarded as direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and produce dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0059] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0060] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0061] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0062] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present specification may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0066] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0067] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0068] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0069] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "includes at least one ..." do not exclude the presence of other identical elements in the process, method, commodity or device including the elements.

[0070] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0071] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A method for processing a document image, comprising: Extracting features from image blocks in the document image to obtain image features; Performing attention calculation on the virtual text features generated based on the feature space of the large language model and the image features, and calculating the attention image features according to the obtained attention weights and the image features; Extracting features of the parsing task text of the document image by using the large language model to obtain text features, and concatenating the text features with the attention image features to obtain concatenated features; The concatenated features are input into the processing module of the large language model to perform image parsing processing on the attention image features according to the text features to obtain an image parsing result.

2. The document image processing method according to claim 1, wherein the step of extracting features from image blocks in the document image to obtain image features comprises: Using a sliding window to segment the document image to obtain the image blocks; The image block is input into the image encoder of the large language model to extract image features to obtain the image features.

3. The method for processing document images according to claim 2, before the sub-step of segmenting the document image using a sliding window to obtain the image blocks is performed, further comprising: The resolution of the document image is adjusted according to a preset resolution, and the resolution of the adjusted document image is the same as the preset resolution.

4. The document image processing method according to claim 1, wherein the step of performing attention calculation on the virtual text features generated based on the feature space of the large language model and the image features comprises: The virtual text feature is used as a query and the image feature is used as a key and a value to perform cross-attention calculation, and the obtained cross-attention weight is used as the attention weight.

5. The method for processing document images according to claim 4, wherein the step of performing cross attention calculation using the virtual text feature as a query and the image feature as a key and a value, and using the obtained cross attention weight as the attention weight, comprises: Using the virtual text feature as a query and the image feature as a key and a value, calculating the similarity between the text element included in the virtual text feature and the image element included in the image feature to obtain a similarity matrix; The similarity matrix is ​​normalized to obtain the attention weight.

6. The method for processing document images according to claim 1, wherein the step of calculating the attention image feature based on the obtained attention weight and the image feature comprises: The image features are weighted by using the attention weights to obtain the attention image features.

7. The document image processing method according to claim 1, wherein the step of extracting features from the parsed task text of the document image to obtain text features comprises: The parsing task text is input into the text encoder of the large language model for feature extraction to obtain the text features.

8. The document image processing method according to claim 1, wherein the image parsing process of the attention image features according to the text features comprises: Performing semantic recognition based on text features included in the concatenated features to obtain a parsing task type; Image parsing processing is performed on the attention image features included in the splicing features according to the parsing task type to obtain the image parsing result.

9. The document image processing method according to claim 1, applied to the large language model, the input of the large language model includes the document image and the parsing task text, and the output includes the image parsing result; The large language model includes an image encoder, a text encoder, a splicing module and the processing module; in, The operation of extracting features from the parsing task text of the document image to obtain text features is implemented by the image encoder; The step of performing attention calculation on the virtual text features generated based on the feature space of the large language model and the image features, and calculating the attention image features according to the obtained attention weights and the image features, and the operation of splicing the text features with the attention image features to obtain the splicing features are implemented through the splicing module.

10. A document image processing device, comprising: A feature extraction module is configured to extract features from image blocks in the document image to obtain image features; an attention calculation module, configured to perform attention calculation on the virtual text features generated based on the feature space of the large language model and the image features, and calculate an attention image feature according to the obtained attention weight and the image features; A feature splicing module is configured to extract features of the parsing task text of the document image through the large language model to obtain text features, and splice the text features with the attention image features to obtain splicing features; The parsing processing module is configured to input the concatenated features into the processing module of the large language model to perform image parsing processing on the attention image features according to the text features to obtain an image parsing result.

11. A document image processing device, comprising: processor; and a memory configured to store computer executable instructions that, when executed, cause the processor to: Extracting features from image blocks in the document image to obtain image features; Performing attention calculation on the virtual text features generated based on the feature space of the large language model and the image features, and calculating the attention image features according to the obtained attention weights and the image features; Extracting features of the parsing task text of the document image by using the large language model to obtain text features, and concatenating the text features with the attention image features to obtain concatenated features; The concatenated features are input into the processing module of the large language model to perform image parsing processing on the attention image features according to the text features to obtain an image parsing result.

12. A computer-readable storage medium for storing computer-executable instructions, wherein the computer-executable instructions implement the steps of the method of claim 1 when executed.

Images