Document identification method and device, related equipment and computer program product

Abstract

The invention discloses a document recognition method and device, related equipment and a computer program product, a to-be-recognized text can be split into multiple pages of document images, the visual features of each page of document image are extracted respectively, the visual features of the multiple pages of document images can be sent into a large model at the same time, and the recognition efficiency is improved. Meanwhile, text features of a first document recognition instruction are extracted and sent to the large model, the first document recognition instruction is used for indicating the large model to output the document recognition result of the document image, and therefore the recognition result of the multi-page document can be output end to end through the large model. The large model has stronger modeling capability, and meanwhile, the visual features of the multi-page document images are sent into the large model, so that context information among different pages of document images can be reserved, context semantics of a text can be understood, and a hierarchical structure and a content relationship of the document can be analyzed more accurately; therefore, the accuracy of text recognition and layout understanding is improved, and the accuracy of the recognition result of the multi-page document is improved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and more specifically, to a document recognition method, apparatus, related equipment and computer program product. Background Art

[0002] Document recognition tasks include one or both of the text detection and positioning tasks and the layout analysis tasks. The text detection and positioning tasks are to identify the content of the text lines contained in the document and the position coordinates of the detection boxes corresponding to the text lines. The layout analysis tasks are to identify the types of layout elements contained in the document and the position coordinates of the detection boxes. Examples of layout element types include: text boxes, image areas, tables, titles, headers, footers, and other elements.

[0003] The current document recognition task mainly focuses on single-page documents, which are recognized through the OCR (Optical Character Recognition) model. For multi-page documents, the processing method is also to first divide them into several single-page documents, and then perform document recognition on each page of the document. After that, the recognition results of each page of the document are spliced ​​together to obtain the recognition results of the multi-page document. This multi-page document recognition method seriously destroys the contextual semantic information of the multi-page document, resulting in inaccurate document recognition results. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a document recognition method, apparatus, related equipment and computer program product to improve the accuracy of recognition results of multi-page documents. The specific solution is as follows:

[0005] In a first aspect, a document recognition method is provided, comprising:

[0006] Cut the document to be recognized into document images of more than one page;

[0007] Extract visual features of each page of document image through visual encoding module;

[0008] Extracting text features of a set first document recognition instruction through a text encoding module, wherein the first document recognition instruction is used to instruct the large model to output a document recognition result of a document image;

[0009] The visual features of each page of the document image and the text features of the first document recognition instruction are fused, and the fused features are sent to the feature processing layer of the configured large model to obtain the document recognition results output by the large model.

[0010] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the first document recognition instruction includes: a text recognition instruction, which is used to instruct the large model to output a text line recognition result of the document image;

[0011] Based on this, the document recognition results output by the large model include: text line recognition results.

[0012] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the first document recognition instruction further includes: a layout analysis instruction, which is used to instruct the large model to output layout element information of the document image;

[0013] Based on this, the document recognition result output by the large model also includes: layout element information.

[0014] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0015] Extracting text features of a set second document recognition instruction through a text encoding module, wherein the second document recognition instruction is used to instruct the large model to output layout element information of a document image;

[0016] The visual features of each page document image and the text features of the second document recognition instruction are fused, and the fused features are sent to the feature processing layer of the large model to obtain the layout element information output by the large model.

[0017] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the visual encoding module includes a visual encoder and a sampler, and the process of extracting visual features of each page of the document image through the visual encoding module includes:

[0018] Extracting initial visual features of each page of document image through visual encoder;

[0019] The sampler extracts visual features of m visual unit tokens from the initial visual features of each page of the document image, where m is a set constant.

[0020] In a possible design, in another implementation of the first aspect of the embodiment of the present application, before fusing the visual features of the document images of each page with the text features of the first document recognition instruction, the method further includes:

[0021] Determine the visual position coding corresponding to each page of the document image, wherein the visual position coding includes an intra-page position coding, and the intra-page position coding is composed of a plurality of learnable position coding vectors, wherein the plurality of learnable position coding vectors are updated and fixed during the large model training process, and the number of learnable position coding vectors is the same as the number of visual unit tokens corresponding to the visual features of each page of the document image;

[0022] For each page of the document image, the visual features of the current page of the document image are fused with the visual position code to obtain updated visual features of the current page of the document image.

[0023] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the visual position code also includes an inter-page position code, and the inter-page position code is composed of an absolute position code.

[0024] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the training process of the large model includes:

[0025] Obtain training document samples and corresponding document recognition result labels;

[0026] Cutting the training document sample into more than one page of training document images;

[0027] Extracting visual features of m tokens of each page of training document image through a visual encoding module, and extracting text features of a set first document recognition instruction through a text encoding module;

[0028] For each page of training document image, add the visual features of the m tokens of the current page training document image and the m learnable position encoding vectors element by element at the corresponding positions, and add the visual features of the m tokens after addition to the inter-page position encoding of the current page training document image to obtain the updated visual features of the current page training document image;

[0029] Fusing the updated visual features of each page of training document images with the text features of the first document recognition instruction, and sending the fused features to the feature processing layer of the large model to obtain the document recognition results output by the large model;

[0030] Taking the document recognition result approaching the document recognition result label as the training goal, the network parameters of the large model and m learnable position encoding vectors are updated until the training end condition is reached.

[0031] In a second aspect, a document recognition device is provided, comprising:

[0032] A document segmentation unit, used for segmenting the document to be identified into document images of more than one page;

[0033] A visual feature extraction unit, used for extracting visual features of each page of the document image through a visual encoding module;

[0034] A text feature extraction unit, used to extract text features of a set first document recognition instruction through a text encoding module, wherein the first document recognition instruction is used to instruct the large model to output a document recognition result of a document image;

[0035] The large model inference unit is used to fuse the visual features of each page of the document image with the text features of the first document recognition instruction, and send the fused features to the feature processing layer of the configured large model to obtain the document recognition results output by the large model.

[0036] In a third aspect, an electronic device is provided, comprising: a memory and a processor;

[0037] The memory is used to store programs;

[0038] The processor is used to execute the program to implement the document recognition method described in any one of the first aspects of the present application.

[0039] In a fourth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the document recognition method described in any one of the first aspects of the present application is implemented.

[0040] In a fifth aspect, a computer program product is provided, including a computer program, which, when executed by a processor, implements the document recognition method described in any one of the first aspects of the present application.

[0041] By means of the above technical solution, the present application proposes a document recognition method based on a large model, in which the text to be recognized can be split into multiple pages of document images, and the visual features of each page of the document image can be extracted respectively. The visual features of the multiple pages of document images can be simultaneously fed into the large model, and the text features of the first document recognition instruction can be simultaneously extracted and also fed into the large model, so that the recognition results of the multiple pages of documents can be output end-to-end through the large model. Since the large model has a stronger modeling capability, the contextual information between the document images of different pages can be retained by feeding the visual features of the multiple pages of document images into the large model at the same time, so that the contextual semantics of the text can be understood, and the hierarchical structure and content relationship of the document can be analyzed more accurately, thereby improving the accuracy of text recognition and layout understanding, and improving the accuracy of the recognition results of multiple pages of documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0043] Figure 1 A schematic diagram of an implementation system architecture of the document recognition method provided in an embodiment of the present application;

[0044] Figure 2A schematic diagram of a terminal structure provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of a server structure provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of a process flow of a document recognition method provided in an embodiment of the present application;

[0047] Figure 5 Another schematic diagram of the process of document recognition provided by the embodiment of the present application;

[0048] Figure 6 An example of a schematic diagram of the overall framework of document recognition is shown;

[0049] Figure 7 A schematic diagram of the structure of a document recognition device provided in an embodiment of the present application;

[0050] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] Before introducing this application solution, the relevant concepts involved in this article are first explained:

[0052] prompt: Instructions. When interacting with AI (such as an artificial intelligence model), you need to send instructions to AI. It can be a text description, such as "Please recommend me a pop song" when you interact with AI, or it can be a parameter description in a certain format. For example, if you want AI to draw in a certain format, you need to describe the relevant drawing parameters.

[0053] Large models: In the field of artificial intelligence, large models usually refer to large-scale pre-trained models. Such models are called "large" because they can be pre-trained on a large amount of data and can be transferred to a variety of downstream tasks. Their full English name is Large Pre-Trained Models or Large-Scale Pre-Training Models. Large models are characterized by their large scale and contain billions or even more parameters, which help them learn complex patterns in the data. The emerging capabilities of large models include but are not limited to: contextual learning, instruction following, code generation, step-by-step reasoning capabilities, etc. Large models can include large language models (LLM) and multimodal large models. Large language models are mainly used to process text modal data. Multimodal large models further integrate multimodal capabilities on the basis of large language models and can process information in multiple modalities, such as images, text, audio, etc.

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] This application provides a document recognition method that can be applied to Figure 1 The system architecture shown in FIG. 1 may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 A server is included as an example for explanation).

[0056] The terminal 100 or the server 200 can be used alone to execute the document recognition method provided in the embodiment of the present application. In addition, the terminal 100 and the server 200 can also be used together to execute the document recognition method provided in the embodiment of the present application.

[0057] Next describe Figure 1 The product form of the mid-terminal 100;

[0058] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a teaching screen, a learning machine, a wearable device, a vehicle-mounted device, a conference terminal, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0059] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0060] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160, a speaker 161, a microphone 162, an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0061] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0062] Among them, other input devices 132 can receive input data and so on.

[0063] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 may be used to display various interactive interfaces, processing results, etc. in the document recognition method.

[0064] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0065] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0066] Among them, the memory 120 can be used to store software codes related to the document recognition method, the processor 170 can execute the steps of the document recognition method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve corresponding functions.

[0067] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0068] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive the processing result sent by the server 200. Exemplarily, the RF unit 110 sends the document to be recognized to the server 200, and the server 200 recognizes the text to be recognized and returns the document recognition result to the terminal 100.

[0069] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0070] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0071] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0072] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0073] Next describe Figure 1 The product form of the server 200;

[0074] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0075] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0076] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0077] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0078] The memory 204 may be used to store software codes related to the document recognition method, and the processor 202 may execute the steps of the document recognition method and may also schedule other units to implement corresponding functions.

[0079] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0080] With the advancement of information technology and the promotion of digital transformation, a large number of paper documents need to be converted into electronic formats for more convenient storage, management, retrieval and sharing. Especially in enterprises, government agencies, academia and the legal field, the growing demand for large-scale document processing and information management has prompted the development of multi-page document recognition and parsing technology. Multi-page documents usually have more complex structures and rich semantic information, the page content is divided into multiple parts, and there are more extensive contextual information and other issues, while single-page documents do not have these problems.

[0081] At present, multi-page documents usually contain multiple pages, each of which may have different layouts and structures, such as chapters, paragraphs, tables, etc., and may even have cross-references and repeated content. The existing multi-page document recognition scheme divides the document into several pictures according to the rules, and sends them to the single-page document recognition and parsing engine respectively. A large amount of contextual information is lost for the elements such as the tables, paragraphs, and pictures that are divided, which seriously reduces the recognition accuracy of the divided elements. At the same time, the contextual information of a single-page document is limited, and the content can only be parsed within the current page, making it difficult to accurately understand the semantics and logic of the document as a whole. For example, if a table is divided into two parts, the recognition and parsing of the second half of the table lacks header information, and there is a lack of obvious semantic guidance for the reconstruction of the table rows and columns. At the same time, the existing scheme not only involves more human-designed modules and rules, but also has a long process and is not an end-to-end solution, which greatly increases the time cost of actual research and development.

[0082] Taking into account the defects of the above-mentioned prior art, this application proposes a document recognition solution based on a large model LLM, which can use its powerful language modeling capabilities and combine contextual information to more accurately recognize characters. In addition, LLM performs well in processing multilingual texts and can learn the commonalities and differences between different languages ​​through pre-trained language models. This ability makes LLM important in international applications and multilingual document processing, especially in multinational companies, institutions and international cooperation projects.

[0083] The document recognition solution based on LLM in this application is more suitable for the recognition of long documents (including multi-page documents). By simultaneously feeding the visual features of multi-page document images into LLM and outputting the recognition results of multi-page documents end-to-end, and utilizing more extensive contextual information, such as the chapter and paragraph structure of the entire document, the content of each page can be understood more accurately, and the context and reference relationships in the document can be better handled. Two types of task prompt instructions taskprompt can be designed in this application: text recognition instructions and layout analysis instructions. The above-mentioned task prompt instructions control LLM to output the text line recognition results and layout element information of multi-page documents respectively, so as to meet all actual document recognition needs as much as possible. The solution of this application can provide more comprehensive, efficient and accurate information extraction and processing capabilities when processing complex, multi-chapter and large-scale documents, and has important advantages and values ​​in the application of modern document management and informatization.

[0084] Of course, the document recognition solution of the present application is more advantageous than the prior art when processing long documents. This does not mean that the document recognition solution of the present application is only applicable to long documents. For short documents, such as documents containing only one page, the solution of the present application can still be applied.

[0085] The present invention provides a document recognition method, which is applied to Figure 1 Taking the computer device in the example as an example, the computer device can be Figure 1 The terminal 100 or the system consisting of the terminal 100 and the server 200. Figure 4 , the document recognition method specifically includes the following steps:

[0086] Step S100: dividing the document to be recognized into document images of more than one page.

[0087] Specifically, in this embodiment, the document to be identified can be a long document, that is, a document containing multiple pages, which can be segmented according to the set rules to obtain multiple pages of documents, and converted into image format to obtain multiple pages of document images. Of course, the document to be identified can also be a short document, such as a document containing only one page, in which case a single page of document can be obtained after segmentation, and converted into image format to obtain a single page of document images.

[0088] Step S110: extracting visual features of each page of the document image through a visual encoding module.

[0089] The visual encoding module may include a visual encoder, which may select a model structure such as a convolutional neural network (CNN) or a vision transformer according to the computing power of the actual application scenario. The visual encoder may extract the visual features corresponding to each page of the document image.

[0090] In some possible implementations, in order to further effectively reduce the amount of calculation, the visual encoding module may further include a sampler. For the initial visual features of each page of the document image extracted by the visual encoder, the initial visual features of each page of the document image may be further sampled by the sampler to extract visual features of m tokens (referring to the smallest visual unit for image processing). Wherein, m is a set constant, and examples of values ​​are constants such as 256 or 512.

[0091] Taking n document images obtained after segmentation of the document to be recognized as an example, the visual features of n×m tokens can be extracted in this step.

[0092] By cascading a sampler after the visual encoder, the document image can be efficiently compressed, the number of visual features extracted by the visual encoder can be reduced, and the amount of calculation can be reduced.

[0093] This embodiment further provides an optional structure of a sampler, which can be composed of a Cross Attention layer. The forward calculation process of the sampler is shown in the following formula:

[0094]

[0095] Among them, Q represents m learnable query vectors, K and V are the output results of the visual encoder, and the Attention result sequence calculated according to the above formula is used as the visual feature of each page of the document image, with a length of m.

[0096] Among them, the m learnable query vectors can be updated during the large model training phase and fixed at the end of training.

[0097] Step S120: extracting text features of a set first document recognition instruction through a text encoding module, where the first document recognition instruction is used to instruct the large model to output a document recognition result of a document image.

[0098] Specifically, according to the document recognition task of the present application, a first document recognition instruction can be pre-set to instruct the large model to output the document recognition result of the document image.

[0099] In some possible implementations, the first document recognition instruction may include a text recognition instruction for instructing the large model to output a text line recognition result of the document image, wherein the text line recognition result may include text line content and a coordinate frame.

[0100] Optionally, the first document recognition instruction may further include a layout analysis instruction for indicating the layout element information of the document image output by the large model, wherein the layout element information may include the layout element type and the coordinate frame.

[0101] From the above, it can be seen that the first document recognition instruction in the present application can simultaneously include two task prompt instructions, namely, a text recognition instruction and a layout analysis instruction, thereby instructing the large model to output the document recognition results and layout element information of the document image.

[0102] An optional example of text recognition instructions and layout analysis instructions is given in an embodiment of the present application, wherein the text recognition instruction example is such as "please output the text line content and coordinate box of all document images", and the layout analysis instruction example is such as "please output the layout structure of all document images".

[0103] The text recognition instruction can instruct the large model to output the text line recognition results and coordinate frames of each document image. The layout analysis instruction can instruct the large model to output the element types and their coordinate frames contained in each document image. Combining the subordinate relationship between the coordinate frames of the elements and the coordinate frames of the text lines, this application not only realizes the recognition of comprehensive chapters, but also realizes multi-level document information extraction to meet diverse practical application needs.

[0104] In this step, the text features of the first document recognition instruction are extracted through the text encoding module so as to be fused with the visual features of each page document image in the subsequent steps and sent to the large model for processing.

[0105] It should be noted that the process of generating tokens through reasoning by the large model can refer to the results of historical generation. That is, as the large model generates tokens one by one through reasoning, when predicting the next token, the previously predicted tokens can be spliced ​​with the first document recognition instruction into a text sequence, and then the text features of the text sequence can be extracted through the text encoding module. For the predicted tokens, they can be divided according to the document images to which they belong, and the set identifiers can be added before and after the token sequence recognized by each document image to indicate the beginning and end of the recognition content of each document image. Exemplarily, you can add before the recognition content of each document image. <text> , add after the recognition content of each document image< / text> .

[0106] Different from traditional text embedding methods, the text encoding module in this application encodes the entire text sequence, taking into account the semantics and contextual information of the entire text, rather than just the representation of a single word, which can help large models understand and process longer text sequences.

[0107] Step S130, fuse the visual features of each page of the document image with the text features of the first document recognition instruction, and send the fused features to the feature processing layer of the configured large model to obtain the document recognition result output by the large model.

[0108] Specifically, a large model usually refers to a deep learning model with billions to tens of billions of parameters, which is used to process and generate natural language text. The large model is pre-trained on a large scale to learn the structure, semantics, and grammatical rules of the language on massive text data, so that it can perform well in various language understanding and generation tasks. The large model can not only understand the surface meaning of the input text, but also infer the semantics and logic behind it. The powerful language understanding and context modeling capabilities of the large model provide a strong guarantee for end-to-end multi-page document recognition and parsing.

[0109] In this step, the visual features of each page of the document image and the text features of the first document recognition instruction are spliced ​​and fused, and sent to the feature processing layer of the large model as an input sequence. Through the reasoning of the large model, the final document recognition result can be output.

[0110] Depending on the difference of the first document recognition instruction, the document recognition result may also be different. Exemplarily, if the first document recognition instruction only includes the text recognition instruction, the document recognition result output by the large model only includes the text line content and the coordinate frame. If the first document recognition instruction includes both the text recognition instruction and the layout analysis instruction, the document recognition result output by the large model may include the text line content and its coordinate frame, as well as the layout element information.

[0111] In this embodiment, in order to effectively model the contextual relationship between multiple pages of documents, the attention mask matrix of the large model can adopt a lower triangular matrix, so that the features of different document images can interact with each other, thereby improving the contextual modeling capability.

[0112] The embodiment of the present application proposes a document recognition method based on a large model. The text to be recognized can be split into multiple pages of document images, and the visual features of each page of the document image can be extracted separately. The visual features of the multiple pages of document images can be simultaneously fed into the large model, and the text features of the first document recognition instruction are simultaneously extracted and also fed into the large model, so that the recognition results of the multiple pages of documents can be output end-to-end through the large model. Since the large model has a stronger modeling capability, the visual features of the multiple pages of document images are simultaneously fed into the large model, so that the contextual information between the document images of different pages can be retained, and then the contextual semantics of the text can be understood, and the hierarchical structure and content relationship of the document can be analyzed more accurately, thereby improving the accuracy of text recognition and layout understanding, and improving the accuracy of the recognition results of multiple pages of documents.

[0113] If the goal of the text recognition task is to simultaneously recognize text line recognition results and layout element information, the above embodiment introduces a solution in which the first document recognition instruction simultaneously includes a text recognition instruction and a layout analysis instruction, that is, a task prompt instruction task prompt can be used to instruct the large model to simultaneously output text line recognition results and layout element information.

[0114] This embodiment further introduces another optional method of using two different task prompt instructions to call the large model twice to output the text line recognition results and layout element information respectively. Figure 5 As shown, the document recognition method provided in this embodiment may include the following steps:

[0115] Step S200: dividing the document to be recognized into document images of more than one page.

[0116] Step S210: extracting visual features of each page of the document image through a visual encoding module.

[0117] Step S220: extracting text features of a set first document recognition instruction through a text encoding module, where the first document recognition instruction is used to instruct the large model to output a text line recognition result of a document image.

[0118] The first document recognition instruction may specifically be a text recognition instruction, which is used to instruct the large model to output a text line recognition result of a document image.

[0119] Step S230: extracting text features of a set second document recognition instruction through a text encoding module, where the second document recognition instruction is used to indicate layout element information of a document image output by the large model.

[0120] The second document recognition instruction may specifically be a layout analysis instruction, which is used to instruct the large model to output layout element information of the document image.

[0121] Step S240, fuse the visual features of each page of the document image with the text features of the first document recognition instruction, and send the fused features to the feature processing layer of the configured large model to obtain the text line recognition result output by the large model.

[0122] Step S250, fuse the visual features of each page document image and the text features of the second document recognition instruction, and send the fused features to the feature processing layer of the large model to obtain the layout element information output by the large model.

[0123] The method provided in this embodiment uses two different task prompt instructions (a first document recognition instruction and a second document recognition instruction) to call the large model twice for reasoning, thereby obtaining text line recognition results and layout element information respectively. By calling the large model twice, each time only instructing the large model to perform one type of document recognition task, the accuracy of the large model reasoning results can be improved.

[0124] In some embodiments of the present application, another document recognition method is provided. In order to effectively locate the position information of the visual feature, in this embodiment, based on the previous embodiment, after obtaining the visual features of each page of the document image, the position code can be further added to the visual feature to obtain the updated visual feature, so as to facilitate the subsequent fusion with the text feature.

[0125] Specifically, based on the above-mentioned embodiment, this embodiment may further include the following steps:

[0126] Determine the visual position encoding corresponding to each page of the document image.

[0127] Among them, the visual position coding includes intra-page position coding, and the intra-page position coding is composed of a number of learnable position coding vectors. The number of position coding vectors is the same as the number of visual unit tokens corresponding to the visual features of each page of the document image. Taking the number of tokens m corresponding to the visual features of each page of the document image as an example, the number of learnable position coding vectors is also m, which is used to determine the position information between the visual features of the m tokens in each document image.

[0128] Further optionally, the visual position coding may further include inter-page position coding, wherein the inter-page position coding is composed of absolute position coding and is used to distinguish the positional relationship between different document images.

[0129] The m learnable position encoding vectors corresponding to the intra-page position encoding are updated and fixed during the large model training process. The absolute position encoding corresponding to the inter-page position encoding can be calculated using the following formula:

[0130]

[0131] Among them, 2i, 2i+1 represent the index values ​​of the channel dimension of the position encoding, d represents the number of channels of the encoding vector, and pos represents the page number of the document.

[0132] Furthermore, for each page of the document image, the visual features of the current page of the document image are fused with the visual position code to obtain updated visual features of the current page of the document image.

[0133] Specifically, the visual features of the current page document image include visual features of m visual unit tokens. When fused with the visual position code, the visual features of the m tokens in the current page document image are fused with the intra-page position code of the current page document image in an element-wise addition manner at corresponding positions, and the visual features of the m tokens after the addition are added to the inter-page position code of the current page document image to obtain the updated visual features of the current page document image.

[0134] By fusing the visual features of each page of the document image with the visual position encoding, the visual features of the final n×m tokens not only have the relative position information within the same page, but also include the distinctiveness between different pages, laying the foundation for the subsequent large model's multi-page document recognition.

[0135] In this embodiment, after obtaining the updated visual features of each page of the document image, it can be further fused with the text features of the first document recognition instruction, and the fused features are sent to the feature processing layer of the large model to obtain the document recognition results output by the large model. The above-mentioned process of fusing the visual features with the text features and the subsequent processing process can be referred to the previous introduction, and will not be repeated here.

[0136] Reference Figure 6 , which illustrates a schematic diagram of the overall framework of document recognition.

[0137] For a multi-page document (n pages), each page of the document image is sent to the visual encoding module after splitting. The visual encoding module includes a visual encoder and a sampler. Each page of the document image is encoded to obtain the visual features of m tokens. Therefore, a total of n×m token visual features can be obtained for n pages of documents.

[0138] Furthermore, the visual position coding of each page of the document image is determined through the position coding module, which includes the intra-page position coding and the inter-page position coding. The visual features of each page of the document image and the visual position coding are fused to obtain updated visual features of each page of the document image.

[0139] On the other hand, the text sequence consisting of the task prompt and the historical recognition results can be encoded by the text encoding module to obtain text features. Decoding the i-th token with the large model LLM i For example, each token decoded historically can be used as a historical recognition result.

[0140] The visual features and text features are concatenated and sent to the feature processing layer of the large model LLM. After the large model is inferred, the document recognition results are output.

[0141] Among them, the task prompt instructions may include two types, namely text recognition instructions and layout analysis instructions. Correspondingly, the document recognition results output by the large model may include: text line recognition results and layout element information.

[0142] For the large model used in the embodiments of the present application, a pre-training process can be performed before reasoning. A variety of model training strategies can be used during pre-training, such as using the LoRA training strategy to fine-tune the large model, which can not only retain the language understanding characteristics of the large model, but also significantly reduce the memory usage of the large model during training.

[0143] The present application provides a large model training method, which may include the following steps:

[0144] S1. Obtain training document samples and corresponding document recognition result labels.

[0145] S2. Divide the training document sample into more than one page of training document images.

[0146] S3. Extract visual features of m tokens of each page of training document image through a visual encoding module, and extract text features of the set first document recognition instruction through a text encoding module.

[0147] S4. For each page of training document image, the visual features of the m tokens of the current page training document image are added element by element to the corresponding positions of the m learnable position encoding vectors, and the visual features of the m tokens after addition are added with the inter-page position encoding of the current page training document image to obtain the updated visual features of the current page training document image.

[0148] S5. Fuse the updated visual features of each page of training document images with the text features of the first document recognition instruction, and send the fused features to the feature processing layer of the large model to obtain the document recognition results output by the large model.

[0149] S6. Taking the document recognition result approaching the document recognition result label as the training goal, update the network parameters of the large model and m learnable position encoding vectors until the training end condition is met.

[0150] It should be further explained that if the visual encoding module adopts Figure 6 The combination of the visual encoder and the sampler shown in the figure can adopt the Cross Attention layer structure introduced in the above embodiment for the sampler, where Q represents m learnable query vectors, and the m learnable query vectors can participate in the update during the training process until the training end condition is met.

[0151] Through the training process of the above large model, the network parameters of the large model can be updated. At the same time, m learnable position encoding vectors are obtained, which are used as the intra-page position encoding of each page of the document image in the subsequent reasoning stage.

[0152] The document recognition device provided in an embodiment of the present application is described below. The document recognition device described below and the document recognition method described above can be referenced to each other.

[0153] See also Figure 7 , Figure 7 A schematic diagram of the structure of a document recognition device disclosed in an embodiment of the present application.

[0154] like Figure 7 As shown, the device may include:

[0155] The document segmentation unit 11 is used to segment the document to be identified into document images of more than one page;

[0156] A visual feature extraction unit 12, used to extract visual features of each page of the document image through a visual encoding module;

[0157] A text feature extraction unit 13, used to extract text features of a set first document recognition instruction through a text encoding module, wherein the first document recognition instruction is used to instruct the large model to output a document recognition result of a document image;

[0158] The large model inference unit 14 is used to fuse the visual features of each page of the document image with the text features of the first document recognition instruction, and send the fused features to the feature processing layer of the configured large model to obtain the document recognition result output by the large model.

[0159] In a possible implementation, the first document recognition instruction includes: a text recognition instruction for instructing the large model to output a text line recognition result of the document image; based on this, the document recognition result output by the large model includes: a text line recognition result.

[0160] Further optionally, the first document recognition instruction also includes: a layout analysis instruction, used to instruct the large model to output layout element information of the document image; based on this, the document recognition result output by the large model also includes: layout element information.

[0161] In another possible implementation, if the first document recognition instruction is specifically a text recognition instruction, which is used to instruct the large model to output the text line recognition result of the document image, the text feature extraction unit can also be used to: extract the text features of the set second document recognition instruction, and the second document recognition instruction is used to instruct the large model to output the layout element information of the document image. On this basis, the large model reasoning unit can also be used to: fuse the visual features of each page of the document image and the text features of the second document recognition instruction, and send the fused features to the feature processing layer of the large model to obtain the layout element information output by the large model.

[0162] In a possible implementation, the visual encoding module includes a visual encoder and a sampler, and the process of extracting the visual features of each page of the document image by the visual encoding module by the visual feature extraction unit includes:

[0163] Extracting initial visual features of each page of document image through visual encoder;

[0164] The sampler extracts visual features of m visual unit tokens from the initial visual features of each page of the document image, where m is a set constant.

[0165] In a possible implementation, the device of the present application may further include:

[0166] a position coding unit, used for determining a visual position coding corresponding to each page of the document image before fusing the visual features of each page of the document image with the text features of the first document recognition instruction, wherein the visual position coding includes an intra-page position coding and an inter-page position coding, wherein the intra-page position coding is composed of m learnable position coding vectors, and the inter-page position coding is composed of absolute position coding, wherein the m learnable position coding vectors are updated and fixed during the large model training process;

[0167] The position coding fusion unit is used to fuse the visual features of the current page document image with the visual position coding for each page document image to obtain updated visual features of the current page document image.

[0168] In a possible implementation, the device of the present application may further include:

[0169] The large model training unit is used to train the large model. The training process includes:

[0170] Obtain training document samples and corresponding document recognition result labels;

[0171] Cutting the training document sample into more than one page of training document images;

[0172] Extracting visual features of m tokens of each page of training document image through a visual encoding module, and extracting text features of a set first document recognition instruction through a text encoding module;

[0173] For each page of training document image, add the visual features of the m tokens of the current page training document image and the m learnable position encoding vectors element by element at the corresponding positions, and add the visual features of the m tokens after addition to the inter-page position encoding of the current page training document image to obtain the updated visual features of the current page training document image;

[0174] Fusing the updated visual features of each page of training document images with the text features of the first document recognition instruction, and sending the fused features to the feature processing layer of the large model to obtain the document recognition results output by the large model;

[0175] Taking the document recognition result approaching the document recognition result label as the training goal, the network parameters of the large model and m learnable position encoding vectors are updated until the training end condition is reached.

[0176] The present application also provides an electronic device in an embodiment. Figure 8 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, tablet computers, teaching large screens, wearable devices, etc. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0177] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603 to implement the document recognition method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0178] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0179] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any document recognition method provided in the embodiment of the present application.

[0180] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any document recognition method provided in an embodiment of the present application.

[0181] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0182] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0183] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0184] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0185] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A document recognition method, characterized in that: include: Cut the document to be recognized into document images of more than one page; Extract visual features of each page of document image through visual encoding module; Extracting text features of a set first document recognition instruction through a text encoding module, wherein the first document recognition instruction is used to instruct the large model to output a document recognition result of a document image; The visual features of each page of the document image and the text features of the first document recognition instruction are fused, and the fused features are sent to the feature processing layer of the configured large model to obtain the document recognition results output by the large model.

2. The method according to claim 1, characterized in that: The first document recognition instruction includes: a text recognition instruction for instructing the large model to output a text line recognition result of a document image; Based on this, the document recognition results output by the large model include: text line recognition results.

3. The method according to claim 2, characterized in that The first document recognition instruction also includes: a layout analysis instruction for instructing the large model to output layout element information of the document image; Based on this, the document recognition result output by the large model also includes: layout element information.

4. The method according to claim 2, characterized in that: Also includes: Extracting text features of a set second document recognition instruction through a text encoding module, wherein the second document recognition instruction is used to instruct the large model to output layout element information of a document image; The visual features of each page document image and the text features of the second document recognition instruction are fused, and the fused features are sent to the feature processing layer of the large model to obtain the layout element information output by the large model.

5. The method according to claim 1, characterized in that The visual encoding module includes a visual encoder and a sampler. The process of extracting the visual features of each page of the document image through the visual encoding module includes: Extracting initial visual features of each page of document image through visual encoder; The sampler extracts visual features of m visual unit tokens from the initial visual features of each page of the document image, where m is a set constant.

6. The method according to claim 1, characterized in that Before fusing the visual features of the document images of each page with the text features of the first document recognition instruction, the method further includes: Determine the visual position coding corresponding to each page of the document image, wherein the visual position coding includes an intra-page position coding, and the intra-page position coding is composed of a plurality of learnable position coding vectors, wherein the plurality of learnable position coding vectors are updated and fixed during the large model training process, and the number of learnable position coding vectors is the same as the number of visual unit tokens corresponding to the visual features of each page of the document image; For each page of the document image, the visual features of the current page of the document image are fused with the visual position code to obtain updated visual features of the current page of the document image.

7. The method according to claim 6, characterized in that The visual position code also includes an inter-page position code, and the inter-page position code is composed of an absolute position code.

8. The method according to claim 7, characterized in that The training process of the large model includes: Obtain training document samples and corresponding document recognition result labels; Cutting the training document sample into more than one page of training document images; Extracting visual features of m tokens of each page of training document image through a visual encoding module, and extracting text features of a set first document recognition instruction through a text encoding module; For each page of training document image, add the visual features of the m tokens of the current page training document image and the m learnable position encoding vectors element by element at the corresponding positions, and add the visual features of the m tokens after addition to the inter-page position encoding of the current page training document image to obtain the updated visual features of the current page training document image; Fusing the updated visual features of each page of training document images with the text features of the first document recognition instruction, and sending the fused features to the feature processing layer of the large model to obtain the document recognition results output by the large model; Taking the document recognition result approaching the document recognition result label as the training goal, the network parameters of the large model and m learnable position encoding vectors are updated until the training end condition is reached.

9. A document recognition device, characterized in that: include: A document segmentation unit, used for segmenting the document to be identified into document images of more than one page; A visual feature extraction unit, used for extracting visual features of each page of the document image through a visual encoding module; A text feature extraction unit, used to extract text features of a set first document recognition instruction through a text encoding module, wherein the first document recognition instruction is used to instruct the large model to output a document recognition result of a document image; The large model inference unit is used to fuse the visual features of each page of the document image with the text features of the first document recognition instruction, and send the fused features to the feature processing layer of the configured large model to obtain the document recognition results output by the large model.

10. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the document recognition method according to any one of claims 1 to 8.

11. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the document recognition method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the document recognition method according to any one of claims 1 to 8 is implemented.

Images