Document image recognition method, related device, equipment and storage medium

Abstract

The invention discloses a document image recognition method, a related device, equipment and a storage medium, and the method comprises the steps: carrying out the image segmentation based on a document image, and obtaining all document blocks in the document image; wherein the type to which the document block belongs at least comprises a text; sequentially executing the following steps on each document block belonging to the text type: selecting the document block as a current block, selecting the document block belonging to the text type except the current block as a reference block based on a target text sequence, and selecting the document block belonging to the text type except the current block as the reference block based on the initial recognition text of the current block and the initial recognition text of each reference block; determining a reference block that should follow the current block; wherein the current block is used as a starting point to connect any reference block to meet the target text sequence; and obtaining an identification document of the document image at least based on the connection sequence between the document blocks of which the category is the text and the initial identification text. According to the scheme, the accuracy of the text sequence can be improved in the document image recognition process.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a document image recognition method and related devices, equipment and storage media. Background Art

[0002] Image recognition technology is widely used in business, office, study and other scenarios. For example, image recognition technology can be used to identify the text in an image and organize the recognized text into an editable document, greatly improving the convenience in business, office, study and other scenarios.

[0003] However, existing image recognition technology still has certain defects. For example, when faced with scenes with complex content such as drug instructions, there are still many problems after converting document images into digital formats, especially difficulty in identifying the accurate text order. In view of this, how to improve the accuracy of text order in the document image recognition process has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a document image recognition method and related devices, equipment and storage media, which can improve the accuracy of text sequence in the document image recognition process.

[0005] In order to solve the above-mentioned technical problems, the first aspect of the present application provides a document image recognition method, comprising: performing image segmentation based on a document image to obtain various document blocks in the document image; wherein the categories to which the document blocks belong include at least text; performing the following steps in sequence on each document block belonging to the category of text: selecting a document block as the current block, and based on the target text sequence, selecting a document block belonging to the category of text outside the current block as a reference block, and determining a reference block that should be connected to the current block based on the initial recognition text of the current block and the initial recognition text of each reference block; wherein connecting any reference block with the current block as the starting point complies with the target text sequence; obtaining a recognized document of the document image based at least on the connection order and the initial recognition text between each document block belonging to the category of text.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a document image recognition device, including: an image segmentation module, a sequence recognition module and a document acquisition module, the image segmentation module is used to perform image segmentation based on the document image to obtain each document block in the document image; wherein the category of the document block at least includes text; the sequence recognition module is used to sequentially perform the following steps on each document block belonging to the category of text: select a document block as the current block, and based on the target text sequence, select a document block belonging to the category of text outside the current block as a reference block, and determine the reference block that should be connected to the current block based on the initial recognition text of the current block and the initial recognition text of each reference block; wherein, connecting any reference block with the current block as the starting point conforms to the target text sequence; the document acquisition module is used to obtain the recognition document of the document image based at least on the connection sequence and initial recognition text between each document block belonging to the category of text.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the document image recognition method in the above first aspect.

[0008] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the document image recognition method of the first aspect.

[0009] The above scheme performs image segmentation based on the document image to obtain each document block in the document image, and the document block belongs to at least text, and then performs the following steps on each document block belonging to the text type: select a document block as the current block, and select a document block belonging to the text type other than the current block as a reference block based on the target text sequence, and determine the reference block that should be connected to the current block based on the initial recognition text of the current block and the initial recognition text of each reference block, and connect any reference block with the current block as the starting point to meet the target text sequence. On this basis, at least based on the connection sequence and initial recognition text between each document block belonging to the text type, obtain the recognition document of the document image, that is, first select the reference block according to the target text sequence, and can preliminarily screen out the document block that may be connected to the current block according to the target text sequence, and then further determine it through the initial recognition text of both the current block and the reference block. Therefore, when determining the connection sequence of each document block through a two-stage process, the document block that interferes with the connection sequence can be reasonably excluded first, and then determined according to the initial recognition text. Therefore, the accuracy of the text sequence can be improved in the document image recognition process. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1It is a flowchart of an embodiment of a document image recognition method of the present application; Figure 2a is a schematic diagram of an embodiment of a document image; Figure 2b It is a process diagram of an embodiment of the document image recognition method of the present application; Figure 2c is a schematic diagram of another embodiment of a document image; Figure 2d It is a process schematic diagram of another embodiment of the document image recognition method of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of a document image recognition device of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the document image recognition method of the present application. Specifically, it may include the following steps: Step S11: performing image segmentation based on the document image to obtain various document blocks in the document image.

[0015] In the embodiments of the present disclosure, the categories to which the document block belongs include at least text. It should be noted that the document block belonging to the category of text may be mainly composed of text, but it does not mean that it must be composed entirely of pure text. Exemplarily, a character string such as a formula formed by characters through a specific structure or an unspecified structure may also be inserted into the document block belonging to the category of text. In addition, in some possible examples, the categories to which the document block belongs may also include tables, graphics, etc., and the categories to which the document block belongs are not limited here.

[0016] In one implementation scenario, the document image can be obtained by capturing the image of a paper document by an image capture device such as a camera or a scanning device. It should be noted that the document image can at least include a drug instruction image, that is, the paper document from which the document image originates can specifically be a drug instruction. Of course, it is not limited to this, such as the document image can also include but not limited to: tool instruction images, home appliance instruction images and other instruction images. In addition, the document image can also include but not limited to: book images, newspaper images, etc. The specific source of the document image is not limited here, and no examples are given one by one.

[0017] In one implementation scenario, after obtaining the document image and before processing the document image, the document image may be preprocessed. Exemplarily, the preprocessing may include but is not limited to: denoising, correcting, trimming, etc., to improve the recognition accuracy as much as possible.

[0018] In one implementation scenario, as a possible example, image segmentation can be achieved through an image segmentation model. Exemplarily, the image segmentation model can include but is not limited to: FCN, U-Net, SegNet and other neural network models, and the specific structure of the image segmentation model is not limited here. In order to improve the accuracy of the image segmentation model, a number of sample document images can be obtained in advance, and the sample document images are annotated with sample document blocks, so that the sample document images can be segmented based on the image segmentation model to obtain predicted document blocks in the sample document images, and then the network parameters of the image segmentation model can be adjusted based on the difference between the sample document blocks and the predicted document blocks. After the image segmentation model training converges, the document image can be segmented based on the image segmentation model to obtain each document block in the document image. Exemplarily, please refer to Figure 2a , Figure 2a Schematic diagram of an embodiment of a document image. Figure 2aAs shown, the document image is divided into the following document blocks in order of priority from top to bottom and secondarily from left to right: document block 1 (text block 1), document block 2 (graphic block 1), document block 3 (text block 2), document block 4 (text block 3), document block 5 (text block 4), document block 6 (table block 1), document block 7 (text block 5), document block 8 (text block 6), document block 9 (text block 7), document block 10 (graphic block 2), document block 11 (text block 8), document block 12 (text block 9). Of course, the above examples are only several possible examples of document images being divided into several document blocks in actual application, and other possible situations of document images being divided into document blocks will not be given examples one by one here.

[0019] In one implementation scenario, as another possible example, edge detection technology can be used to perform edge detection on a document image to divide the complete document image into a number of blocks. On this basis, for each block, image segmentation can be performed based on the block to obtain at least one document block in the block. The specific process of performing image segmentation on the block can refer to the above-mentioned related description, which will not be repeated here.

[0020] Step S12: For each document block of the text type, the following steps are performed in sequence: a document block is selected as the current block, and based on the target text order, a document block of the text type other than the current block is selected as a reference block, and based on the initial recognition text of the current block and the initial recognition text of each reference block, a reference block that should be connected to the current block is determined.

[0021] In the embodiment of the present disclosure, connecting any reference block with the current block as the starting point conforms to the target text order. As a possible example, the target text order may be the possible text order of the original document to which the document image belongs. For example, it may include but is not limited to from top to bottom, from left to right, etc., and the target text order is not specifically limited here.

[0022] In an implementation scenario, when the category of the document block also includes non-text, before the above steps are sequentially performed on each document block of the text category, the document block of the non-text category can also be processed first. Specifically, each document block of the non-text category can be selected as the target block in turn, and the document block adjacent to the target block and belonging to the text category can be selected as the candidate block, and based on the initial recognition text of the target block and the initial recognition text of each candidate block, the candidate block attached to the target block is selected as the attached block, and then the candidate block as the attached block is merged into the target block to obtain a new target block. It should be noted that the non-text includes at least one of a graphic and a table, and the initial recognition text in the candidate block attached to the target block represents any one of the title and the comment of the target block. In the above method, when the category of the document block also includes non-text, the initial recognition text of the document block of the non-text category and the document block adjacent to it and belonging to the text category are first used to distinguish whether the latter is attached to the former in the semantic dimension, and merged when it is identified as attached, which can improve the construction accuracy of the document block of the non-text category.

[0023] In a specific implementation scenario, according to one of the possible text order, each document block of non-text type can be selected in turn as the target block, and then the document block of text type adjacent to the target block can be selected as the candidate block. Figure 2a , for each document block of non-text type: document block 2 (graphic block 1), document block 6 (table block 1), document block 10 (graphic block 2), if the possible order of writing is from top to bottom, document block 2 (graphic block 1), document block 6 (table block 1), document block 10 (graphic block 2) can be selected in turn as target blocks. Exemplarily, when document block 6 (table block 1) is selected as the target block, document blocks adjacent to document block 6 (table block 1) and belonging to text type, namely document block 4 (text block 3), document block 5 (text block 4), document block 7 (text block 5), document block 8 (text block 6), etc. can be selected as candidate blocks. Of course, the above example is only a possible example in the actual application process, and other possible situations are not given one by one here. It should be noted that being adjacent to the target block can mean being adjacent to the target block, or the number of document blocks separated from the target block can be within a threshold (e.g., within 1 document block, within 2 document blocks, etc.), and the specific requirements for being adjacent to the target block are not limited here.

[0024] In a specific implementation scenario, after selecting the target block and the corresponding candidate blocks, a prompt command can be constructed based on the initial recognition text of the target block and the initial recognition text of each candidate block, and the prompt command instructs the large language model to distinguish whether the initial recognition text of the candidate block is any of the title and comment of the target block. It should be noted that the large language model may include but is not limited to: LLAMA, Bloom and other open source large models, or the large language model may also include a fine-tuned large model after fine-tuning the parameters of the open source large model through customized corpus; or the large language model may also be a custom large model, and the specific source of the large language model is not limited here. In addition, as a possible example, taking the following specific example as an example: the initial recognition text of the target block is "patients with drug resistance variation (%) phenotype * genotype and phenotype * adults and adolescents 4 / 1245 (0.32%) 5 / 1245 (0.4%) children (1-12 years old) 19 / 464 (4.1%) 25 / 464 (5.4%)", and the initial recognition text of the candidate block above the target block is "variation of drug resistance in patients" and the initial recognition text of the candidate block below the target block is "full genotyping was not performed in all studies", the following prompt instructions can be constructed: Patients with drug-resistant variants (%) Phenotype* Genotype and phenotype* Adults and adolescents 4 / 1245 (0.32%) 5 / 1245(0.4%) children (1-12 years old) 19 / 464(4.1%) 25 / 464(5.4%).

[0025] Variation of drug resistance among patients.

[0026] *Full genotyping was not performed in all studies.

[0027] The first paragraph of the above input content is the table content, and the remaining paragraphs may contain table titles and table comments. If it exists, please output its title or comment, and do not provide other descriptions.

[0028] Of course, the above example is only a possible example of the specific content of the prompt instruction in the actual application process, and does not limit the specific content of the prompt instruction. After obtaining the prompt instruction, the prompt instruction can be input into the large language model, so that the candidate block attached to the target block can be determined as the attached block based on the output result of the large language model triggered by the prompt instruction. Still taking the above example as an example, the output result of the large language model may include: Title: Variation of drug resistance in patients.

[0029] Note: *Full genotyping was not performed in all studies.

[0030] On this basis, the candidate block located one block above the target block can be used as a subsidiary block attached to the target block, and the candidate block located one block below the target block can be used as a subsidiary block attached to the target block, and the two can be merged into the target block to form a new target block. The above method constructs a prompt instruction based on the initial recognition text of the target block and the initial recognition text of each candidate block, and determines the candidate block attached to the target block as a subsidiary block based on the output result of the large language model triggered by the prompt instruction. This can make full use of the general understanding ability of the large language model to perform subsidiary identification in the semantic dimension, which helps to improve the efficiency and accuracy of subsidiary identification.

[0031] In another specific implementation scenario, different from the aforementioned attachment identification method, as another possible example, please refer to Figure 2b , Figure 2b 1 is a process diagram of an embodiment of the document image recognition method of the present application. Figure 2b As shown, it is also possible to first select a document block below the target block and belonging to the type of text (such as two document blocks below the target block and belonging to the type of text) as the first candidate block, and then obtain the first discrimination result of the first candidate block based on the initial recognition text of the target block and the initial recognition text of each first candidate block, and the first discrimination result includes any of the following: the initial recognition text of the first candidate block as the title of the target block, and the initial recognition result of the first candidate block as the annotation of the target block. It should be noted that the first discrimination result of the first candidate block can refer to the specific process of the subsidiary identification in the aforementioned example, which will not be repeated here. On this basis, it can be determined whether to continue to select the document block above the target block and belonging to the type of text as the second candidate block based on whether the first discrimination result includes the first candidate block as the title. The above method can help improve the efficiency of subsidiary discrimination by first performing subsidiary discrimination on the candidate block below the target block and then performing subsidiary discrimination on the candidate block above the target block as appropriate.

[0032] Please continue to refer to Figure 2b As a possible scenario, when there is a first discrimination result including a first candidate block as a title, in response to the existence of only the first discrimination result including the first candidate block as the title, the first discrimination result including the first candidate block as the title can be selected as an auxiliary block, and the auxiliary discrimination is ended, and in response to the existence of the first discrimination result including the first candidate block as a comment (that is, there is both the first discrimination result including the first candidate block as the title and the first discrimination result including the first candidate block as the comment), the first discrimination result including the first candidate block as the title and the first discrimination result including the first candidate block as the comment can be selected as auxiliary blocks, and the auxiliary discrimination is ended.

[0033] Please continue to refer to Figure 2bAs another possible situation, when there is no first candidate block as a title in the first discrimination result, it can be determined whether to obtain the second discrimination result of the second candidate block based on the initial recognition text of the target block and the initial recognition text of each second candidate block after selecting the first candidate block as a comment as an auxiliary block. Specifically, if there is still a first candidate block as a comment in the first discrimination result, then after selecting the first candidate block as a comment as an auxiliary block, the second discrimination result of the second candidate block is obtained based on the initial recognition text of the target block and the initial recognition text of each second candidate block. On the contrary, if there is no first candidate block as a comment in the first discrimination result, the second discrimination result of the second candidate block is obtained directly based on the initial recognition text of the target block and the initial recognition text of each second candidate block. It should be noted that the second discrimination result includes: whether the initial recognition text of the second candidate block is the title of the target block. In addition, the second discrimination result of the second candidate block can refer to the specific process of the auxiliary identification in the above example, which will not be repeated here. On this basis, it can be determined whether to select the second candidate block as an auxiliary block of the target block based on the second discrimination result of the second candidate block. Exemplarily, when the second discrimination result of the second candidate block includes the initial recognition text of the second candidate block as the title of the target block, the second candidate block can be selected as an auxiliary block of the target block, and when the second discrimination result of the second candidate block includes that the initial recognition text of the second candidate block is not as the title of the target block, the second candidate block may not be selected as an auxiliary block of the target block.

[0034] In an implementation scenario, please refer to Figure 2a and Figure 2c , Figure 2c FIG. 2 is a schematic diagram of another embodiment of a document image. Figure 2a and Figure 2c As shown, after the above subsidiary determination, Figure 2a Some of the document blocks will be merged, such as document block 5 (text block 4) and document block 7 (text block 5) will be merged into document block 6 (table block 1) to form a new document block 5 (table block 1), and document block 8 (text block 6) will be updated to document block 6 (text block 4), while document block 9 (text block 7) and document block 11 (text block 8) will be merged into document block 10 (graphic block 2) to form a new document block 7 (graphic block 2), and document block 12 (text block 9) will be updated to document block 8 (text block 5). In other words, when each document block is numbered in advance according to one of the possible text sequences, after the candidate blocks as subsidiary blocks are merged into the target block to obtain a new target block, the numbers of each new document block can be updated simultaneously.

[0035] In an implementation scenario, for each document block of the text type, the document blocks can be selected in sequence according to one of the possible text sequences as the current block. Figure 2c , when following the possible order of writing from top to bottom, you can select document block 1 (text block 1), document block 3 (text block 2), document block 4 (text block 3), document block 6 (text block 4), and text block 8 (text block 5) as the current block in turn. Of course, the above example is only a possible example in the actual application process. The selection order of each document block of the type of text is not specifically limited here, and no examples are given here one by one. On this basis, you can select document blocks of the type of text other than the current block as reference blocks according to the target order of writing from top to bottom, from left to right, etc. For example, when document block 3 (text block 2) is selected as the current block, according to the target text order from top to bottom, document block 4 (text block 3) can be selected as the reference block, that is, document block 3 (text block 2) is used as the starting point to connect to document block 4 (text block 3) in accordance with the target text order from top to bottom. At the same time, according to the target text order from left to right, document block 6 (text block 4) can be selected as the reference block, that is, document block 3 (text block 2) is used as the starting point to connect to document block 6 (text block 4) in accordance with the target text order from left to right. Of course, the above example is only a possible example in the actual application process, and other possible situations are not limited thereto, and examples are not given one by one here. After selecting the current block and its reference block, the reference block that should be connected to the current block can be determined based on the initial recognition text of the current block and the initial recognition text of each reference block.

[0036] In a specific implementation scenario, as a possible example, in order to determine the reference block that should be connected to the current block, a prompt instruction can be constructed based on the initial recognition text of the current block and the initial recognition text of each reference block. Different from the aforementioned prompt instruction, the prompt instruction in this implementation scenario instructs the large language model to distinguish the reference block that should be connected after the current block, and then determines the reference block that should be connected to the current block based on the output result of the large language model triggered by the prompt instruction. The above method uses the general understanding ability of the large language model to respectively determine whether each reference block should be connected to the current block, which can improve the accuracy of sequential determination of document blocks belonging to the text type.

[0037] In a specific implementation scenario, for example, the initial recognition text of the current block may include "[Adverse Reaction] The total number of subjects in the series of clinical trials conducted on this product in China is XXXX people...", the initial recognition text of one reference block may include "The results of protection, effect index and incidence rate by age group are detailed in the table below...", and the initial recognition text of another reference block may include "The Phase III clinical protection test was conducted in XXXX healthy subjects aged XX months according to the 0 and 28 day immunization program...". Taking this as an example, a prompt instruction can be constructed: Current block: [Adverse Reactions] The total number of subjects in the series of clinical trials of this product conducted in China is XXXX people...

[0038] Reference Block 1: The results of protection, effectiveness index and incidence rate by age group are shown in Table X….

[0039] Reference Block 2: Phase III clinical protection trial in XXXX healthy adults aged XX months according to the 0-28 day immunization schedule sequence…….

[0040] The positions of the above three paragraphs may be disordered. Select the ones that need to understand their contents and output them. Which of the following two paragraphs should the first paragraph of text be spliced ​​with? Please output your judgment result: reference block 1 or reference block 2. Don't output anything else.

[0041] The above prompt instruction is input into the large language model, and the large language model outputs the following result: Refer to block 2.

[0042] On this basis, the reference block that should be connected to the current block can be determined to be "reference block 2". By performing the above operation on each document block of the text type, the connection order between each document block of the text type can be determined.

[0043] Step S13: obtaining a recognition document of the document image based at least on the connection sequence between the document blocks belonging to the text category and the initial recognition text.

[0044] In one implementation scenario, after obtaining the connection sequence between the document blocks of the text type, the initial recognition texts of the document blocks of the text type can be sequentially spliced ​​according to the connection sequence between the document blocks of the text type to obtain the recognition text of the document image. In addition, when the document blocks also include non-text (such as graphics, tables, etc.), in the splicing process, in response to the document blocks that pass through the middle of the connection sequence and belong to the non-text type, the content contained in the document block (such as graphics, tables, etc.) can be inserted between the initial recognition texts of the document blocks of the text type to obtain the recognition document of the document image. Please refer to Figure 2c, when the connection order between the document blocks of the text type is: document block 1 (text block 1), document block 3 (text block 2), document block 4 (text block 3), document block 6 (text block 4), document block 8 (text block 5), the table contained in document block 5 (table block 1) can be inserted between the initial recognition text of document block 4 and the initial recognition text of document block 6, and the graphics contained in document block 7 (graphic block 2) can be inserted between the initial recognition text of document block 6 and the initial recognition text of document block 8, so as to obtain the recognition document of the document image. It should be noted that in the actual application process, the processing is not limited to this. For example, the position information of each document block in the document image can also be recorded, and the position information of the content contained in each document block (such as the recognition text of the text block, the table contained in the table block, the graphics contained in the graphic block, etc.) in the recognition document can be determined directly according to the position information of each document block in the document image, so as to obtain the recognition document of the document image, and the editable recognition document can be restored as it is. The specific methods of processing to obtain the recognition document are not given one by one here.

[0045] In another implementation scenario, in order to improve the readability of the table contained in the table block (i.e., the document block of the type of table) in the recognition document, before forming the recognition document, the table contained in the table block can also be processed, such as processing structured forms such as markdown, or processing it into text form. In order to improve the processing efficiency of the table block, the table block can be processed by a large language model. Exemplarily, the initial recognition text of the table in the table block and the title and comments in the table block can be obtained, that is, based on the initial recognition text, title and comments of the table, a prompt instruction can be constructed, and the prompt instruction instructs the large language model to organize the table into a specific form (such as a structured form such as markdown, or a text form) according to the initial recognition text, title and comments of the table, and then the prompt instruction is input into the large language model to obtain the processed table content. Of course, the specific processing form can be set according to the actual application needs. For example, when the recognition document needs to be displayed to relevant personnel for viewing, it can be processed into a text form; or, when the recognition document needs to be handed over to the machine for further processing, it can be processed into a structured form, which is not limited here.

[0046] As a possible example, taking markdown and other structured forms as an example, when the initial identification text of the table is "Patients with drug resistance variation (%) Phenotype* Genotype and phenotype* Adults and adolescents 4 / 1245 (0.32%) 5 / 1245 (0.4%) Children (1-12 years old) 19 / 464 (4.1%) 25 / 464 (5.4%)", the title is "Variation of drug resistance in patients" and the annotation is "*Full genotyping was not performed in all studies", the prompt instruction can be constructed: Initial identification text for the table: Patients with drug resistance variants (%) Phenotype* Genotype and Phenotype* Adults and Adolescents 4 / 1245 (0.32%) 5 / 1245 (0.4%) Children (1-12 years old) 19 / 464 (4.1%) 25 / 464 (5.4%)

[0047] Title: Variation of drug resistance in patients.

[0048] Note: *Full genotyping was not performed in all studies.

[0049] The above input content is table text, table notes, titles, etc. The format is a bit messy, please correct it and Output a structured table in JSON format.

[0050] After the above prompt instructions are input into the large language model, the processed table content in a structured form can be obtained: { "Table Title": "Variation of drug resistance among patients" "Patients with drug-resistant mutations": { "Patient population": "Adults and adolescents" "Phenotype":{ "Number": "4" "Total": "1245" "Percentage": "0.32%" } "Genotype and Phenotype":{ "Number": "5" "Total": "1245" "Percentage": "0.4%" } } { "Patient Person": "Child (1-12 years)" "Phenotype": { "Number": "19" "Total": "464" "Percentage": "4.1%" } "Genotype and Phenotype":{ "Number": "25" "Total": "464" "Percentage": "5.4%" } } ] "Table Notes": "*Full genotyping was not performed in all studies" } As a possible example, taking the processing as text form as an example, when the initial identification text of the table is "Patients with drug resistance variants (%) Phenotype* Genotype and phenotype* Adults and adolescents 4 / 1245 (0.32%) 5 / 1245 (0.4%) Children (1-12 years old) 19 / 464 (4.1%) 25 / 464 (5.4%)", the title is "Variation of drug resistance in patients" and the annotation is "*Full genotyping was not performed in all studies", the prompt instruction can be constructed: Initial identification text for the table: Patients with drug resistance variants (%) Phenotype* Genotype and Phenotype* Adults and Adolescents 4 / 1245 (0.32%) 5 / 1245 (0.4%) Children (1-12 years old) 19 / 464 (4.1%) 25 / 464 (5.4%)

[0051] Title: Variation of drug resistance in patients.

[0052] Note: *Full genotyping was not performed in all studies.

[0053] The above input content is table text, table notes, titles, etc. The format is a bit messy, please correct it and Outputs readable text line by line in plain text form. The overall output is one paragraph, and each line is a sentence.

[0054] After the above prompt instructions are input into the large language model, the processed table content in text form can be obtained: Table title: Patient population: Adults and adolescents. Patient population: Adults and adolescents. Patients with drug-resistant mutations (genotype and phenotype): 4 / 1245 (0.32%). Patients with drug-resistant mutations (genotype and phenotype): 5 / 1245 (0.4%). Children (1-12 years old). Patients with drug-resistant variants (phenotype): 19 / 464 (4.1%). Patients with drug-resistant variants (genotype and phenotype): 25 / 464 (5.4%). Table Notes: *Full genotyping was not performed in all studies.

[0055] It should be noted that the above examples are only two possible examples of processing table blocks in actual application, and other possible processing methods are not limited thereto, and will not be given examples one by one here.

[0056] In another implementation scenario, different from the aforementioned implementation, as another possible implementation, before obtaining the recognized document of the document image based on at least the connection order and initial recognition text between the document blocks of the text type, a first description text about the designated person, a second description text about the error correction example, a third description text about the error correction requirement, and a fourth description text about the error correction method can also be obtained. On this basis, the initial recognition texts of the document blocks of the text type can be sequentially spliced ​​based on the connection order to obtain the initial document of the document image, and then based on the first description text, the second description text, the third description text, the fourth description text and the initial document, a prompt instruction is constructed, and the prompt instruction instructs the large language model to play the designated person and refer to the error correction example to correct the initial document according to the error correction requirement using the error correction method, and then the recognized document can be obtained based on the output result of the large language model triggered by the third prompt instruction. In the above method, the general understanding ability of the large language model is assisted by relevant instructions such as the designated person, error correction example, error correction requirement and error correction method, which can improve the accuracy of error correction of the initial document.

[0057] In a specific implementation scenario, as a possible example, still taking the document image as an image of a drug instruction sheet as an example, the first description text about the specified persona may include but is not limited to the following content: Role: Medical knowledge editor.

[0058] Background: Medical background, familiar with the standard format and regulatory requirements of drug instructions.

[0059] Task: Format and proofread the disorganized drug instructions, correct text errors, and ensure the document Clear, professional and compliant with regulatory standards.

[0060] It should be noted that the above specific example of the first description text is only a possible example of specifying a persona in actual application, and does not limit the specific content of the first description text.

[0061] In a specific implementation scenario, as a possible example, still taking the document image as a drug instruction sheet image as an example, the second description text about the error correction example may include but is not limited to an input example before error correction and an output example after error correction, such as: Input example: Ibuprofen tablets Ingredients: Ibuprofen Appearance: This product is a white tablet

Indications

[0062]

Notes

[0063] -Can be used to treat fever.

[0064] Dosage and Administration: - 200 mg each time, 3 times a day.

[0065] Note: -When taking this product for a long time or in large quantities, liver and kidney function should be monitored regularly.

[0066] It should be noted that the above specific example of the second description text is only a possible example of an error correction example in actual application, and does not limit the specific content of the second description text.

[0067] In a specific implementation scenario, as a possible example, still taking the document image as a drug instruction sheet image as an example, the third description text regarding the error correction requirement may include but is not limited to the following content: 1. Text cleaning: Remove unnecessary spaces and line breaks, leaving only the necessary formatting space for the content.

[0068] All punctuation marks are unified into full-width format, for example, English colon becomes Chinese colon.

[0069] 2. Correction of typos: Only obvious typos and typos were corrected to ensure that wording conformed to standard medical terminology.

[0070] 3. Unified format: Clarify and unify the title format, such as indications and dosage, and ensure that the content of each section is consistent from the new Lines should be used as appropriate, and lists (such as bullet points) should be used to enhance clarity.

[0071] 4. Clear hierarchical structure: Make sure there is adequate spacing between headings and content, using indents or line breaks to separate different sections; use blank lines and Indentation clarifies the hierarchy of different sections while keeping the original presentation of the content unchanged.

[0072] 5. Content review: Check the transcribed information for completeness and make sure that no part of the original text has been omitted.

[0073] 6. Final review: Review documents for consistent formatting and professional appearance, and ensure all information flows logically.

[0074] It should be noted that the above specific example of the third description text is only a possible example of the error correction requirement in the actual application process, and does not limit the specific content of the third description text.

[0075] In a specific implementation scenario, as a possible example, still taking the document image as a drug instruction sheet image as an example, the fourth description text about the error correction method may include but is not limited to the following content: Continuous punctuation errors: Remove unnecessary punctuation and keep only the necessary punctuation to maintain the meaning of the sentence.

[0076] Incorrect paragraph alignment: Always use left alignment unless otherwise required by regulation or professional standards.

[0077] The title and content are not clearly formatted: Use different font sizes or styles (such as bold) to distinguish them.

[0078] Improper use of list items: Formatting for clear list items but not changing the content.

[0079] Confused hierarchy: Use appropriate blank lines and indentation to make the hierarchy clear.

[0080] Incomplete or outdated information: Check the latest available medical data to ensure completeness of the information and only revise the necessary parts.

[0081] It should be noted that the above specific example of the fourth description text is only a possible example of the error correction method in actual application, and does not limit the specific content of the fourth description text.

[0082] In a specific implementation scenario, based on the above content, a prompt instruction can be constructed: Please refer to the above specified persona, error correction examples, error correction requirements and error correction methods to optimize the following content. Intentionally only output the optimized results without any explanation: Aspirin tablets Ingredients Aspirin Appearance: This product is white or almost white tablet Indications: This product is used to prevent cardiovascular embolism. 0.1g each time, once a day.

[0083] Precautions Overdose may cause anemia After inputting the above prompt instructions into the large language model, the optimized content can be obtained: Generic name: Aspirin tablets Ingredients: Aspirin Appearance: This product is white or almost white tablets Indications: This product is used to prevent embolism in cardiovascular and cerebrovascular diseases Dosage and Administration: 0.1g each time, once a day Note: If taken in excess, it may cause bleeding It should be noted that the above example is only a possible example in the actual application process, and the optimized content is not specifically limited here. In addition, the above content error correction, different forms of output and other implementation methods may be used one by one in the actual application process, or they may be arbitrarily selected and used in combination, which are not limited here.

[0084] In an implementation scenario, please refer to Figure 2d , Figure 2d FIG. 1 is a process diagram of another embodiment of the document image recognition method of the present application. Figure 2d As shown, the original document can be obtained through image acquisition to obtain a document image, and the document image can be segmented to obtain several document blocks and their types (such as text blocks, table blocks, graphic blocks, etc.), and the position information of each document block in the document image is recorded, so that the editable identification document can be restored later according to the position information of each document block. On this basis, the document blocks can be initially sorted, such as according to the possible order of writing (such as first from top to bottom, and second from left to right). Based on this, for document blocks of non-text types (such as table blocks, graphic blocks, etc.), it can be determined whether the upper and lower text blocks are titles and comments. If so, the text block can be merged into the document block and the sorting can be updated. Otherwise, the text block can be maintained. In addition, for a document block of the text type (i.e., a text block), the candidate block that should be connected to the current block (a text block other than the current block) can be determined, and after determining the candidate block that should be connected to the current block, the order is updated. If the current block is the last text block, the connection order between the text blocks can be obtained, and the connection process ends. On the contrary, if the current block is not the last text block, the connection process can continue until the current block is the last text block. The specific implementation of each step in the above process can refer to the above-mentioned related description, which will not be repeated here.

[0085] The above scheme performs image segmentation based on the document image to obtain each document block in the document image, and the document block belongs to at least text, and then performs the following steps on each document block belonging to the text type: select a document block as the current block, and select a document block belonging to the text type other than the current block as a reference block based on the target text sequence, and determine the reference block that should be connected to the current block based on the initial recognition text of the current block and the initial recognition text of each reference block, and connect any reference block with the current block as the starting point to meet the target text sequence. On this basis, at least based on the connection sequence and initial recognition text between each document block belonging to the text type, obtain the recognition document of the document image, that is, first select the reference block according to the target text sequence, and can preliminarily screen out the document block that may be connected to the current block according to the target text sequence, and then further determine it through the initial recognition text of both the current block and the reference block. Therefore, when determining the connection sequence of each document block through a two-stage process, the document block that interferes with the connection sequence can be reasonably excluded first, and then determined according to the initial recognition text. Therefore, the accuracy of the text sequence can be improved in the document image recognition process.

[0086] See also Figure 3 , Figure 3 It is a schematic diagram of the framework of an embodiment of the document image recognition device of the present application. The document image recognition device 30 includes: an image segmentation module 31, a sequence recognition module 32 and a document acquisition module 33. The image segmentation module 31 is used to perform image segmentation based on the document image to obtain each document block in the document image; wherein the category of the document block at least includes text; the sequence recognition module 32 is used to sequentially perform the following steps on each document block belonging to the category of text: select a document block as the current block, and based on the target text sequence, select a document block belonging to the category of text outside the current block as a reference block, and determine the reference block that should be connected to the current block based on the initial recognition text of the current block and the initial recognition text of each reference block; wherein, connecting any reference block with the current block as the starting point conforms to the target text sequence; the document acquisition module 33 is used to obtain the recognition document of the document image based on at least the connection sequence and the initial recognition text between each document block belonging to the category of text.

[0087] In the above scheme, the document image recognition device 30 performs image segmentation based on the document image to obtain each document block in the document image, and the document block belongs to at least text, and then performs the following steps for each document block belonging to the text type: select a document block as the current block, and select a document block belonging to the text type other than the current block as a reference block based on the target text sequence, and determine the reference block that should be connected to the current block based on the initial recognition text of the current block and the initial recognition text of each reference block, and connect any reference block with the current block as the starting point to meet the target text sequence. On this basis, at least based on the connection sequence and initial recognition text between each document block belonging to the text type, obtain the recognition document of the document image, that is, first select the reference block according to the target text sequence, and can preliminarily screen out the document block that may be connected to the current block according to the target text sequence, and then further determine it through the initial recognition text of both the current block and the reference block. Therefore, when determining the connection sequence of each document block through a two-stage process, the document block that interferes with the connection sequence can be reasonably excluded first, and then determined according to the initial recognition text. Therefore, the accuracy of the text sequence can be improved in the document image recognition process.

[0088] In some disclosed embodiments, the sequence recognition module 32 includes a first construction submodule, which is used to construct a first prompt instruction based on the initial recognition text of the current block and the initial recognition text of each reference block; wherein the first prompt instruction instructs the large language model to distinguish the reference block that should follow the current block; the sequence recognition module 32 includes a sequence determination submodule, which is used to determine the reference block that should follow the current block based on the output result of the large language model triggered by the first prompt instruction.

[0089] In some disclosed embodiments, the document image recognition device 30 also includes a block selection module for selecting each document block of the non-text category as a target block in turn, and selecting a document block adjacent to the target block and of the text category as a candidate block when the category to which the document block belongs also includes non-text; the document image recognition device 30 also includes an attachment selection module for selecting a candidate block attached to the target block as an attachment block based on the initial recognition text of the target block and the initial recognition text of each candidate block; the document image recognition device 30 also includes a block merging module for merging the candidate block as an attachment block into the target block to obtain a new target block; wherein the non-text includes at least one of a graphic and a table, and the initial recognition text in the candidate block attached to the target block represents any one of a title and a comment of the target block.

[0090] In some disclosed embodiments, the subsidiary selection module includes a second construction submodule, which is used to construct a second prompt instruction based on the initial recognition text of the target block and the initial recognition text of each candidate block; wherein the second prompt instruction instructs the large language model to distinguish whether the initial recognition text of the candidate block is any of the title and comments of the target block; the subsidiary selection module includes a subsidiary determination submodule, which is used to determine the candidate block attached to the target block as a subsidiary block based on the output result of the large language model triggered by the second prompt instruction.

[0091] In some disclosed embodiments, the block selection module is specifically used to select a document block below the target block and belonging to the type of text as a first candidate block; the subsidiary selection module includes a first discrimination submodule, which is used to obtain a first discrimination result of the first candidate block based on the initial recognition text of the target block and the initial recognition text of each first candidate block; wherein the first discrimination result includes any one of the following: the initial recognition text of the first candidate block as the title of the target block, the initial recognition result of the first candidate block as the annotation of the target block; the subsidiary selection module includes a second discrimination submodule, which is used to determine whether to continue to select the document block above the target block and belonging to the type of text as a second candidate block for subsidiary discrimination based on whether the first discrimination result includes the first candidate block as the title.

[0092] In some disclosed embodiments, the second discrimination submodule includes a first response unit for, when there is a first discrimination result including a first candidate block as a title, in response to the fact that only the first discrimination result includes the first candidate block as the title, selecting the first discrimination result including the first candidate block as the title as an auxiliary block and terminating the auxiliary discrimination; the second discrimination submodule includes a second response unit for, when there is a first discrimination result including the first candidate block as the title, in response to the fact that the first discrimination result also includes the first candidate block as a comment, selecting both the first discrimination result including the first candidate block as the title and the first discrimination result including the first candidate block as the comment as auxiliary blocks and terminating the auxiliary discrimination.

[0093] In some disclosed embodiments, the second discrimination submodule includes an execution determination unit for determining, when there is no first discrimination result including the first candidate block as a title, whether to obtain a second discrimination result of the second candidate block based on the initial recognition text of the target block and the initial recognition texts of each second candidate block after selecting the first candidate block including the comment as a subsidiary block in the first discrimination result based on whether there is still a first discrimination result including the first candidate block as a comment; the second discrimination submodule includes a selection discrimination unit for determining, based on the second discrimination result of the second candidate block, whether to select the second candidate block as a subsidiary block of the target block; wherein the second discrimination result includes: whether the initial recognition text of the second candidate block is used as the title of the target block.

[0094] In some disclosed embodiments, the document image recognition device 30 also includes a description acquisition module, which is used to acquire a first description text about a specified persona, a second description text about an error correction example, a third description text about an error correction requirement, and a fourth description text about an error correction method; the document acquisition module 33 includes an initial acquisition submodule, which is used to sequentially splice the initial recognition texts of at least each document block of the text type based on a connection order to obtain an initial document of the document image; the document acquisition module 33 includes a third construction submodule, which is used to construct a third prompt instruction based on the first description text, the second description text, the third description text, the fourth description text and the initial document; wherein the third prompt instruction instructs the large language model to play the role of the specified persona and refer to the error correction example to correct the initial document according to the error correction requirement using the error correction method; the document acquisition module 33 includes a document acquisition submodule, which is used to obtain a recognized document based on the output result of the large language model triggered by the third prompt instruction.

[0095] In some disclosed embodiments, the document image includes at least an image of a drug instruction sheet; and / or the target text sequence is a possible text sequence of an original document to which the document image belongs.

[0096] See also Figure 4 , Figure 4 4 is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other, the memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above document image recognition method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include but is not limited to a server, a smart phone, a tablet computer, etc., and the specific type of the electronic device 40 is not limited here.

[0097] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above document image recognition method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0098] In the above scheme, the electronic device 40 performs image segmentation based on the document image to obtain each document block in the document image, and the document block belongs to at least text, and then performs the following steps for each document block belonging to the text type: select a document block as the current block, and select a document block belonging to the text type other than the current block as a reference block based on the target text sequence, and determine the reference block that should be connected to the current block based on the initial recognition text of the current block and the initial recognition text of each reference block, and connect any reference block with the current block as the starting point to meet the target text sequence. On this basis, at least based on the connection sequence and initial recognition text between each document block belonging to the text type, obtain the recognition document of the document image, that is, first select the reference block according to the target text sequence, and can preliminarily screen out the document block that may be connected to the current block according to the target text sequence, and then further determine it through the initial recognition text of both the current block and the reference block. Therefore, when determining the connection sequence of each document block through a two-stage process, the document block that interferes with the connection sequence can be reasonably excluded first, and then determined according to the initial recognition text. Therefore, the accuracy of the text sequence can be improved in the document image recognition process.

[0099] See also Figure 5 , Figure 5 The figure is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps in any of the above document image recognition method embodiments.

[0100] In the above scheme, the computer-readable storage medium 50 performs image segmentation based on the document image to obtain each document block in the document image, and the document block belongs to at least text, and then performs the following steps on each document block belonging to the text type: select a document block as the current block, and select a document block belonging to the text type other than the current block as a reference block based on the target text sequence, and determine the reference block that should be connected to the current block based on the initial recognition text of the current block and the initial recognition text of each reference block, and connect any reference block with the current block as the starting point to meet the target text sequence. On this basis, at least based on the connection sequence and initial recognition text between each document block belonging to the text type, obtain the recognition document of the document image, that is, first select the reference block according to the target text sequence, and can preliminarily screen out the document block that may be connected to the current block according to the target text sequence, and then further determine it through the initial recognition text of both the current block and the reference block. Therefore, when determining the connection sequence of each document block through a two-stage process, the document block that interferes with the connection sequence can be reasonably excluded first, and then determined according to the initial recognition text. Therefore, the accuracy of the text sequence can be improved in the document image recognition process.

[0101] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0102] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0103] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0104] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0105] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0107] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A document image recognition method, characterized in that: include: Performing image segmentation based on the document image to obtain various document blocks in the document image; wherein the document blocks belong to at least a text type; The following steps are sequentially performed for each document block of the text type: the document block is selected as the current block, and based on the target text sequence, a document block of the text type other than the current block is selected as a reference block, and based on the initial recognition text of the current block and the initial recognition text of each reference block, a reference block that should be connected to the current block is determined; wherein, connecting any of the reference blocks with the current block as the starting point conforms to the target text sequence; A recognition document of the document image is obtained based at least on the connection sequence between the document blocks belonging to the text category and the initial recognition text.

2. The method according to claim 1, characterized in that: The determining, based on the initial recognition text of the current block and the initial recognition text of each of the reference blocks, a reference block that should be subsequent to the current block comprises: Based on the initial recognition text of the current block and the initial recognition text of each of the reference blocks, construct a first prompt instruction; wherein the first prompt instruction instructs the large language model to distinguish the reference block that should follow the current block; Based on the output result of the large language model triggered by the first prompt instruction, a reference block that should be subsequent to the current block is determined.

3. The method according to claim 1, characterized in that In the case that the document block also belongs to a non-text category, before sequentially performing the following steps on each document block belonging to the text category, the method further includes: Selecting each document block of non-text type as a target block in turn, and selecting a document block of text type adjacent to the target block as a candidate block; Based on the initial recognition text of the target block and the initial recognition text of each of the candidate blocks, selecting a candidate block attached to the target block as an attached block; The candidate block as the subsidiary block is merged into the target block to obtain a new target block; wherein the non-text includes at least one of a graphic and a table, and the initial recognition text in the candidate block attached to the target block represents any one of a title and a comment of the target block.

4. The method according to claim 3, characterized in that The selecting, based on the initial recognition text of the target block and the initial recognition text of each of the candidate blocks, the candidate block attached to the target block as the attached block comprises: Based on the initial recognition text of the target block and the initial recognition text of each of the candidate blocks, construct a second prompt instruction; wherein the second prompt instruction instructs the large language model to distinguish whether the initial recognition text of the candidate block is any one of the title and the comment of the target block; Based on the output result of the large language model triggered by the second prompt instruction, a candidate block attached to the target block is determined as an attached block.

5. The method according to claim 3, characterized in that: The step of selecting a document block adjacent to the target block and belonging to a text type as a candidate block includes: Selecting a document block below the target block and belonging to the text type as a first candidate block; The selecting, based on the initial recognition text of the target block and the initial recognition text of each of the candidate blocks, the candidate block attached to the target block as the attached block comprises: Based on the initial recognition text of the target block and the initial recognition text of each of the first candidate blocks, a first discrimination result of the first candidate block is obtained; wherein the first discrimination result includes any one of the following: the initial recognition text of the first candidate block is used as the title of the target block, and the initial recognition result of the first candidate block is used as the annotation of the target block; Based on whether the first discrimination result includes a first candidate block as a title, it is determined whether to continue selecting a document block above the target block and belonging to a text type as a second candidate block for subsidiary discrimination.

6. The method according to claim 5, characterized in that When there is a first candidate block as the title in the first discrimination result, the method includes: In response to there being only the first candidate block including the first candidate block as the title as a result of the first discrimination, selecting the first candidate block including the first candidate block as the title as a subsidiary block, and ending subsidiary discrimination; In response to the first candidate block including the comment as the first discrimination result still existing, the first candidate block including the title as the first discrimination result and the first candidate block including the comment as the first discrimination result are both selected as the subsidiary blocks, and the subsidiary discrimination is ended.

7. The method according to claim 5, characterized in that When there is no first candidate block included in the first discrimination result as the title, the method includes: Based on whether there is still a first candidate block including the annotation in the first discrimination result, determining whether to obtain a second discrimination result of the second candidate block based on the initial recognition text of the target block and the initial recognition text of each of the second candidate blocks after selecting the first candidate block including the annotation in the first discrimination result as the subsidiary block; Based on a second discrimination result of the second candidate block, determine whether to select the second candidate block as a subsidiary block of the target block; wherein the second discrimination result includes: whether the initial recognition text of the second candidate block is used as the title of the target block.

8. The method according to claim 1, characterized in that Before obtaining the recognized document of the document image based at least on the connection sequence between the document blocks of the text type and the initial recognized text, the method further comprises: Obtaining a first description text about a specified persona, a second description text about an error correction example, a third description text about an error correction requirement, and a fourth description text about an error correction method; The step of obtaining the recognition document of the document image based on at least the connection sequence between the document blocks of the text type and the initial recognition text comprises: Sequentially splicing at least the initial recognized texts of each document block belonging to the text type based on the connection sequence to obtain the initial document of the document image; Based on the first description text, the second description text, the third description text, the fourth description text and the initial document, a third prompt instruction is constructed; wherein the third prompt instruction instructs the large language model to play the specified persona and refer to the correction example to correct the initial document according to the correction requirement and adopt the correction method; The recognition document is obtained based on the output result of the large language model triggered by the third prompt instruction.

9. The method according to any one of claims 1 to 8, characterized in that: The document image at least includes a drug instruction sheet image; And / or, the target text sequence is a possible text sequence of the original document to which the document image belongs.

10. A document image recognition device, characterized in that: include: An image segmentation module, used for performing image segmentation based on a document image to obtain various document blocks in the document image; wherein the document blocks belong to at least a type of text; The sequential recognition module is used to sequentially perform the following steps on each document block of the text type: select the document block as the current block, and based on the target text sequence, select a document block of the text type other than the current block as a reference block, and determine a reference block that should be connected to the current block based on the initial recognition text of the current block and the initial recognition text of each reference block; wherein connecting any of the reference blocks with the current block as the starting point conforms to the target text sequence; The document acquisition module is used to obtain the recognition document of the document image based on at least the connection sequence between each document block belonging to the text type and the initial recognition text.

11. An electronic device, characterized in that: The document image recognition method comprises at least a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the document image recognition method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the document image recognition method according to any one of claims 1 to 9.

Images