Visual rich document information extraction method and device for drug instruction field

Abstract

The application discloses a visual rich document information extraction method and device for the field of drug instruction, which comprises the following steps: firstly, a semi-automatic data set is constructed, that is, drug instruction pictures are acquired, and format-compliant documents are stored in pictures; secondly, italic data set is manually labeled, and a target detection model is trained to carry out picture cutting; thirdly, a PaddleOCR model is trained by using the picture cutting data set to obtain a word table; then, the drug instruction pictures are labeled by using PPOCRLabel, unreasonable places are manually adjusted, special texts are marked with symbols, and the final data set containing images and corresponding JSON files is generated based on the word table; and finally, the final data set is fine-tuned on a multi-modal pre-training language model LayoutLM to realize information extraction, that is, a LayoutLM multi-modal fusion model is obtained by fusing text, position, image and italic picture features; entity matching results are obtained according to word embedding and visual features, key-value pair entity classification results are determined by calculating semantic similarity, and thus, key information of the drug instruction is extracted.

Description

Technical Field

[0001] This invention relates to the field of multimodal information extraction technology, which integrates NLP (Natural Language Processing) and computer vision in the field of computer science and technology, and in particular to a method and apparatus for extracting rich visual document information for the field of drug instructions. Background Technology

[0002] Information extraction, a crucial task in Natural Language Processing (NLP), aims to extract specific information from structured or semi-structured documents and images. In the digital age, these documents often contain rich multimodal information, encompassing text, layout, and image information. The essence of multimodal technology is to leverage multimodal joint modeling, enabling computers to acquire information from multiple domains, interactively fuse effective information, and ultimately derive a final decision based on feature parameters. Practice has shown that conclusions derived through multimodal fusion are more accurate and exhibit superior model performance compared to single-modal methods.

[0003] Currently, common multimodal information extraction methods are mainly based on several major categories. However, most of these methods only consider the image information corresponding to the text location in a general way. Specifically, they usually use an OCR (Optical Character Recognition) engine to obtain the text content and its spatial location in the document image, and then organize the text content in a left-to-right and top-to-bottom order.

[0004] While some methods attempt to enhance the model's adaptability to visually rich documents with complex and varied layouts by encoding the spatial location of text and the layout information of images, they generally neglect key information such as font weight and size. In fact, this font information has significant value and, if used appropriately, could further improve the accuracy and efficiency of information extraction. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for visual rich document information extraction in the field of drug instructions, so as to build a one-stop framework that integrates the construction of drug instructions-specific domain data and multimodal information extraction. This framework effectively alleviates the predicament of the scarcity of open source datasets in the multimodal domain, and successfully solves the problem of recognition errors caused by the lack of utilization of key features.

[0006] To achieve the above objectives, this invention provides a method for extracting visual rich document information in the field of drug instructions, comprising the following steps:

[0007] Step 1: Construct a dataset for the drug instruction manual domain using a semi-automated method, specifically including:

[0008] S1, Obtain images of drug instruction manuals: Collect and organize drug instruction manual documents published online, check whether the format of each page of the document meets the requirements, and if it does, convert the document into an image and save it in the specified folder;

[0009] S2, Use an object detection model to detect bold italics and cut out the image: Manually create labeled categories, label the bold italics dataset, train the bold italics dataset, and cut out the image in the original image based on the recognition results to obtain the cut-out dataset;

[0010] S3, train the PaddleOCR model to perform text detection on the image matting dataset to obtain a vocabulary;

[0011] S4, Annotate drug instruction manuals and generate the final dataset: Annotate drug instruction manual images using PPOCRLabel, and save the four coordinates of the text and annotation boxes. Manually adjust for missing annotations, text errors, and unreasonable annotation box positions. Save the adjusted annotation results in a txt file. For question category text that is too long or lacks obvious image features, add a specific symbol before the question category text content. When building the dataset later, first determine whether there are labels. If there are, classify it into question category and remove the specific symbols. Generate the final dataset based on the vocabulary. This dataset contains image files and corresponding JSON files containing image text detection annotation boxes and category labels.

[0012] Step 2: Fine-tune the multimodal pre-trained language model LayoutLM using the final dataset from Step 1 to achieve the information extraction task, specifically:

[0013] S5, by fusing the text features and positional encoding features contained in the JSON file with the image features and bold italic image features extracted by the convolutional neural network through the LayoutLM model, the LayoutLM multimodal fusion model is obtained;

[0014] S6. Based on the visual features of the word embedding model and document graph, obtain entity matching results with text matching relationships of question category and answer category. Calculate semantic similarity between the text semantic features of question category and answer category and all categories to be extracted. Select the category with the highest semantic similarity as the entity classification result of the key-value pair to obtain the key information in the drug instructions.

[0015] Further, in step S2, the LabelImg software is used manually to create annotation categories and annotate the bold italic dataset. During annotation, the target location is selected by "creating a block", the target category label is set, and each annotated image from step S1 is saved as an XML file. The size and position of the annotation box are represented by the annotation box.

[0016] Furthermore, in step S3, when labeling targets in multiple categories, the priority labeling rules include: brackets are labeled with the first priority, bold is labeled with the second priority, italics are labeled with the third priority, and underline is labeled with the fourth priority.

[0017] Furthermore, in step S6, key information includes, for example, the drug name, ingredients, adverse reactions, indications, and method of administration.

[0018] Furthermore, in step S6, the visual features include information about the font and bracket style.

[0019] This invention also provides a visual rich document information extraction device for the field of drug instructions, comprising:

[0020] The dataset construction module is used to construct datasets in the field of drug instruction manuals using semi-automatic methods, specifically including:

[0021] The drug instruction manual image acquisition unit is used to collect and organize drug instruction manual documents published online, check whether the format of each page of the document meets the requirements, and if it does, convert the document into an image and save it in a designated folder.

[0022] The detection and matting unit is used to manually create labeled categories, label the bold italic dataset, train on the bold italic dataset, and perform matting on the original image based on the recognition results to obtain the matted dataset;

[0023] The vocabulary acquisition unit is used to train the PaddleOCR model to perform text detection on the image matting dataset and obtain the vocabulary.

[0024] The drug instruction manual dataset annotation unit is used to annotate drug instruction manual images using PPOCRLabel, and save the four-point coordinates of the text and annotation boxes. It manually adjusts for missing annotations, text errors, and unreasonable annotation box positions. The adjusted annotation results are saved in a txt file. For question category text that is too long or lacks obvious image features, a specific symbol is added before the question category text content. When building the dataset later, it first determines whether there are labels. If there are, it is classified into question category and the specific symbols are removed. The final dataset is generated based on the vocabulary. This dataset contains image files and corresponding JSON files containing image text detection annotation boxes and category labels.

[0025] The key information extraction module is used to fine-tune the multimodal pre-trained language model LayoutLM using the final dataset from step one to achieve the information extraction task. Specifically, it includes:

[0026] The visual feature extraction unit is used to fuse the text features and positional encoding features contained in the JSON file with the image features and bold italic image features extracted by the convolutional neural network to obtain the LayoutLM multimodal fusion model.

[0027] The key information acquisition unit of the drug instruction manual is used to obtain entity matching results with text matching relationships of question category and answer category based on the visual features of word embedding model and document graph. Based on the text semantic features of question category and answer category, semantic similarity is calculated with all categories to be extracted. The category with the highest semantic similarity is selected as the entity classification result of key-value pair to obtain the key information in the drug instruction manual.

[0028] Furthermore, in the detection and matting unit, the LabelImg software is used manually to create annotation categories and annotate the bold italic dataset. During annotation, the target location is selected by "creating a block", the target category label is set, and each annotated image in step S1 is saved as an XML file. The size and position of the annotation box are represented by the annotation box.

[0029] Furthermore, in the vocabulary acquisition unit, when labeling multiple categories of targets, the priority labeling rules include: brackets are labeled with the first priority, bold is labeled with the second priority, italics are labeled with the third priority, and underline is labeled with the fourth priority.

[0030] Furthermore, in the key information acquisition unit of the drug instruction manual, key information includes, for example, the drug name, ingredients, adverse reactions, indications, and method of administration.

[0031] Furthermore, in the key information acquisition unit of the drug instruction manual, visual features include information on font and bracket style.

[0032] The present invention has the following advantages due to the adoption of the above technical solutions:

[0033] 1. Because this invention proposes a semi-automatic data construction method for drug instruction manual datasets, which integrates font size and bracket style features, it not only meets the demand for high-quality data in drug instruction manual information extraction, but also provides a valuable reference example for information extraction of visual rich text implementation files in other fields, which helps to promote the common development of information extraction technologies in multiple fields.

[0034] 2. The LayoutLM model used in this invention organically integrates text and location information, and further integrates visual information on this basis, which is then seamlessly applied to downstream tasks. This information fusion method integrates multimodal information, significantly enhancing the model's ability to understand and process complex information, thereby improving the overall accuracy and reliability of information extraction.

[0035] 3. This invention addresses the field of drug instruction manuals. It addresses the unique characteristics of semi-structured document content, such as drug names, ingredients, adverse reactions, indications, and methods of administration, presented in bold and bracketed form. By fully utilizing key features such as font styles and brackets, it accurately identifies and extracts this important information, effectively solving the identification errors caused by traditional methods failing to utilize these key features. This provides a solid guarantee for the accurate extraction of information from drug instruction manuals. Attached Figure Description

[0036] Figure 1 This is a flowchart of a semi-structured visual rich document information extraction method based on multimodal information fusion according to an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of a classification ontology library in the field of drug instructions in an embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram of a general-purpose question in an embodiment of the present invention.

[0039] Figure 4 This is a schematic diagram of a non-general type of question in an embodiment of the present invention.

[0040] Figure 5 This is a schematic diagram of a bold italic image identified using YOLOv5s in an embodiment of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] To address the issues of missing open-source datasets in the field of drug instruction manuals and entity recognition errors caused by the failure to utilize key features such as font and bracket styles, such as... Figure 1 As shown in the figure, this embodiment proposes a pipeline-based method for extracting information from drug instruction manuals, including:

[0044] Step 1: Construct a dataset for the drug instruction manual domain using a semi-automated method.

[0045] Step 2: Fine-tune the information extraction task using the final dataset from Step 1 on the multimodal pre-trained language model LayoutLM.

[0046] In one embodiment, step one specifically includes:

[0047] S1, Obtain the image of the drug instruction manual.

[0048] We collect and organize drug instruction leaflets published online. The length of these leaflets varies, typically ranging from 10 to 30 pages, but is not limited to this. The collected and organized drug instruction leaflets are usually presented in PDF (Portable Document Format) format.

[0049] Each collected and organized drug instruction manual is opened, and its document format is checked to ensure it meets requirements. A standardized rasterization engine is used to render the PDF pages as raster image files with a preset resolution, typically set to 300 DPI (Dots Per Inch). This is an industry-recognized standard for balancing image detail and file size. If the requirements are met, the document is converted into an image and stored in a designated folder, thus obtaining the drug instruction manual image. In this embodiment, the preferred image format is PNG (Portable Network Graphics).

[0050] S2 uses an object detection model to detect bold italics and cut out the image.

[0051] Since the text categories in drug instruction leaflet images typically appear in pairs—question and answer—with questions being the minority, priority is given to labeling the question category, and the remaining categories are then assigned to the answer category. The category to which the text content of the answer category belongs can be determined based on the text content of the question category.

[0052] Drug instructions are highly structured, rich text documents containing a wealth of formatted information. Text in bold, italic, underlined, and bracketed fonts can typically be categorized. For example... Figure 2 As shown, Figure 2This is an example of a classification ontology in the field of drug instruction manuals. In addition, for each different drug, there are categories that are difficult to categorize and require manual annotation.

[0053] If existing multimodal prototype models are used without considering the features of bold and italic fonts, these categories often cannot be classified well. Conversely, utilizing the features of bold and italic fonts can better distinguish them.

[0054] Based on the above analysis, the question categories can be further divided into two main categories.

[0055] One type is general questions (such as...) Figure 3 The general question class in the text is called `question`, and the other type is a non-general question class (such as...). Figure 4 (The non-generic class `question` in the text). Among them:

[0056] General-purpose questions often have obvious image features, such as bold, italic, underlines, and brackets. As legally mandated formats, these image features tend to appear frequently. A model can be trained to recognize bold and italic text in images.

[0057] Non-general questions arise because each drug's instructions differ in content, chemical agents, and special ingredients. Therefore, each drug often has a specific question category. This type of text may or may not have special image features, but it is characterized by infrequent appearance and longer length. For non-general text, manual annotation using annotation software is necessary.

[0058] Given the difficulty in creating an end-to-end model that can both detect bold and italic text and perform character recognition, and the scarcity of datasets suitable for this task, it was decided to split the task into two sub-tasks to reduce its complexity: an object detection model and a character recognition model.

[0059] In one embodiment, the object detection model and the text recognition model are trained using a YOLOv5s model.

[0060] The training dataset was entirely created manually, using LabelImg software to create the annotation categories. LabelImg is a very convenient and user-friendly image annotation tool that allows users to create annotation categories manually. During annotation, the target location is selected through "Create Block," the target category label is set, and each annotated image from step S1 is saved as an XML (eXtensible Markup Language) file. In the XML file, the size and position of the annotation box are represented by the coordinates of its top-left and bottom-right points.

[0061] In the annotation process, situations arise where targets fall into multiple categories. For example, many titles contain both brackets and bold text. To reduce the computational cost of annotation, and since the model's goal is to detect text, and the correctness of the detected category is not currently required, only one category needs to be labeled in such cases. The priority rules for labeling targets with multiple categories are as follows: Brackets are given the highest priority because they are more obvious features than bold, italic, and underline; bold is given the second highest priority because it is easier to identify; italic is given the third highest priority; and underline is given the fourth highest priority.

[0062] Based on the aforementioned priority annotation rules, a bold italic text dataset is obtained. This invention uses the YOLOv5s model to train and recognize bold italic text images in the dataset, thereby training a text recognition model. Specifically, the dataset format used by the YOLOv5s model is different from the XML format used for annotation storage. Therefore, before training, the XML format used for annotation storage is converted to the dataset format used by the YOLOv5s model according to its requirements.

[0063] like Figure 5 As shown, Figure 5 To obtain the target front image using the best YOLOv5s training model weight file, we can see that most of the bold, italic, underline, and bracket text were recognized. Most of the recognized image text is the question category required by the YOLOv5s model, thus achieving the purpose of this project.

[0064] Finally, using existing Python code, image matting is performed on the original images based on the recognition results, thus obtaining the matted dataset. Python is a high-level, general-purpose, interpreted programming language that emphasizes code readability, enabling developers to implement complex functions with less code; therefore, it is often referred to as "executable pseudocode."

[0065] S3, Training the PaddleOCR model to obtain the vocabulary: The lightweight open-source model PaddleOCR was selected, and the open-source datasets XFUN and S2's image matting dataset were used. At the same time, the pre-trained model for Chinese recognition provided by PaddleOCR was used for training. Text detection was performed on the image matting dataset to obtain the vocabulary.

[0066] S4, Labeling drug instructions and generating the final dataset: Use PPOCRLabel to label the drug instruction images from step S3, and generate JSON files of image text detection annotation boxes and category labels. Manually adjust for omissions, text errors, unreasonable annotation box positions, etc., so as to facilitate subsequent classification based on labels, thereby enabling the extraction of key information from drug instructions based on relevant terms.

[0067] After adjustment, the annotation results are saved in a txt (Text File) file. For excessively long question-category text or text lacking obvious image features, a specific symbol, such as "#", is added before the question-category text content to indicate that the text belongs to the question category. When building the dataset later, it is prioritized to determine if there are labels; if so, the text is classified as a question category, and the "#" symbol is removed. The final dataset is generated based on the vocabulary. This dataset contains image files and corresponding JSON files containing image text detection bounding boxes and category labels. JSON stands for JavaScript Object Notation, a lightweight, easy-to-read, and easy-to-write data exchange format mainly used for storing and transmitting structured data (such as key-value pairs, lists, nested data, etc.).

[0068] In one embodiment, step two specifically includes:

[0069] S5 extracts feature maps of visual images and visual features at text locations using a convolutional neural network.

[0070] Specifically, a model pre-trained on the ImageNet large-scale classification dataset using a residual network is used to extract visual image feature maps from the visual rich documents obtained from steps S1-S4. ImageNet is a massive and highly influential image database in the field of computer vision. The ImageNet large-scale classification dataset is currently the largest database for image recognition.

[0071] Then, RoIAlign (Region of Interest Align) is used to extract the visual features of the corresponding text based on the spatial location of each text segment in the dataset obtained in step S4. RoIAlign is a key algorithm in the field of computer vision, mainly used to solve the feature alignment problem in object detection and instance segmentation tasks.

[0072] Residual networks are used to address the network degradation problem caused by too many hidden layers in deep neural networks. They allow the neural network to learn potential residual mappings rather than identity mappings.

[0073] The LayoutLM model is used to fuse the text features and positional encoding features contained in the JSON file, as well as the image features and bold italic image features extracted by the convolutional neural network in S5.

[0074] S6. Based on the word embedding model and visual features of the document graph from step S5, which include information about font and bracket styles, entity matching results with text matching relationships of question and answer categories are obtained. Following the classification method in S2, YOLOv5s is used to identify text blocks with visual features such as bold italics and brackets in the drug instructions. Combined with PaddleOCR to recognize text content, high-frequency texts are selected to construct a general-purpose question dictionary. Simultaneously, semantic boundaries are defined: the question category consists of concise attribute category words (e.g., "specifications"), and the answer category consists of specific information within the corresponding category (e.g., "3.5mg / bottle"), serving as the basis for semantic similarity calculation. Referring to the legally mandated content of the drug instructions compiled in S2, the core categories to be extracted are determined, including drug name, ingredients, indications, etc., to ensure matching with medication needs. For the text to be classified, semantic similarity is calculated with the categories in the general-purpose question dictionary. The category with the highest similarity is selected to initially determine if it is a question. If it is a question, the corresponding answer text is matched to determine key-value pairs. If the similarity is low, manual assistance is provided to complete entity classification.

[0075] In step two, multimodal information such as brackets and font styles (e.g., bold italics) is used to extract visual rich document information in the pharmaceutical field. This effectively solves the problems of recognition errors caused by the lack of open-source datasets in the field of visual rich documents and the failure to utilize key features such as bold italics and bracket styles, thus supplementing existing visual rich document information extraction methods.

[0076] This invention provides a visual rich document information extraction device for the field of drug instructions, which includes a dataset construction module and a key information extraction module. The dataset construction module is used to construct a dataset for the field of drug instructions using a semi-automatic method. The key information extraction module is used to fine-tune the final dataset from step one on a multimodal pre-trained language model LayoutLM to achieve the information extraction task.

[0077] The dataset construction module specifically includes a drug instruction manual image acquisition unit, an image detection and matting unit, a vocabulary acquisition unit, and a drug instruction manual dataset annotation unit.

[0078] The drug instruction manual image acquisition unit is used to collect and organize drug instruction manual documents published online, check whether the format of each page of the document meets the requirements, and if it does, convert the document into an image and save it in a designated folder.

[0079] The detection and matting unit is used to manually create labeled categories, label the bold italic dataset, train on the bold italic dataset, and perform matting on the original image based on the recognition results to obtain the matted dataset.

[0080] The vocabulary acquisition unit is used to train the PaddleOCR model to perform text detection on the image matting dataset and obtain the vocabulary.

[0081] The drug instruction manual dataset annotation unit is used to annotate drug instruction manual images using PPOCRLabel. PPOCRLabel stores the four coordinates of the text and annotation boxes. Manual adjustments are made for missing annotations, text errors, and unreasonable annotation box positions. The adjusted annotation results are saved in a txt file. For excessively long or question-category text without obvious image features, a "#" is added before the question-category text content to indicate that the text belongs to the question category. When building the dataset later, it is first determined whether there are labels; if so, it is classified as a question category, and the "#" is removed. The final dataset is generated based on the vocabulary. This dataset contains image files and corresponding JSON files containing image text detection annotation boxes and category labels.

[0082] The key information extraction module specifically includes a visual feature extraction unit and a key information acquisition unit for drug instructions. The visual feature extraction unit uses a LayoutLM model to fuse text features and positional encoding features contained in the JSON file, along with image features and bold italic image features extracted by a convolutional neural network, to obtain a LayoutLM multimodal fusion model. The key information acquisition unit for drug instructions uses visual features from a word embedding model and document graph to obtain entity matching results with text matching relationships of question and answer categories. Based on the semantic features of the question and answer categories, semantic similarity is calculated with all categories to be extracted. The category with the highest semantic similarity is selected as the entity classification result for the key-value pair to obtain the key information in the drug instructions.

[0083] In one embodiment, in the detection and matting unit, the LabelImg software is used manually to create annotation categories and annotate the bold italic dataset. During annotation, the target location is selected by "creating a block", the target category label is set, and each annotated image in step S1 is saved as an XML file. In the XML file, the size and position of the annotation box are represented by the coordinates of the upper left and lower right points of the annotation box.

[0084] In one embodiment, when the vocabulary acquisition unit labels targets in multiple categories, the priority labeling rules include: since brackets are more obvious features than bold, italic, and underline, the labeling of brackets has the first priority; since bold is easier to identify, the labeling of bold has the second priority; italic has the third priority; and underline has the fourth priority.

[0085] In one embodiment, the key information in the drug instruction manual key information acquisition unit includes, for example, drug name, ingredients, adverse reactions, indications, and method of administration.

[0086] In one embodiment, the visual features in the key information acquisition unit of the drug instruction manual include information on font and bracket style.

[0087] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for extracting visual rich document information for drug instruction manuals, characterized in that, Includes the following steps: Step 1: Construct a dataset for the drug instruction manual domain using a semi-automated method, specifically including: S1, Obtain images of drug instruction manuals: Collect and organize drug instruction manual documents published online, check whether the format of each page of the document meets the requirements, and if it does, convert the document into an image and save it in the specified folder; S2, Use an object detection model to detect bold italics and cut out the image: Manually create labeled categories, label the bold italics dataset, train the bold italics dataset, and cut out the image in the original image based on the recognition results to obtain the cut-out dataset; S3, train the PaddleOCR model to perform text detection on the image matting dataset to obtain a vocabulary; S4, Annotate drug instruction manuals and generate the final dataset: Annotate drug instruction manual images using PPOCRLabel, and save the four coordinates of the text and annotation boxes. Manually adjust for missing annotations, text errors, and unreasonable annotation box positions. Save the adjusted annotation results in a txt file. For question category text that is too long or lacks obvious image features, add a specific symbol before the question category text content. When building the dataset later, first determine whether there are labels. If there are, classify it into question category and remove the specific symbols. Generate the final dataset based on the vocabulary. This dataset contains image files and corresponding JSON files containing image text detection annotation boxes and category labels. Step 2: Fine-tune the multimodal pre-trained language model LayoutLM using the final dataset from Step 1 to achieve the information extraction task, specifically: S5, by fusing the text features and positional encoding features contained in the JSON file with the image features and bold italic image features extracted by the convolutional neural network through the LayoutLM model, the LayoutLM multimodal fusion model is obtained; S6. Based on the visual features of the word embedding model and document graph, obtain entity matching results with text matching relationships of question category and answer category. Calculate semantic similarity between the text semantic features of question category and answer category and all categories to be extracted. Select the category with the highest semantic similarity as the entity classification result of the key-value pair to obtain the key information in the drug instructions.

2. The visual rich document information extraction method for the field of drug instruction manuals as described in claim 1, characterized in that, In step S2, the LabelImg software is used manually to create annotation categories and annotate the bold italic dataset. During annotation, the target location is selected by "creating a block", the target category label is set, and each annotated image from step S1 is saved as an XML file. The size and position of the annotation box are represented by the annotation box.

3. The visual rich document information extraction method for the field of drug instruction manuals as described in claim 1 or 2, characterized in that, In step S3, when labeling targets in multiple categories, the priority labeling rules include: brackets are labeled with the first priority, bold is labeled with the second priority, italics are labeled with the third priority, and underline is labeled with the fourth priority.

4. The visual rich document information extraction method for the field of drug instruction manuals as described in claim 1 or 2, characterized in that, In step S6, key information includes, for example, the drug name, ingredients, adverse reactions, indications, and method of administration.

5. The method for extracting visual rich document information for the field of drug instructions as described in claim 1 or 2, characterized in that, In step S6, the visual features include information about the font and bracket style.

6. A visual rich document information extraction device for the field of drug instruction manuals, characterized in that, include: The dataset construction module is used to construct datasets in the field of drug instruction manuals using semi-automatic methods, specifically including: The drug instruction manual image acquisition unit is used to collect and organize drug instruction manual documents published online, check whether the format of each page of the document meets the requirements, and if it does, convert the document into an image and save it in a designated folder. The detection and matting unit is used to manually create labeled categories, label the bold italic dataset, train on the bold italic dataset, and perform matting on the original image based on the recognition results to obtain the matted dataset; The vocabulary acquisition unit is used to train the PaddleOCR model to perform text detection on the image matting dataset and obtain the vocabulary. The drug instruction manual dataset annotation unit is used to annotate drug instruction manual images using PPOCRLabel, and save the four-point coordinates of the text and annotation boxes. It manually adjusts for missing annotations, text errors, and unreasonable annotation box positions. The adjusted annotation results are saved in a txt file. For question category text that is too long or lacks obvious image features, a specific symbol is added before the question category text content. When building the dataset later, it first determines whether there are labels. If there are, it is classified into question category and the specific symbols are removed. The final dataset is generated based on the vocabulary. This dataset contains image files and corresponding JSON files containing image text detection annotation boxes and category labels. The key information extraction module is used to fine-tune the multimodal pre-trained language model LayoutLM using the final dataset from step one to achieve the information extraction task. Specifically, it includes: The visual feature extraction unit is used to fuse the text features and positional encoding features contained in the JSON file with the image features and bold italic image features extracted by the convolutional neural network to obtain the LayoutLM multimodal fusion model. The key information acquisition unit of the drug instruction manual is used to obtain entity matching results with text matching relationships of question category and answer category based on the visual features of word embedding model and document graph. Based on the text semantic features of question category and answer category, semantic similarity is calculated with all categories to be extracted. The category with the highest semantic similarity is selected as the entity classification result of key-value pair to obtain the key information in the drug instruction manual.

7. The visual rich document information extraction device for the field of drug instructions as described in claim 6, characterized in that, In the detection and matting unit, the LabelImg software is used manually to create annotation categories and annotate the bold italic dataset. During annotation, the target location is selected by "creating a block", the target category label is set, and each annotated image in step S1 is saved as an XML file. The size and position of the annotation box are represented by the annotation box.

8. The visual rich document information extraction device for the field of drug instruction manuals as described in claim 6 or 7, characterized in that, In the vocabulary acquisition unit, when labeling targets in multiple categories, the priority labeling rules are as follows: brackets are labeled with the first priority, bold is labeled with the second priority, italics are labeled with the third priority, and underline is labeled with the fourth priority.

9. The visual rich document information extraction device for the field of drug instruction manuals as described in claim 6 or 7, characterized in that, The key information in the drug instruction manual includes information such as drug name, ingredients, adverse reactions, indications, and administration method.

10. The visual rich document information extraction device for the field of drug instructions as described in claim 6 or 7, characterized in that, In the key information acquisition unit of the drug instruction manual, visual features include information on font and bracket style.

Images