A method and system for information extraction of a chemical process diagram

Abstract

The application provides a chemical process diagram information extraction method and system, and relates to the technical field of image processing. The method comprises the following steps: text extraction is performed on the process diagram to obtain each text and a corresponding bounding box; chemical object detection is performed on the process diagram to obtain attribute information of each chemical object in the process diagram; based on the positional relationship between the bounding box of the target text and each chemical object, the chemical object to which the target text belongs is determined, and the two are bound; based on the direction vector of the material flow line in the process diagram and the bounding box of the chemical object, the predecessor and successor of each chemical object are determined; based on the topological relationship and text binding relationship of all obtained chemical objects, a structured prompt word is constructed; through a visual language model, process information extraction is performed on the process diagram under the constraint of the structured topological relationship of the structured prompt word, and a process diagram file corresponding to the process diagram is obtained. The application aims to improve the information extraction accuracy of the chemical process diagram.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a method and system for extracting information from chemical process diagrams. Background Technology

[0002] The chemical industry is a vital pillar industry, and process flow diagrams (PID / PFD) embody core design wisdom, production knowledge, and operational experience. With the development of intelligent industry, the chemical industry urgently needs to shift from experience-driven to data- and model-driven decision-making, and the digitization and structuring of process flow data is the foundation for achieving this transformation.

[0003] Currently, enterprise-archived process flow diagrams are stored in a variety of formats (raster formats such as JPEG / PNG, vector formats such as DWG / DXF), all of which are unstructured data and cannot be directly understood by computers. Conventional extraction methods primarily involve the following: Traditional manual data extraction, where engineers visually identify equipment, pipelines, and other elements in the drawings, manually record relevant parameters, and input them into a database to form structured data. Simple drawing recognition software can only convert drawing formats (e.g., scanned documents to PDF) or perform partial text OCR recognition; it cannot recognize the semantic relationships and topological connections between process elements, nor can it generate standardized data models. Both of these extraction methods require significant engineering personnel involvement, ultimately resulting in low extraction efficiency and high labor costs.

[0004] Although the application of model-driven decision-making in the chemical industry requires high-quality structured data input from chemical process diagrams, there is currently no efficient and accurate way to extract structured data from unstructured chemical process diagram drawings. Summary of the Invention

[0005] In view of this, this application provides a method for extracting information from chemical process diagrams. It aims to solve or partially solve the problems existing in the background art.

[0006] The first aspect of this application provides a method for extracting information from a chemical process flow diagram, the method comprising: Text extraction is performed on the flowchart to obtain each text segment and its bounding box. Chemical object detection is performed on the flowchart to obtain the attribute information of each chemical object in the flowchart. The attribute information includes category and bounding box. Among them, the attribute information of chemical objects whose category is logistics line also includes the direction vector of the logistics line in the flowchart. Based on the positional relationship between the bounding box of the target text and each chemical object, the chemical object to which the target text belongs is determined, and the target text is bound to the chemical object to which it belongs. The target text is any extracted text segment. Based on the direction vector of the logistics line in the flowchart and the bounding box of the chemical object, the predecessor and successor chemical objects of each chemical object are determined. Based on the category, bounding box, bounding text, predecessor chemical objects, and successor chemical objects of all chemical objects, a structured prompt word is constructed. The structured prompt word records the category, bounding text, center point coordinates of the chemical object, set of predecessor chemical objects, and set of successor chemical objects for each chemical object. Using a pre-trained visual language model, and constrained by the structured topological relationships of the structured prompts, process information is extracted from the flowchart to obtain the flowchart file corresponding to the flowchart.

[0007] A second aspect of this application provides an information extraction system for chemical process diagrams, the system comprising: The text extraction module is used to extract text from the flowchart to obtain each piece of text in the flowchart and the bounding box of each piece of text. The chemical object detection module is used to detect chemical objects in the flowchart and obtain the attribute information of each chemical object in the flowchart. The attribute information includes category and bounding box. Among them, the attribute information of the chemical object as a logistics line also includes the direction vector of the logistics line in the flowchart. The text-to-chemicals-object binding module is used to determine the chemical object to which the target text belongs based on the positional relationship between the bounding box of the target text and each chemical object, and to bind the target text to the chemical object to which it belongs. The target text is any extracted text segment. The precursor and successor chemical object determination module is used to determine the precursor and successor chemical objects of each chemical object based on the direction vector of the logistics line in the flow diagram and the bounding box of the chemical object. The structured prompt word construction module is used to construct structured prompt words based on the category, bounding box, bounding text, predecessor chemical objects and successor chemical objects of all chemical objects. The structured prompt words record the category, bounding text, center point coordinates of the chemical object, the set of predecessor chemical objects and the set of successor chemical objects for each chemical object. The flowchart extraction module is used to extract process information from the flowchart using a pre-trained visual language model, under the constraint of the structured topological relationship of the structured prompt words, to obtain the flowchart file corresponding to the flowchart.

[0008] The method for extracting information from a chemical process flow diagram provided in this application has the following advantages: The information extraction method for the chemical process flow diagram provided in this application employs a pre-parsing scheme that includes text extraction (such as using OCR technology to extract text and text bounding boxes in the chemical process flow diagram), object detection (such as using an object detection model to detect and determine the category and bounding box information of chemical objects in the chemical process flow diagram), and constructing structured prompt words based on the text extraction and object detection results (such as constructing a structured Prompt based on the text extraction and object detection results, which records the category of each chemical object, the bound text, the center point coordinates of the chemical object, the set of predecessor chemical objects, and the set of successor chemical objects). The structured prompt words are first constructed based on the original chemical process flow diagram.

[0009] Subsequently, the method of this application does not directly input the original chemical process flow diagram into the pre-trained visual language model (i.e., VLM Visual Language Model, such as Qwen3-vl model). Instead, after constructing the corresponding structured prompt words based on the original chemical process flow diagram, the original chemical process flow diagram drawing and the constructed structured prompt words are input into the pre-trained visual language model to extract process information, so as to output the structured flow diagram file corresponding to the chemical process flow diagram.

[0010] The method provided in this application allows the pre-trained visual language model to no longer rely solely on "visual intuition" to infer the process flow. Instead, based on the input explicit structured information (i.e., constructed structured cue words), and constrained by the spatial layout and geometric constraints of chemical objects in the chemical process diagram, it performs reasoning and information extraction on the input chemical process diagram. This significantly reduces errors in understanding the flow direction and topological relationships in the chemical process diagram, thereby improving the accuracy of the conversion from unstructured image data to structured data. Furthermore, the entire process is automatically performed by the relevant model, greatly improving the efficiency of the conversion process. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating an information extraction method from a chemical process flow diagram, as shown in one embodiment of this application. Figure 2 This is a schematic diagram illustrating an information extraction system for a chemical process flow diagram, as shown in one embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] Before explaining the information extraction method of the chemical process diagram provided in this application, the overall idea and background of this application will be explained first.

[0015] Currently, enterprise-archived process flow diagrams are stored in a variety of formats (paper scans, raster formats such as JPEG / PNG, and vector formats such as DWG / DXF), all of which are unstructured data and cannot be directly understood by computers, resulting in low data utilization. Conventional extraction methods mainly employ the following approaches: Traditional manual data extraction primarily involves engineers visually identifying equipment, pipelines, and other elements in drawings, manually recording relevant parameters, and entering them into a database to form structured data. This traditional conversion of unstructured drawings into structured data relies on manual interpretation and data entry, resulting in low efficiency, high costs, and difficulty in guaranteeing data integrity and accuracy. In summary, this method has the following drawbacks: extremely low efficiency, requiring significant manpower for massive amounts of drawings and resulting in long conversion cycles; high costs, with labor costs and time costs adding up, placing a heavy burden on enterprises; poor data reliability, as manual interpretation is prone to omissions and misjudgments, leading to insufficient data integrity and accuracy; and it cannot adapt to the needs of large-scale, high-frequency drawing conversion.

[0016] Simple drawing recognition software can only convert drawing formats (e.g., scanned documents to PDF) or perform partial text OCR recognition. It cannot recognize the semantic relationships and topological connections between process elements, nor can it generate standardized data models. Both of these extraction methods require significant engineering personnel involvement, resulting in low extraction efficiency and high labor costs. In summary, this approach has the following drawbacks: limited recognition range, only capable of processing text or simple graphics, unable to identify core process elements such as equipment and pipelines; lack of semantic understanding capability, unable to establish topological relationships between elements (e.g., connections between equipment and pipelines, material flow direction); and no standardized output, unable to generate machine-readable structured data models such as JSON / XML, thus failing to provide effective structured chemical process flow diagram input for downstream large-scale models.

[0017] This paper explains the proposed method for extracting information from a chemical process flow diagram. First, this application aims to use an end-to-end Vision-Language Model (VLM) to extract information from the image data of an unstructured chemical process flow diagram, thereby obtaining a structured result. However, in practical applications, it has been found that the end-to-end VLM model struggles to accurately understand the material flow direction and topology in chemical process flow diagrams. Even when using advanced Vision-Language Models such as GPT4-o, the model still exhibits the following problems when directly inputting image data from the chemical process flow diagram: it cannot reliably distinguish between the inflow and outflow directions of materials; it easily confuses the upstream and downstream relationships between multiple pipelines; and it makes incorrect inferences in questions requiring a clear process sequence (such as "what is the next piece of equipment?"). In chemical process flow diagrams, this directly leads to problems such as incorrect judgment of upstream and downstream equipment relationships and reversed process flow order.

[0018] To address this issue, this application still employs an end-to-end VLM model to extract information from the image data of unstructured chemical process diagrams. However, building upon this, this application further introduces a pre-analysis method, which comprises: extracting text from the chemical process diagram requiring information extraction using OCR technology (e.g., extracting the specific content of each text segment and the specific coordinates of the bounding box of each text segment in the image), where the object of information extraction in this application is the chemical process diagram, which is image data; simultaneously, performing object detection on the chemical process diagram to detect various chemical objects (e.g., chemical equipment, logistics lines) within the chemical process diagram, and outputting the corresponding detection results (e.g., outputting the category and bounding box of each chemical object, and, when the chemical object is a logistics line, also outputting the direction vector of the logistics line in the chemical process diagram); and constructing corresponding structured prompts based on the text extraction results and object detection results. Finally, when using an end-to-end VLM model to extract information from the image data of unstructured chemical process diagrams, the structured prompt word is input simultaneously. That is, the chemical process diagram and the corresponding structured prompt word are input into the end-to-end VLM model at the same time. Under the structured topological constraints of the structured prompt word, the end-to-end VLM model extracts process information from the chemical process diagram to obtain more accurate extraction results.

[0019] This application preferably uses Deepseek OCR to extract text information from the chemical process flow diagram. It should be understood that this is only one optional method for text extraction; other techniques capable of extracting text and bounding box coordinates from images can also be used to extract text information from the chemical process flow diagram. This application preferably uses the DAMO-YOLO object detection model to detect chemical objects in the chemical process flow diagram. It should be understood that this is only one optional method for object detection; other object detection techniques can also be used to detect chemical objects in the chemical process flow diagram. This application preferably uses the Qwen3-vl visual language model to extract information from the chemical process flow diagram. It should be understood that this is only one optional method for information extraction; other visual language models can also be used to extract information from the chemical process flow diagram to generate a corresponding structured chemical process flow diagram. The visual language model in this application not only relies on visual intuition to infer the structure of the chemical process flow diagram, but also on displayed structured information (i.e., structured cue words) for reasoning. This significantly reduces errors in understanding process direction and topological relationships, thereby improving the overall accuracy of generating structured chemical process flow diagrams (i.e., flowchart files ultimately extracted and generated by the visual language model, such as GraphML format files). The core of this application is to convert an unstructured chemical process flow diagram image into a structured flowchart file, such as a GraphML format flowchart file, so that downstream models can directly understand and apply it.

[0020] refer to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of a chemical process flow diagram information extraction method. Figure 1 As shown, the method includes: Step S01: Extract text from the flowchart to obtain each text segment and its bounding box.

[0021] In this embodiment, a chemical process diagram (image) requiring structured transformation is first obtained. Then, text extraction is performed on the diagram using OCR technology to obtain each text segment and its bounding box. The bounding box of each text segment is represented by its pixel coordinates. For example, the bounding box of text segment 'a' is the area enclosed by connecting the pixel coordinates (u1, v1), (u2, v2), (u3, v3), and (u4, v4) in sequence. The purpose of extracting the bounding box of each text segment is to subsequently bind the text to the corresponding chemical object, i.e., to specify which chemical object a text segment describes.

[0022] This application preferably uses Deepseek OCR to extract text information from chemical process diagrams. It should be understood that this is only one optional method for extracting text information. This application can also use other techniques that can extract text and bounding box coordinate information from images to extract text information from chemical process diagrams.

[0023] Step S02: Perform chemical object detection on the flowchart to obtain the attribute information of each chemical object in the flowchart. The attribute information includes category and bounding box. Specifically, the attribute information of chemical objects categorized as logistics lines further includes the direction vector of the logistics line in the flowchart.

[0024] In this embodiment, for the chemical process flow diagram that needs to be structured, a target detection model is used to detect chemical objects in the flow diagram, identifying each chemical object and outputting attribute information for each object. This attribute information includes the specific category and bounding box of the chemical object. When the chemical object is a material flow line, the attribute information includes not only the specific category and bounding box but also the direction vector of the material flow line. This direction vector consists of the start and end coordinates in the pixel coordinate system, indicating the connection relationship between chemical equipment in the flow diagram. The purpose of determining the bounding box of each chemical object in this application is to subsequently bind it to text, i.e., to indicate which chemical object a text describes. The bounding box of each chemical object is represented by its coordinate position in the pixel coordinate system.

[0025] In this application, the categories of chemical objects include two main categories: chemical equipment and logistics lines. Chemical equipment is just a general term. The category of chemical objects output by the target detection model is the specific type of chemical equipment, such as reaction vessel, distillation tower, filter, and specific instrument type (such as pressure gauge).

[0026] This application preferably uses the DAMO-YOLO object detection model to perform object detection on chemical objects in the chemical process flow diagram. It should be understood that this is only an optional object detection method, and other object detection technologies can also be used to perform object detection on chemical objects in the chemical process flow diagram.

[0027] Step S03: Based on the positional relationship between the bounding box of the target text and each chemical object, determine the chemical object to which the target text belongs, and bind the target text to the chemical object to which it belongs. The target text is any extracted text segment.

[0028] In this embodiment, after obtaining the extraction results of each text segment and corresponding bounding box of the same chemical process flow diagram through steps S01 to S02, and obtaining the attribute information of each chemical object therein, each text segment is bound to the chemical object it actually describes. This application mainly determines which chemical object each text segment describes based on the positional relationship between the bounding box of each text segment and each chemical object in the pixel coordinate system, and then binds each text segment to the chemical object it describes.

[0029] Since each piece of text is bound to the actual chemical object it describes in the same way, this application uses a piece of text as an example to illustrate an optional binding method, referred to as the target text. This optional binding method is as follows: Based on the positional relationship between the bounding box of the target text and each chemical object in the pixel coordinate system, determine which chemical object the bounding box of the target text is closest to. The chemical object whose bounding box is closest to the target text is then identified as the chemical object actually described by the target text. At this point, the target text is determined to belong to that chemical object, and the target text is bound to that chemical object. For example, based on the positional relationship between the bounding box of target text 'a' and each chemical object in the pixel coordinate system, determine that the chemical object whose bounding box of target text 'a' is closest to is chemical object 1. At this point, target text 'a' is determined to belong to chemical object 1, and target text 'a' is bound to chemical object 1. After binding a piece of text to a chemical object, it indicates that the text is used to describe that chemical object in the chemical process flow diagram.

[0030] Step S04: Based on the direction vector of the logistics line in the flowchart and the bounding box of the chemical object, determine the predecessor chemical object and the successor chemical object of each chemical object.

[0031] In this embodiment, the direction of the material flow lines in the chemical process flow diagram defines the topological connection relationship between chemical equipment in the flow diagram. For example, in a material flow line connecting two chemical equipments, the material flow line indicates which chemical equipment the material flows from to which chemical equipment. Therefore, this application will determine the predecessor and successor chemical objects of each chemical object based on the direction vector of the material flow lines in the chemical process flow diagram obtained in step S02 and the pointing relationship between the vector vector and the bounding box of the corresponding chemical object. Since multiple chemical equipments in the chemical process flow diagram are connected to the same chemical equipment through corresponding material flow lines, a single chemical equipment may have multiple predecessor chemical objects and multiple successor chemical objects.

[0032] A predecessor chemical object of any chemical object refers to a chemical object that points to that chemical object, and a successor chemical object of any chemical object refers to the chemical object that the chemical object points to. For example, chemical object 1 is connected to chemical object 2 through a logistics line, and the starting point of the logistics line is connected to chemical object 1, and the ending point of the logistics line is connected to chemical object 2. Therefore, chemical object 1 is a predecessor chemical object of chemical object 2, and chemical object 2 is a successor chemical object of chemical object 1.

[0033] Among them, only chemical objects belonging to the category of chemical equipment have precursor chemical objects and successor chemical objects. At the same time, precursor chemical objects and successor chemical objects are also chemical equipment. Chemical objects in the category of logistics lines will not be considered as precursor chemical objects and successor chemical objects.

[0034] Step S05: Based on the category, bounding box, bounding text, predecessor chemical object and successor chemical object of all chemical objects, construct a structured prompt word. The structured prompt word records the category, bounding text, center point coordinates of the chemical object, set of predecessor chemical objects and set of successor chemical objects for each chemical object.

[0035] In this embodiment, the present application predefines a prompt word template. The content to be filled in this prompt word template includes the specific category of each chemical object, the bound text segment, the center point coordinates (the center point coordinates of the chemical equipment are taken as the center of its bounding box), the set of predecessor chemical objects, and the set of successor chemical objects. This information for each chemical object in the chemical process flow diagram is obtained through the processing described in steps S01 to S04 above. Then, this information for each chemical object in the obtained chemical process flow diagram is filled into the prompt word template to obtain the corresponding structured prompt word. This structured prompt word will be associated with the chemical process flow diagram. Figure 1 Generate structured flowchart files from the input visual language model.

[0036] Specifically, for any chemical object, if the processing steps S01 to S04 do not yield the relevant content required to be filled in the prompt word template for that chemical equipment, then the missing relevant content will be left blank in the prompt word template. For example, for a chemical object belonging to a logistics line, it will not have preceding or succeeding chemical objects; therefore, for a chemical object belonging to a logistics line, both its preceding and succeeding chemical object sets required to be filled in the prompt word template will be left blank. For example, for a chemical object belonging to chemical equipment, if the target detection model does not identify its specific category, then its category required to be filled in the prompt word template will be left blank. For example, for any chemical object, if the processing steps S01 to S04 do not yield the text segment bound to it, then the text segment bound to it required to be filled in the prompt word template will be left blank.

[0037] Step S06: Using a pre-trained visual language model, under the constraint of the structured topological relationship of the structured prompt words, extract process information from the flowchart to obtain the flowchart file corresponding to the flowchart.

[0038] In this embodiment, after obtaining the structured prompt words corresponding to the chemical process flow diagram through steps S01 to S05, the structured prompt words and the chemical process flow diagram are input into a pre-trained visual language model. Under the constraint of the structured topological relationship of the structured prompt words, the visual language model extracts process information from the chemical process flow diagram, thereby generating a structured process flow diagram file corresponding to the chemical process flow diagram.

[0039] The information extraction method for the chemical process flow diagram provided in this application employs a pre-parsing scheme that includes text extraction (such as using OCR technology to extract text and text bounding boxes in the chemical process flow diagram), object detection (such as using an object detection model to detect and determine the category and bounding box information of chemical objects in the chemical process flow diagram), and constructing structured prompt words based on the text extraction and object detection results (such as constructing a structured Prompt based on the text extraction and object detection results, which records the category of each chemical object, the bound text, the center point coordinates of the chemical object, the set of predecessor chemical objects, and the set of successor chemical objects). The structured prompt words are first constructed based on the original chemical process flow diagram.

[0040] Subsequently, the method of this application does not directly input the original chemical process flow diagram into the pre-trained visual language model (i.e., VLM Visual Language Model, such as Qwen3-vl model). Instead, after constructing the corresponding structured prompt words based on the original chemical process flow diagram, the original chemical process flow diagram drawing and the constructed structured prompt words are input into the pre-trained visual language model to extract process information, so as to output the structured flow diagram file corresponding to the chemical process flow diagram.

[0041] The method provided in this application allows the pre-trained visual language model to no longer rely solely on "visual intuition" to infer the process flow. Instead, based on the input explicit structured information (i.e., constructed structured cue words), and constrained by the spatial layout and geometric constraints of chemical objects in the chemical process diagram, it performs reasoning and information extraction on the input chemical process diagram. This significantly reduces errors in understanding the flow direction and topological relationships in the chemical process diagram, thereby improving the accuracy of the conversion from unstructured image data to structured data. Furthermore, the entire process is automatically performed by the relevant model, greatly improving the efficiency of the conversion process.

[0042] In conjunction with the above embodiments, in one implementation, this application also provides a method for extracting information from a chemical process flow diagram. In this method, the chemical objects include chemical equipment and material flow lines; step S03 may include steps S031 to S034: Step S031: Determine whether there is a first chemical equipment among all chemical equipment whose positional relationship between its bounding box and the bounding box of the target text satisfies the set conditions.

[0043] In this embodiment, this application provides a method for determining the chemical object to which a text segment belongs, so that the text segment can be accurately bound to the actual chemical object it describes. This allows the text segment to be input into a pre-trained visual language model in a spatially semantic form to constrain the extraction of process information, avoiding problems such as incorrect equipment number binding, confusion between logistics numbers and equipment, and incomplete process semantics, ultimately improving the accuracy of process information extraction. By determining the chemical object to which the text segment belongs and binding the two, the text information in the chemical process flow diagram has a clear equipment or logistics affiliation before entering the pre-trained visual language model, avoiding semantic drift.

[0044] In this embodiment, based on the specific position of the bounding box of each chemical equipment in the pixel coordinate system obtained in step S02, it is determined which chemical equipment's bounding box and the bounding box of the target text (the target text is any extracted text segment) satisfy a predefined set condition. The chemical equipment whose bounding box and the bounding box of the target text satisfy the set condition is identified as the first chemical equipment. One optional implementation of the set condition is whether the distance between the two bounding boxes being compared is lower than a set threshold; if it is lower, the set condition is determined to be satisfied. Alternatively, it is whether the overlap area of ​​the two bounding boxes being compared is greater than a set threshold; if it is greater, the set condition is determined to be satisfied.

[0045] Step S032: If a first chemical equipment that meets the set conditions exists, determine that the target text belongs to the first chemical equipment.

[0046] In this embodiment, if a first chemical device is detected among the chemical equipment in the chemical process flow diagram, the target text is determined to belong to the first chemical device, and the target text is bound to the first chemical device during binding.

[0047] Step S033: In the absence of a first chemical equipment that meets the set conditions, based on the positional relationship between the bounding box of the target text and each logistics line, determine the target logistics line corresponding to the target text, and determine that the target text belongs to the corresponding target logistics line. The target logistics line is the logistics line whose own logistics line body and the bounding box of the target text are less than a first threshold.

[0048] In this embodiment, if no such first chemical equipment is found among the chemical equipment detected from the chemical process flow diagram, it is further determined whether the target text belongs to a certain logistics line. The specific determination method is as follows: When determining whether a target text belongs to a specific logistics line, since logistics lines are mostly not straight lines, the bounding box of the target text is directly compared with the body of each logistics line. The body of a logistics line consists of multiple coordinate points in a pixel coordinate system. When comparing the positional relationship between the bounding box of the target text and the body of a logistics line, the bounding box of the target text is compared with the multiple coordinate points that make up the body of that logistics line. By comparing the positional relationship between the bounding box of the target text and the body of each logistics line, the logistics line whose body is less than a first threshold is identified as the target logistics line. The target text is then identified as belonging to that target logistics line, and during the binding process, the target text is bound to that target logistics line.

[0049] Step S034: If there is no target logistics line corresponding to the target text, determine that the target text belongs to the second chemical equipment, and the second chemical equipment is a chemical equipment whose bounding box overlaps with the bounding box of the target text.

[0050] In this embodiment, if neither a first chemical equipment meeting the set conditions nor a target logistics line corresponding to the target text exists, then the bounding box of which chemical equipment overlaps with the bounding box of the target text is determined. The chemical equipment whose bounding box overlaps with the bounding box of the target text is identified as the second chemical equipment, and the target text is determined to belong to the second chemical equipment. Since the second chemical equipment is not a better binding choice for the target text (as is the first chemical equipment meeting the set conditions, and the target logistics line whose distance is below the first threshold), this application records the binding relationship as a medium confidence relationship when the target text is bound to the second chemical equipment to which it belongs. The recorded medium confidence binding relationship is also recorded in the finally generated structured prompt words, allowing the pre-trained visual language model to provide confirmation or correction results for the medium confidence binding relationship at the end.

[0051] If there is neither a first chemical equipment that meets the set conditions, nor a target logistics line corresponding to the target text, nor a second chemical equipment that overlaps with it, then the system determines which chemical equipment's bounding box is closest to the target text's bounding box, and which logistics line's body is closest to the target text's bounding box, selecting the closest chemical equipment and the closest logistics line. If the distance to the closest chemical equipment is less than the distance to the closest logistics line, then the target text belongs to that closest chemical equipment; if the distance to the closest chemical equipment is greater than the distance to the closest logistics line, then the target text belongs to that closest logistics line. When the target text is bound to its closest chemical equipment (or closest logistics line), this application records this binding relationship as a low-confidence relationship. This low-confidence binding relationship is also recorded in the final generated structured prompt, allowing the pre-trained visual language model to provide confirmation or correction results for this low-confidence binding relationship at the end.

[0052] Using the same implementation method, each piece of text extracted from the chemical process flow diagram can be used to determine the chemical object to which it belongs.

[0053] In conjunction with the above embodiments, in one implementation, this application also provides a method for extracting information from a chemical process flow diagram. In this method, step S031 may include steps S031_1 to S031_5: Step S031_1: Based on the positional relationship between the bounding box of each chemical equipment and the bounding box of the target text, determine the candidate chemical equipment whose bounding box overlaps with the bounding box of the target text.

[0054] In this embodiment, in another optional implementation, the predefined conditions of this application mainly determine the first chemical equipment to which the target text belongs based on the area overlap angle. The area overlap angle mainly includes two aspects: one is whether the overlap area is greater than or equal to a second threshold, and the other is whether the geometric center of the bounding box of the target text is located within the bounding box of a certain chemical equipment. In this embodiment, the specific process of determining the first chemical equipment corresponding to the target text based on the set conditions is as follows: Based on the positional relationship between the bounding box of each chemical equipment and the bounding box of the target text in the pixel coordinate system, it is determined which chemical equipment bounding boxes overlap with the bounding box of the target text, and the chemical equipment whose bounding boxes overlap with the bounding box of the target text is identified as candidate chemical equipment.

[0055] Step S031_2: Determine the overlap area between the bounding box of the target text and the bounding box of the candidate chemical equipment.

[0056] In this embodiment, for candidate chemical equipment corresponding to the target text, the overlap area between the bounding boxes of these candidate chemical equipment and the bounding box of the target text is determined.

[0057] Step S031_3: If the overlapping area is greater than or equal to the second threshold, determine the candidate chemical equipment as the first chemical equipment that meets the set conditions.

[0058] In this embodiment, candidate chemical equipment whose overlap area with the target text's bounding box is greater than or equal to a second threshold is determined as the first chemical equipment that meets the set conditions.

[0059] Step S031_4: If the overlapping area is less than the second threshold, determine whether the geometric center of the bounding box of the target text is located inside the bounding box of the candidate chemical equipment.

[0060] In this embodiment, if the overlap area between the bounding boxes of all candidate chemical equipment and the bounding box of the target text is less than the second threshold, then it is further determined which candidate chemical equipment's bounding box the geometric center of the target text's bounding box is located inside.

[0061] Step S031_5: If the geometric center of the bounding box of the target text is located inside the bounding box of the candidate chemical equipment, the candidate chemical equipment is determined to be the first chemical equipment that meets the set conditions.

[0062] In this embodiment, if the geometric center of the bounding box of the target text is located inside the bounding box of a candidate chemical device, the candidate chemical device is determined to be the first chemical device that meets the set conditions. If the geometric center of the bounding box of the target text is not located inside the bounding box of any candidate chemical device, then there is no first chemical device that meets the set conditions.

[0063] In conjunction with the above embodiments, in one implementation, this application also provides a method for extracting information from a chemical process flow diagram. This method further includes: in the case that no first chemical equipment meets the set conditions, and no corresponding target logistics line and corresponding second chemical equipment exist, determining that the target text belongs to the chemical object closest to the target text, and marking the binding relationship corresponding to the target text recorded in the structured prompt as low confidence.

[0064] In this embodiment, if the target text has neither a corresponding first chemical equipment that meets the set conditions, nor a target logistics line corresponding to the target text, nor a second chemical equipment that does not overlap with it, then it is determined which chemical equipment's bounding box is closest to the target text's bounding box, and which logistics line's main body is closest to the target text's bounding box. If the distance to the nearest chemical equipment is less than the distance to the nearest logistics line, then the target text is determined to belong to the nearest chemical equipment; if the distance to the nearest chemical equipment is greater than the distance to the nearest logistics line, then the target text is determined to belong to the nearest logistics line. When the target text is bound to its nearest chemical equipment (or nearest logistics line), this application records this binding relationship as a low-confidence relationship. This low-confidence binding relationship is also recorded in the final generated structured prompt, allowing the pre-trained visual language model to provide confirmation or correction results for this low-confidence binding relationship at the end.

[0065] In conjunction with the above embodiments, in one implementation, this application also provides a method for extracting information from a chemical process flow diagram. This method further includes: Step S001: Calculate the vertical distance between text bounding boxes.

[0066] In this embodiment, to avoid semantic errors caused by excessive extraction of text segments, this application further clusters and merges closely spaced text segments after extracting text from the flowchart to restore the complete semantics of chemical equipment and logistics. Before binding text segments to chemical objects, this application performs clustering and merging processing on the text segments. The specific clustering and merging method is as follows: After extracting text from the chemical process flow diagram to obtain each text segment and its respective bounding box, the vertical distance between the text bounding boxes of each text segment is calculated, that is, the vertical distance between the text bounding boxes of every two text segments is calculated.

[0067] Step S002: Determine the vertical distance threshold based on the average character height of the text in the flowchart.

[0068] In this embodiment, the application also calculates a vertical distance threshold based on the average character height of all text in the chemical process flow diagram. This vertical distance threshold is used as the basis for determining whether the text segments need to be clustered and merged. The vertical distance threshold is preferably a multiple of the average character height; for example, if the average character height is 'a', the vertical distance threshold is set to 2a.

[0069] Step S003: Merge text bounding boxes whose vertical spacing is less than or equal to the vertical distance threshold to obtain new text bounding boxes and corresponding new text.

[0070] In this embodiment, when the vertical spacing between the bounding boxes of two text segments is less than or equal to the vertical distance threshold and there is significant overlap in the horizontal projection, the two text segments are determined to be the same semantic block and are clustered and merged to recover the split device label or logistics line annotation text, thereby obtaining a new text segment, and obtaining the bounding box surrounding the new text segment based on the new text segment.

[0071] In conjunction with the above embodiments, in one implementation, this application also provides a method for extracting information from a chemical process flow diagram. In this method, determining the direction vector of the material flow line includes: Step S02_1: For the detected logistics line, search for the corresponding arrow start point and arrow end point near the boundary of the logistics line.

[0072] In this embodiment, for each detected logistics line, the starting point and ending point of each arrow are searched near the boundary of each logistics line.

[0073] In this embodiment, when using a target detection model (such as DAMO-YOLO) for routine target detection in a chemical process flow diagram: the overall category of the logistics line can be detected, but the starting and ending directions of the arrows in the logistics line remain unclear; the starting and ending points of the arrows in the logistics line are small target detection objects, resulting in low performance under a conventional IoU of 0.5–0.95. In a chemical process flow diagram, this problem can lead to reversed logistics direction and incorrect connection of multiple logistics lines near equipment. Therefore, this application provides a method for independent detection and direction recovery of the starting and ending points of logistics line arrows based on a target detection model (such as DAMO-YOLO). The specific implementation is as follows: DAMO-YOLO was used to perform object detection on the chemical process flow diagram. The detection categories included at least: various chemical equipment and material flow lines, as well as the start and end points of arrows on the material flow lines. Small-sized bounding boxes were used for training on the start and end points of arrows. A relaxed IoU threshold, such as (0.10–0.50), was used during the evaluation and inference phases to improve the recall rate of small objects.

[0074] When constructing the training dataset for object detection models (such as DAMO-YOLO), the starting and ending points of the arrows in the logistics line are labeled as independent small targets, using small bounding boxes that closely match their true geometric shape. This ensures the model can learn accurate directional localization features. For example, in the training dataset, the starting and ending points of the arrows in the logistics line are labeled with small bounding boxes that closely match their true geometric range, instead of framing the entire arrow (shaft + head) with a large box. A small portion of pixels is covered by a 30×30 blurred box and mixed into the same category as the logistics line (Arrow body). Since the starting and ending points of the arrows in the logistics line are extremely small targets, a lower IoU matching threshold (e.g., 0.10–0.50) is used for this category during model evaluation and inference to reduce false negatives caused by pixel-level offsets and improve the recall rate of directional key points.

[0075] Step S02_2: If an arrow start point and an arrow end point are found to exist simultaneously near the boundary of the logistics line, construct the direction vector of the logistics line using the geometric center point of the arrow start point and the geometric center point of the arrow end point.

[0076] In this embodiment, when an arrow start point and an arrow end point are simultaneously found near the boundary of a logistics line, a direction vector for the logistics line is constructed, pointing from the geometric center point of the arrow start point to the geometric center point of the arrow end point, using the geometric center points of the arrow start point and the arrow end point. Similarly, when an arrow start point and an arrow end point are simultaneously found near the boundary of any logistics line, a corresponding direction vector is constructed for that logistics line. This method transforms the logistics line direction from a vague visual feature into a geometrically computable object, improving the stability of chemical logistics direction recognition.

[0077] Step S02_3: If the number of arrow starting points and / or arrow ending points found near the boundary of the logistics line is abnormal, determine that the direction vector of the logistics line is empty.

[0078] In this embodiment, if the number of arrow start points and / or arrow end points found near the boundary of a logistics line is abnormal, the direction vector of that logistics line is determined to be empty. Since the direction vector of a logistics line is used to create the predecessor and successor relationships of chemical equipment and further generate corresponding prompts to constrain the information extraction of the chemical process flow diagram, constructing an incorrect direction vector for a logistics line will lead to incorrect constraints on the information extraction of the chemical process flow diagram, thus exacerbating the extraction of erroneous information. Therefore, if the direction vector of a logistics line cannot be accurately determined, it is not constructed; at most, it reduces the constraints on subsequent information extraction of the chemical process flow diagram, preventing the provision of incorrect constraints. If the number of arrow start points found near the boundary of a logistics line is not unique, the number of its arrow start points is determined to be abnormal; if the number of arrow end points found near the boundary of a logistics line is not unique, the number of its arrow end points is determined to be abnormal.

[0079] In conjunction with the above embodiments, in one implementation, this application also provides a method for extracting information from a chemical process flow diagram. In this method, step S04 may include: Step S04_1: Based on the direction vectors of the logistics lines in the flowchart, determine the bounding box of the chemical object to which the starting point of each direction vector points, and the bounding box of the chemical object to which the ending point of each direction vector points.

[0080] In this embodiment, through steps S01 to S03, the direction vector of each material flow line in the flowchart can be obtained, that is, the starting and ending positions of the arrows of each material flow line in the chemical process flow diagram in the pixel coordinate system. Also, the specific positions of the bounding boxes of each chemical object in the chemical process flow diagram in the pixel coordinate system can be obtained. Based on the direction vectors of each material flow line in the chemical process flow diagram, the bounding box of the chemical object pointed to by the starting point of each direction vector can be obtained, as well as the boundary of the chemical object pointed to by the ending point of each direction vector. Since the material flow lines in the chemical process flow diagram connect chemical equipment, the bounding box of the chemical object pointed to by the starting point of each direction vector is actually the bounding box of the chemical equipment, and the bounding box of the chemical object pointed to by the ending point of each direction vector is also actually the bounding box of the chemical equipment.

[0081] Step S04_2: Determine the chemical object that the endpoint of the same directional vector points to as the successor chemical object of the chemical object that the starting point points to.

[0082] Step S04_3: Determine the chemical object that the starting point of the same directional vector points to as the predecessor chemical object of the chemical object that the ending point points to.

[0083] In this embodiment, the chemical object pointed to by the endpoint of a direction vector is determined as the successor chemical object of the chemical object pointed to by its starting point. The chemical object pointed to by the starting point of a direction vector is determined as the predecessor chemical object of the chemical object pointed to by its endpoint. For example, if the chemical object pointed to by the starting point of a direction vector 1 is a1, and the chemical object pointed to by the endpoint of the direction vector 1 is a2, then the chemical object a1 pointed to by the starting point of the direction vector 1 is determined as the predecessor chemical object a2 pointed to by its endpoint, and the chemical object a2 pointed to by the endpoint of the direction vector 1 is determined as the successor chemical object a1 pointed to by its starting point.

[0084] Because individual chemical equipment in chemical process flow diagrams often exhibit multiple inflow and multiple outflow topologies, especially towers, reactors, and thermal integration units, this application's method of determining successor and predecessor chemical objects based on the direction of a directional vector can accurately obtain these multiple inflow and multiple outflow topologies of chemical equipment in the flow diagram. Using this information to generate corresponding prompts to constrain information extraction from the flow diagram can effectively improve the accuracy of extraction. Since a single chemical equipment may have multiple inflow and multiple outflow topologies, there may be multiple successor chemical objects corresponding to a single chemical equipment, forming a set of successor chemical objects. Simultaneously, there may also be multiple predecessor chemical objects corresponding to a single chemical equipment, forming a set of predecessor chemical objects.

[0085] This application treats each chemical equipment as a node and each logistics line, defined by the starting and ending points of an arrow, as a directed edge. Based on spatial proximity, the chemical equipment pointed to by the starting point of the logistics line's arrow is the logistics start point, and the chemical equipment pointed to by the ending point of the logistics line's arrow is the logistics end point. For each chemical equipment node, all its incoming edges are recorded; these sets constitute the set of predecessor chemical objects for that node. Similarly, for each chemical equipment node, all its outgoing edges are recorded; these sets constitute the set of successor chemical objects for that node. This constructs a complete directed process flow diagram structure that fully expresses the multi-entry and multi-outlet relationships, branching and merging, and return flow structures within the chemical process flow diagram. Based on this complete directed process flow diagram structure, a pre-trained visual language model is generated using corresponding prompt words to constrain its information extraction from the chemical process flow diagram, thereby improving extraction accuracy.

[0086] In conjunction with the above embodiments, in one implementation, this application also provides a method for extracting information from a chemical process flow diagram. This method further includes: determining a set of chemical objects forming a closed loop in the topology based on the predecessor and successor chemical objects of each chemical object; and writing the set of chemical objects forming a closed loop in the topology into the structured prompt words to constrain the pre-trained visual language model to extract process information from the flow diagram.

[0087] In this embodiment, the closed-loop topology in the chemical process flow diagram is particularly important. Therefore, this application determines the set of chemical objects that form a closed loop in the topology based on the predecessor and successor chemical objects of each chemical object. For example, in the chemical process flow diagram, chemical equipment a is a predecessor chemical equipment b, chemical equipment b is a predecessor chemical equipment c, and chemical equipment c is a predecessor chemical equipment a. Thus, chemical equipment a, b, and c form a closed-loop topology in the chemical process flow diagram. Finally, the set of chemical objects forming a closed loop in the topology is written into structured prompts to constrain the pre-trained visual language model's extraction of process information from the chemical process flow diagram, thereby improving extraction accuracy.

[0088] In conjunction with the above embodiments, in one implementation, this application also provides a method for extracting information from a chemical process flow diagram. This method further includes: if the number of arrow start points and / or arrow end points found near the boundary of the flow line is abnormal, marking the flow line as having low confidence in the structured prompts.

[0089] In this embodiment, if the number of arrow start points and / or arrow end points found near the boundary of the flow line in the chemical process diagram is abnormal, the flow line is marked as low confidence in the structured prompt words. This informs the pre-trained visual language model that it cannot identify the direction of the flow line, so that it will focus on the flow line when extracting information from the chemical process diagram.

[0090] Based on the same inventive concept, this application provides an information extraction system for chemical process diagrams, such as... Figure 2 As shown, the information extraction system 200 for this chemical process flow diagram includes: The text extraction module 201 is used to extract text from the flowchart to obtain each piece of text in the flowchart and the bounding box of each piece of text. The chemical object detection module 202 is used to perform chemical object detection on the flowchart to obtain the attribute information of each chemical object in the flowchart. The attribute information includes category and bounding box. Among them, the attribute information of the chemical object as a logistics line also includes: the direction vector of the logistics line in the flowchart. The text-to-chemical object binding module 203 is used to determine the chemical object to which the target text belongs based on the positional relationship between the bounding box of the target text and each chemical object, and to bind the target text to the chemical object to which it belongs. The target text is any extracted text segment. Precursor and successor chemical object determination module 204 is used to determine the precursor and successor chemical objects of each chemical object based on the direction vector of the logistics line in the flow diagram and the bounding box of the chemical object. The structured prompt word construction module 205 is used to construct structured prompt words based on the category, bounding box, bounding text, predecessor chemical objects and successor chemical objects of all chemical objects. The structured prompt words record the category, bounding text, center point coordinates of the chemical object, the set of predecessor chemical objects and the set of successor chemical objects for each chemical object. The flowchart extraction module 206 is used to extract flowchart information from the flowchart under the constraints of the structured topological relationship of the structured prompt words through a pre-trained visual language model, so as to obtain the flowchart file corresponding to the flowchart.

[0091] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of this application.

[0092] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0093] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0098] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0099] The above provides a detailed description of the method and system for extracting information from a chemical process flow diagram provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for extracting information from a chemical process flow diagram, characterized in that, The method includes: Text extraction is performed on the flowchart to obtain each text segment and its bounding box. Chemical object detection is performed on the flowchart to obtain the attribute information of each chemical object in the flowchart. The attribute information includes category and bounding box. Among them, the attribute information of chemical objects whose category is logistics line also includes the direction vector of the logistics line in the flowchart. Based on the positional relationship between the bounding box of the target text and each chemical object, the chemical object to which the target text belongs is determined, and the target text is bound to the chemical object to which it belongs. The target text is any extracted text segment. Based on the direction vector of the logistics line in the flowchart and the bounding box of the chemical object, the predecessor and successor chemical objects of each chemical object are determined. Based on the category, bounding box, bounding text, predecessor chemical objects, and successor chemical objects of all chemical objects, a structured prompt word is constructed. The structured prompt word records the category, bounding text, center point coordinates of the chemical object, set of predecessor chemical objects, and set of successor chemical objects for each chemical object. Using a pre-trained visual language model, and constrained by the structured topological relationships of the structured prompts, process information is extracted from the flowchart to obtain the flowchart file corresponding to the flowchart.

2. The method for extracting information from a chemical process flow diagram according to claim 1, characterized in that, Chemical objects include chemical equipment and logistics lines; The determination of the chemical object to which the target text belongs, based on the positional relationship between the bounding box of the target text and each chemical object, includes: Determine whether there exists a first chemical equipment among all chemical equipment whose positional relationship between its bounding box and the bounding box of the target text satisfies the set conditions; If a first chemical equipment exists that meets the set conditions, it is determined that the target text belongs to the first chemical equipment; In the absence of a first chemical equipment that meets the set conditions, based on the positional relationship between the bounding box of the target text and each logistics line, the target logistics line corresponding to the target text is determined, and the target text belongs to the corresponding target logistics line. The target logistics line is the logistics line whose own logistics line body and the bounding box of the target text are less than a first threshold. If there is no target logistics line corresponding to the target text, it is determined that the target text belongs to the second chemical equipment, which is a chemical equipment whose bounding box overlaps with the bounding box of the target text.

3. The method for extracting information from a chemical process flow diagram according to claim 2, characterized in that, Determine whether there exists a first chemical equipment among all chemical equipment whose positional relationship between its bounding box and the bounding box of the target text satisfies a set condition, including: Based on the positional relationship between the bounding box of each chemical equipment and the bounding box of the target text, candidate chemical equipment whose bounding boxes overlap with the bounding box of the target text are identified. Determine the overlap area between the bounding box of the target text and the bounding box of the candidate chemical equipment; If the overlapping area is greater than or equal to the second threshold, the candidate chemical equipment is determined to be the first chemical equipment that meets the set conditions; If the overlapping area is less than the second threshold, determine whether the geometric center of the bounding box of the target text is located inside the bounding box of the candidate chemical equipment; If the geometric center of the bounding box of the target text is located inside the bounding box of the candidate chemical equipment, the candidate chemical equipment is determined to be the first chemical equipment that meets the set conditions.

4. The method for extracting information from a chemical process flow diagram according to claim 2, characterized in that, The method further includes: In the absence of a first chemical equipment that meets the set conditions, and in the absence of a corresponding target logistics line and a corresponding second chemical equipment, the target text is determined to belong to the chemical object closest to the target text, and the binding relationship corresponding to the target text recorded in the structured prompt is marked as low confidence.

5. The method for extracting information from a chemical process flow diagram according to claim 1, characterized in that, The method further includes: Calculate the vertical distance between text bounding boxes; The vertical distance threshold is determined based on the average character height of the text in the flowchart. Text bounding boxes with a vertical spacing less than or equal to the vertical distance threshold are merged to obtain new text bounding boxes and corresponding new text.

6. The method for extracting information from a chemical process flow diagram according to claim 1, characterized in that, Determining the direction vector of the logistics line includes: For the detected logistics line, search for the corresponding arrow start point and arrow end point near the boundary of the logistics line; If an arrow start point and an arrow end point are found to coexist near the boundary of the logistics line, the direction vector of the logistics line is constructed using the geometric center points of the arrow start point and the arrow end point. If the number of arrow start points and / or arrow end points found near the boundary of the logistics line is abnormal, the direction vector of the logistics line is determined to be empty.

7. The method for extracting information from a chemical process flow diagram according to claim 1, characterized in that, Based on the direction vector of the logistics line in the flowchart and the bounding box of the chemical object, the predecessor and successor chemical objects of each chemical object are determined, including: Based on the direction vectors of the logistics lines in the flowchart, determine the bounding box of the chemical object to which the starting point of each direction vector points, and the bounding box of the chemical object to which the ending point of each direction vector points; The chemical object whose endpoint is pointed to by the same directional vector is determined as the successor chemical object of the chemical object whose starting point is pointed to. The chemical object pointed to by the starting point of the same directional vector is determined as the predecessor chemical object pointed to by its ending point.

8. The method for extracting information from a chemical process flow diagram according to claim 7, characterized in that, The method further includes: Based on the predecessor and successor chemical objects of each chemical object, determine the set of chemical objects that form a closed loop in the topology; The set of chemical objects forming a closed loop in the topology is written into the structured prompt words to constrain the pre-trained visual language model to extract process information from the flowchart.

9. The method for extracting information from a chemical process flow diagram according to claim 2, characterized in that, The method further includes: If the number of arrow start points and / or arrow end points found near the boundary of the logistics line is abnormal, the logistics line is marked as low confidence in the structured prompt.

10. An information extraction system for chemical process diagrams, characterized in that, The system includes: The text extraction module is used to extract text from the flowchart to obtain each piece of text in the flowchart and the bounding box of each piece of text. The chemical object detection module is used to detect chemical objects in the flowchart and obtain the attribute information of each chemical object in the flowchart. The attribute information includes category and bounding box. Among them, the attribute information of the chemical object as a logistics line also includes the direction vector of the logistics line in the flowchart. The text-to-chemicals-object binding module is used to determine the chemical object to which the target text belongs based on the positional relationship between the bounding box of the target text and each chemical object, and to bind the target text to the chemical object to which it belongs. The target text is any extracted text segment. The precursor and successor chemical object determination module is used to determine the precursor and successor chemical objects of each chemical object based on the direction vector of the logistics line in the flow diagram and the bounding box of the chemical object. The structured prompt word construction module is used to construct structured prompt words based on the category, bounding box, bounding text, predecessor chemical objects and successor chemical objects of all chemical objects. The structured prompt words record the category, bounding text, center point coordinates of the chemical object, the set of predecessor chemical objects and the set of successor chemical objects for each chemical object. The flowchart extraction module is used to extract process information from the flowchart using a pre-trained visual language model, under the constraint of the structured topological relationship of the structured prompt words, to obtain the flowchart file corresponding to the flowchart.

Images