Multi-modal chemical reaction information extraction and structuring method based on large model

Abstract

The invention relates to a multi-modal chemical reaction information extraction and structuring method based on a large model, comprising the following steps: selecting a public chemical reaction original data set, and structuring the chemical reaction original data set to obtain a basic sample set; according to the corresponding prompt template, performing data enhancement on the basic sample set to obtain a training set D; constructing a multi-task analysis model M and a model loss function, wherein the M comprises a chemical text sharing coding layer, a specific coding layer for each task, a cross-task cross attention layer and a specific decoding layer for each task; training the M to obtain a final trained M '; and selecting a literature Y, extracting text modal data by using M ', and then extracting corresponding required chemical information in the Y in combination with graph modal data. According to the method, chemical reaction elements such as reactants, products and reaction conditions can be extracted from related literatures in an end-to-end mode, and structural expression and traceable management of chemical reaction information are achieved by fusing text and image semantics.

Description

Technical Field

[0001] This invention relates to the fields of software engineering and automated testing technology, and in particular to a method for extracting and structuring multimodal chemical reaction information based on a large model. Background Technology

[0002] Chemical reactions are the core process of chemical scientific research and a fundamental support for fields such as drug development, materials design, and fine chemicals. A large number of new drug discoveries and materials innovations rely on the systematic accumulation, in-depth analysis, and intelligent reuse of existing reaction knowledge. Currently, a vast amount of organic synthesis patents and scientific literature have accumulated globally. These materials contain extremely rich information on reaction pathways, key experimental conditions, and precise molecular structures, forming a potentially enormous treasure trove of chemical knowledge. However, this high-value information is often mixed in with text and images in a highly unstructured form, such as complex natural language descriptions, inconsistently formatted chemical reaction equations, and molecular structure diagrams embedded in documents. This unstructured data format makes it difficult for computer systems to directly parse, store, retrieve, and perform large-scale computational analysis of chemical reaction information, severely hindering the digitization and intelligentization of chemical knowledge.

[0003] In addressing this challenge, two main technological approaches have gradually emerged: The first is traditional manual compilation and expert systems, as seen in authoritative commercial databases such as Reaxys and SciFinder. This approach relies heavily on domain experts and professionals who manually read documents to extract key information such as reactants, products, yields, and reaction conditions, then input this information into a highly structured database model. This method ensures data accuracy and high quality, serving as the cornerstone of chemical research over the past few decades. However, this approach is inefficient and costly, making it difficult to keep pace with the exponential growth of literature. The second approach is automated chemical text information extraction technology. To overcome the bottleneck of manual processing, researchers have turned to automated solutions based on Natural Language Processing (NLP) and rule systems. For example, early chemical text mining tools typically relied on dictionary matching, manually written rule templates, or dependency parsing to identify chemical entities, properties, and relationships within text. While these methods achieve a degree of automated extraction, their performance heavily depends on the completeness and domain adaptability of pre-defined rules.

[0004] In recent years, with the rapid development of artificial intelligence technology, deep learning-based methods for extracting chemical reaction information have received widespread attention. Breakthroughs in natural language processing, in particular, have enabled pre-trained models based on the Transformer architecture—such as BERT and GPT—to automatically learn deep chemical semantic representations from unstructured text, thereby achieving various complex tasks such as end-to-end reaction role recognition, condition extraction, and product prediction. These methods fundamentally eliminate the reliance on cumbersome manual rules and feature engineering, enabling direct learning on large-scale corpora and automatically capturing implicit semantic patterns and complex logical relationships within the text. Compared to traditional methods, deep learning models demonstrate stronger adaptability, generalization ability, and potential, bringing hope for efficiently processing chemical texts from different sources and with different expressions.

[0005] Overall, these chemical information extraction methods based on deep semantic modeling mark a key shift in the field from "symbol recognition" to "semantic understanding," significantly improving the intelligence and automation of chemical knowledge extraction.

[0006] However, despite the significant progress made in deep learning, existing text-based automation methods still face several key limitations that urgently need to be addressed in their journey towards practical application:

[0007] (1) Lack of unified chemical semantic modeling capabilities. Existing methods generally treat tasks such as chemical reaction role identification, experimental operation verb extraction, and reaction type judgment independently, so that the model can only capture local and fragmented information and cannot grasp the overall semantic structure of chemical reactions. The reaction descriptions in actual literature often involve multi-dimensional information such as reactants, conditions, operation actions, and reaction categories. Only by co-modeling these elements under a unified framework can the experimental process be accurately identified, the reaction logic be understood, and the semantic consistency of the extraction results be guaranteed.

[0008] (2) Semantic Disconnection Between Text and Images. Scientific literature and patent texts commonly exhibit complex referential, ellipsis, and context-dependent phenomena, such as the frequent use of symbols or pronouns like "Compound I" and "Intermediate 2," while their corresponding actual chemical structures are often only presented in images or independent reaction formulas. Existing pure text models are completely incapable of handling this, resulting in the inability to achieve semantic completion and cross-modal mapping. The extracted logical relationships of reaction participants are incomplete, severely restricting the accuracy and completeness of structured chemical reaction data.

[0009] (3) Lack of traceability and verifiability mechanisms. Existing research methods mostly focus on improving the accuracy and recall of the information extraction task itself, while generally neglecting to trace the source and reliability of the generated content. As a "black box", the model cannot clearly indicate which text segment or image in the original document a certain extracted reactant or condition corresponds to, which greatly limits the practical application of such automated systems in scenarios with high reliability requirements. Summary of the Invention

[0010] In view of the above-mentioned problems in the existing technology, the technical problem to be solved by the present invention is: how to overcome the defects of semantic modeling fragmentation, image and text information separation and lack of traceability mechanism, and realize end-to-end structured extraction of multimodal chemical reaction information.

[0011] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0012] A method for extracting and structuring multimodal chemical reaction information based on a large model includes the following steps:

[0013] S01: Select the publicly available original dataset of chemical reactions, where each original data point of a chemical reaction is a text description of the chemical reaction and is labeled with the chemical reaction type;

[0014] The original chemical reaction dataset is structured to obtain a basic sample set. The structured processing involves using a large model prompt template Script = {SystemPrompt, UserPrompt, SourceText, RawLabels} to label each sample in the original chemical reaction dataset with task labels. The task labels include chemical reaction role recognition task labels, experimental operation verb extraction task labels, and reaction type task labels. The chemical reaction role recognition task labels and experimental operation verb extraction task labels are both labeled using the BIO sequence labeling method.

[0015] S02: Based on T = {SystemPrompt,UserPrompt,SeedInput,StrategyInput}, LLM is used to augment the basic sample set with task labels to obtain the training set D;

[0016] S03: Construct a multi-task parsing model M, which includes a chemical text shared coding layer based on Transformer, a task-specific coding layer based on feedforward network, a cross-task cross-attention layer based on attention mechanism, and a task-specific decoding layer based on conditional random field.

[0017] S04: Select the i-th training sample from D, and perform word segmentation on i to obtain the token sequence [w1, w2, ..., wn ], w n This represents the nth token, and then [w1, w2, ..., w n As input to the Transformer-based chemical text shared coding layer, the output is the hidden embedding matrix of i. and Where L represents the total number of shared coding layers for chemical text based on Transformer, and d represents the dimension of the hidden embedding matrix;

[0018] S05: Will The input is each task-specific encoding layer based on a feedforward network, and the output is a task-specific representation H for i. i,t H i,t The calculation expression is as follows:

[0019]

[0020] Where t represents a specific task, W i,t b represents the weight of the i-th training sample under t. i,t This represents the bias term of the i-th training sample under t; the specific tasks include chemical reaction role recognition task, experimental operation verb extraction task, and reaction type task;

[0021] S06: H i,t Input a cross-task attention layer based on an attention mechanism, and output H. i,t Enhanced representation The calculation formula is as follows:

[0022]

[0023] Where j represents another arbitrary task of i, α tj V represents the attention weight of t on j. j Represents the value vector of j;

[0024] When t is a chemical reaction role recognition task

[0025] When t is the experimental operation verb extraction task.

[0026] When t is a chemical reaction role recognition task

[0027] S07: Will The input is a task-specific decoding layer based on a conditional random field, and the output is the predicted value of the task label for each token in i. The formula for calculating the predicted value of each task label is as follows:

[0028] The formula for calculating the predicted value of the chemical reaction role recognition task label is:

[0029]

[0030] Among them, y n W represents the predicted label corresponding to the nth token in the input text sequence. ER b represents the weight matrix of the output layer of the chemical reaction role recognition task. ER This represents the bias vector for the task. This indicates that the CRF layer starts from the previous label y. i-1 Move to the current label y i The transition probability matrix parameters, where Z represents the normalization factor;

[0031] The formula for calculating the predicted value of the labels in the experimental verb extraction task is as follows:

[0032]

[0033] Among them, W EA b represents the weight matrix of the output layer of the experimental verb extraction task. EA This represents the bias vector for the task.

[0034] The formula for calculating the predicted value of the reaction type task label is:

[0035]

[0036] Among them, W RT b represents the weight matrix of the output layer for reaction-type tasks. RT This represents the bias vector for the task.

[0037] Traverse all samples in D to obtain the predicted probability of the chemical reaction role recognition task label, the predicted probability of the experimental operation verb extraction task label, and the predicted probability of the reaction type task label for each sample.

[0038] S08: Construct the joint loss function of M The calculation formula is as follows:

[0039]

[0040] in, This indicates a loss of chemical reaction role recognition. This indicates the loss from extracting experimental operation verbs. The loss represents the reaction type, and λ1, λ2, and λ3 are the loss weighting coefficients.

[0041] The Adam optimizer is used to train M, and the parameters of M are updated through gradient descent backpropagation. Training is not stopped when changes occur or the maximum number of iterations is reached, thus obtaining a well-trained M'.

[0042] S09: Select a PDF document Y containing the chemical information Z to be extracted, and use MinerU to parse Y page by page to obtain a JSON file containing location information. The JSON file containing location information is recorded in a uniform format, and then the JSON file is separated into text modal data and image modal data according to the type contained in the format.

[0043] S10: Use M' to parse the text modal data to generate preliminary structured chemical reaction information, use a large visual model to perform image recognition on the image modal data, and construct a compound reference mapping dictionary; replace the references in the preliminary structured chemical reaction information with the corresponding compounds through the compound reference mapping dictionary to obtain complete structured data;

[0044] S11: Associate the complete structured data with the same data in Y, and finally output the chemical information about Z in the associated result as the extraction result.

[0045] Preferably, the content of the training set D obtained in S02 is as follows:

[0046] S021: A predefined type instruction template, in which output constraints are embedded, wherein the output constraints are to output the corresponding chemical text description and corresponding task tags in a structured form.

[0047] S022: Randomly draw seed data x from the base sample set with task labels. seed , will x seed Inputting t into the LLM yields a new base sample x. gen x gen With BIO sequence task tags and reaction type task tags, x gen The expression is as follows:

[0048] x gen =Prompt(x seed ,t)

[0049] In this context, Prompt(·) represents a type instruction template.

[0050] S023: Repeat S022, and obtain a new basic sample set X after several iterations. gen X gen Together with the basic sample set, they constitute the training set D.

[0051] Preferably, the hidden embedding matrix of i is obtained in S04. The steps are as follows:

[0052] S041: Take [w1, w2,..., w n as the input of the Transformer-based chemical text sharing encoding layer, and map it to the word vector E i = [e1, e2,..., e n through the embedding layer of the Transformer, where e n = Embed(w n ), representing the embedding vector corresponding to w n , and Embed(·) represents the embedding operation; then embed the position information in E i to obtain E i,p = [e1 + p1, e2 + p2,..., e n + p n , where p n represents the position vector corresponding to w n ;

[0053] S042: After passing E i,p through the L-layer encoding layer of the Transformer, obtain The position information of each token is generated using sine and cosine functions as follows:

[0054]

[0055] where k represents the dimension index (0 ≤ k < d / 2) and d represents the dimension of the hidden embedding matrix.

[0056] Preferably, the steps to obtain the enhanced representation i,t of H in S06 are as follows:

[0057] S061: Use the attention mechanism to calculate the query representation Q t , key representation K t and value representation V t of t:

[0058] Q t = W Q H i,t , K t = W K H i,t , V t = W V H i,t

[0059] where W Q , W<​​​​​S062: Calculate the attention score A between task t and task j. tj The calculation expression is as follows:

[0061]

[0062] Where, d t The dimension representing task t. The transpose of the key matrix of task j;

[0063] S63: Subsequently, regarding A tj Perform a softmax operation to obtain the normalized attention weights α. tj The calculation expression is as follows:

[0064]

[0065] Where exp(·) denotes an exponential function with the natural constant e as its base;

[0066] S64: Using α tj The value of task j represents V. j We perform a weighted summation to obtain the enhanced representation of task t.

[0067] Preferably, the joint loss function of M is constructed in S08. The steps are as follows:

[0068] S081: Chemical reaction role recognition loss, based on CRF sequence labeling loss, calculated using the following formula:

[0069]

[0070] in, Enhanced representation of chemical reaction role recognition, Indicates the true label, A scoring function representing the true label;

[0071] S082: Experimental verb extraction loss, based on cross-entropy sequence labeling loss, calculated as follows:

[0072]

[0073] Where m represents the length of the text sequence, c l Indicates a real BIO tag, This indicates the probability that the token belongs to a real tag;

[0074] S083: The response type loss uses sentence-level multi-class cross-entropy loss, calculated as follows:

[0075]

[0076] Where K represents the number of reaction types, y k One-hot encoding representing the true reaction type. This indicates the probability that the reaction belongs to the k-th class.

[0077] S084: Total Loss Function The calculation formula is as follows:

[0078]

[0079] Where λ1, λ2, λ3∈[0,1] represent the balance coefficients.

[0080] Compared with the prior art, the present invention has at least the following advantages:

[0081] 1. By constructing a Transformer-based shared encoder and a cross-task attention mechanism, deep semantic interaction was achieved for tasks such as chemical reaction role recognition, experimental operation verb extraction, and reaction type determination. The shared encoding layer extracts common chemical text features, and the cross-task attention layer calculates attention weights between different tasks, fully capturing the implicit logical relationships between reaction roles, operational actions, and reaction types. This solves the semantic fragmentation and information fragmentation problems caused by existing technologies that model each subtask independently.

[0082] 2. Through multimodal collaborative parsing and the construction of a compound reference mapping dictionary, deep integration of text semantics and image information was achieved. The molecular structure parsing module was used to perform semantic recognition and localization of molecular structure diagrams in the document, and combined with OCR technology to construct a reference dictionary. This effectively overcomes the shortcomings of existing pure text models in parsing abstract references such as "compound I" and "intermediate 2," eliminates cross-modal reference ambiguity, and solves the common problem of semantic break between text and images in scientific literature.

[0083] 3. By establishing a reversible, end-to-end traceability mechanism through MinerU, the extracted structured data is precisely mapped to page numbers, paragraph levels, and bounding box coordinates in the original document. This mechanism completely solves the industry pain point that the extraction process of existing deep learning models is "black box" and the results cannot be verified.

[0084] This invention not only significantly improves the accuracy and automation level of chemical reaction information extraction, but also significantly enhances the verifiability and credibility of data through a full-chain traceability mechanism, making the model more practical and applicable to real-world scenarios such as the construction of large-scale chemical reaction databases and knowledge mining for new drug development. It provides a strong data foundation and technical support for the digital transformation and intelligent research in the field of chemistry. Attached Figure Description

[0085] Figure 1 This is a schematic diagram of the model training of the present invention.

[0086] Figure 2 This is a schematic diagram of multi-task cross-attention in the method of the present invention.

[0087] Figure 3 This is a schematic diagram of a framework for end-to-end extraction of structured information.

[0088] Figure 4 This is the custom script prompt template used in the structured processing of this invention.

[0089] Figure 5 This is the prompt template T used in the data expansion process of this invention.

[0090] Figure 6 These are examples of text modal data from some documents obtained during the actual use of this invention.

[0091] Figure 7 These are examples of image modal data from some documents obtained during the actual use of this invention. Detailed Implementation

[0092] The present invention will now be described in further detail.

[0093] See Figures 1-7 A method for extracting and structuring multimodal chemical reaction information based on a large model includes the following steps:

[0094] S01: Select the publicly available original dataset of chemical reactions, where each original data point of a chemical reaction is a text description of the chemical reaction and is labeled with the chemical reaction type;

[0095] A basic sample set is obtained by structuring the original chemical reaction dataset. This structuring involves using a large model prompt template Script = {SystemPrompt, UserPrompt, SourceText, RawLabels} to label each sample in the original chemical reaction dataset with task tags. These task tags include chemical reaction role recognition task tags, experimental operation verb extraction task tags, and reaction type task tags. The chemical reaction role recognition task tags and experimental operation verb extraction task tags are both labeled using the BIO sequence annotation method, a known technology. The chemical reaction role recognition task tags include reactants, products, solvent (CONDITION_CATALYST), catalyst (CONDITION_SOLVEMENT), reaction temperature (TEMPERATURE), reaction time (TIME), and yield (YIELD).

[0096] S02: Based on T = {SystemPrompt, UserPrompt, SeedInput, StrategyInput}, use LLM to perform data augmentation on the basic sample set with task labels to obtain the training set D; the prompt template T = **You are an expert in the field of organic chemistry; please perform data augmentation on the (original chemical reaction text) below according to the specified (augmentation strategy); Input data: Original chemical reaction text: {{x_seed}} Augmentation strategy: {{instruction_type}} Task requirements: Rewrite the text according to the augmentation strategy (such as synonym replacement, sentence transformation, role substitution, or vertical expansion) to ensure that the chemical reaction mechanism is correct and the semantics are coherent; Perform entity recognition on the rewritten text to generate sequence labels that conform to the BIO labeling system (including REACTANTS, PRODUCTS, CONDITION_CATALYST, TEMPERATURE, TIME, YIELD, etc.); It must be output in strict JSON format and should not contain other explanatory text; Output format example: {"text":"Augmented chemical reaction description text...","bio_labels":"O B-REACTANTS"} I-REACTANTS..."}**; Based on the existing labeled basic data sample set, the large model LLM is used to generate extended samples with the same style and structure as the original data. The distribution characteristics of chemical entities, operational behaviors and reaction roles in the generated text are controlled by prompting engineering or conditional generation, thereby expanding the scale of training data and improving the generalization ability and robustness of the subsequent model in the description of different types of chemical reactions.

[0097] The contents of the training set D obtained in S02 are as follows:

[0098] S021: Predefined type instruction template, which includes types such as synonym replacement, sentence transformation, role substitution, and top-bottom expansion, and the type instruction template embeds output constraints, which are to output the corresponding chemical text description and corresponding task tags in a structured form.

[0099] S022: Randomly draw seed data x from the base sample set with task labels. seed , will x seed Inputting t into the LLM yields a new base sample x. gen x gen With BIO sequence task tags and reaction type task tags, x gen The expression is as follows:

[0100] x gen =Prompt(x seed ,t)

[0101] In this context, Prompt(·) represents a type instruction template.

[0102] S023: Repeat S022, and obtain a new basic sample set X after several iterations. gen X gen Together with the basic sample set, they constitute the training set D.

[0103] S03: Construct a multi-task parsing model M, wherein M includes a chemical text shared encoding layer based on Transformer, task-specific encoding layers based on feedforward networks, cross-task cross-attention layers based on attention mechanisms, and task-specific decoding layers based on conditional random fields; Transformer, feedforward networks, attention mechanisms, and conditional random fields are all existing technologies;

[0104] S04: Select the i-th training sample from D, and perform word segmentation on i to obtain the token sequence [w1, w2, ..., w n ], w n This represents the nth token, and then [w1, w2, ..., w n As input to the Transformer-based chemical text shared coding layer, the output is the hidden embedding matrix of i. and Where L represents the total number of shared coding layers for chemical text based on Transformer, and d represents the dimension of the hidden embedding matrix;

[0105] The hidden embedding matrix of i obtained in S04 The steps are as follows:

[0106] S041: Take [w1, w2,..., w n as the input of the Transformer-based chemical text sharing encoding layer, and map it to the word vector E i = [e1, e2,..., e n through the embedding layer of the Transformer, where e n = Embed(w n ), representing the embedding vector corresponding to w n , and Embed(·) represents the embedding operation; then embed the position information in E i to get E i,p = [e1 + p1, e2 + p2,..., e n + p n , where p n represents the position vector corresponding to w n ;

[0107] S042: After passing E i,p through the L-layer encoding layer of the Transformer, get where the position information of each token is generated using sine and cosine functions, as follows:

[0108]

[0109] where k represents the dimension index (0 ≤ k < d / 2) and d represents the dimension of the hidden embedding matrix.

[0110] S05: Input into the task-specific encoding layer based on the feed-forward network, and output the task-specific representation H i,t about i. The calculation expression of H i,t is as follows:

[0111]

[0112] where t represents the specific task, W i,t represents the weight of the i-th training sample under t, and b i,t represents the bias term of the i-th training sample under t; the specific tasks include chemical reaction role recognition task, experimental operation verb extraction task, and reaction type task;

[0113] S06: Input H i,t into the cross-task cross-attention layer based on the attention mechanism, and output the enhanced representation i,t of H The calculation formula is as follows:

[0114]

[0115] Where j represents another arbitrary task of i, α tj V represents the attention weight of t on j. j Represents the value vector of j;

[0116] When t is a chemical reaction role recognition task

[0117] When t is the experimental operation verb extraction task.

[0118] When t is a chemical reaction role recognition task

[0119] The purpose of using a cross-task attention layer here is to enhance the semantic interaction between various tasks, and to fuse information from different tasks through a cross-task attention mechanism.

[0120] H is obtained in S06 i,t Enhanced representation The steps are as follows:

[0121] S061: Compute the query representation Q of t using an attention mechanism. t , key represents K t Sum value represents V t :

[0122] Q t =W Q H i,t ,K t =W K H i,t V t =WvH i,t

[0123] Among them, W Q W K W V Each represents the corresponding weight;

[0124] S062: Calculate the attention score A between task t and task j. tj The calculation expression is as follows:

[0125]

[0126] Where, d t The dimension representing task t. The transpose of the key matrix of task j;

[0127] S63: Subsequently, regarding A tj Perform a softmax operation to obtain the normalized attention weights α. tj The calculation expression is as follows:

[0128]

[0129] Where exp(·) denotes an exponential function with the natural constant e as its base;

[0130] S64: Using α tj The value of task j represents V. j We perform a weighted summation to obtain the enhanced representation of task t.

[0131] S07: Will The input is a task-specific decoding layer based on a conditional random field, and the output is the predicted value of the task label for each token in i. The formula for calculating the predicted value of each task label is as follows:

[0132] The formula for calculating the predicted value of the chemical reaction role recognition task label is:

[0133]

[0134] Among them, y n W represents the predicted label corresponding to the nth token in the input text sequence. ER b represents the weight matrix of the output layer of the chemical reaction role recognition task. ER This represents the bias vector for the task. This indicates that the CRF layer starts from the previous label y. i-1 Move to the current label y i The transition probability matrix parameters, where Z represents the normalization factor;

[0135] The formula for calculating the predicted value of the labels in the experimental verb extraction task is as follows:

[0136]

[0137] Among them, W EA b represents the weight matrix of the output layer of the experimental verb extraction task. EA This represents the bias vector for the task.

[0138] The formula for calculating the predicted value of the reaction type task label is:

[0139]

[0140] Among them, W RT b represents the weight matrix of the output layer for reaction-type tasks. RT This represents the bias vector for the task.

[0141] Traverse all samples in D to obtain the predicted probability of the chemical reaction role recognition task label, the predicted probability of the experimental operation verb extraction task label, and the predicted probability of the reaction type task label for each sample.

[0142] S08: Construct the joint loss function of M The calculation formula is as follows:

[0143]

[0144] in, This indicates a loss of chemical reaction role recognition. This indicates the loss from extracting experimental operation verbs. The loss represents the reaction type, and λ1, λ2, and λ3 are the loss weighting coefficients.

[0145] The Adam optimizer is used to train M, and the parameters of M are updated through gradient descent backpropagation. Training is not stopped when changes occur or the maximum number of iterations is reached, thus obtaining a well-trained M'.

[0146] The joint loss function of M is constructed in S08. The steps are as follows:

[0147] S081: Chemical reaction role recognition loss, based on CRF sequence labeling loss, calculated using the following formula:

[0148]

[0149] in, Enhanced representation of chemical reaction role recognition, Indicates the true label, The scoring function represents the true label; the CRF loss is a current technique; this loss is used to supervise the model in identifying chemical entity boundaries and their roles in reactions;

[0150] S082: Experimental verb extraction loss, based on cross-entropy sequence labeling loss, calculated as follows:

[0151]

[0152] Where m represents the length of the text sequence, c l Indicates a real BIO tag, This represents the probability that the token belongs to the true label; the cross-entropy loss is a current technique used to supervise the model in identifying key operational verb boundaries in chemical reaction text.

[0153] S083: The response type loss uses sentence-level multi-class cross-entropy loss, calculated as follows:

[0154]

[0155] Where K represents the number of reaction types, y k One-hot encoding representing the true reaction type. This represents the probability that the reaction belongs to the k-th class; this loss is used to supervise the model in determining the type of chemical reaction.

[0156] S084: Total Loss Function The calculation formula is as follows:

[0157]

[0158] Where λ1, λ2, λ3 ∈ [0,1] represent the balance coefficients. The balance coefficients are used to adjust the relative contributions of multiple tasks during training; normalization ensures... and The magnitudes are comparable to prevent single-task dominating the training process; λ can be dynamically adjusted during training.

[0159] The model employs a multi-task joint optimization mechanism, which achieves synchronous learning of the reaction text structure and semantics through the collaborative constraints of chemical entity recognition loss and reaction role classification loss.

[0160] S09: Select a PDF document Y containing the chemical information Z to be extracted, and use MinerU to parse Y page by page to obtain a JSON file containing location information. The JSON file containing location information is recorded in a uniform format, and then the JSON file is separated into text modal data and image modal data according to the type contained in the format.

[0161] MinerU is an existing technology. The literature contains data in both text and image formats. The JSON file will mark the type of the corresponding content, such as {"type":"text","text":"The response of flow durationcurves to afforestation","text_level":1,"bbox":[62,480,946,904],"page_idx":0};

[0162] S10: Use M' to parse the text modal data to generate preliminary structured chemical reaction information, use a large visual model to perform image recognition on the image modal data, and construct a compound reference mapping dictionary; replace the references in the preliminary structured chemical reaction information with the corresponding compounds through the compound reference mapping dictionary to obtain complete structured data; the large visual model is Qwen-VL, which is an existing technology;

[0163] For text data, the PDF document structure is initially screened based on the hierarchical identifier (text_level) and title keywords in the MinerU parsing results, removing non-experimental content such as "Abstract," "Conclusion," and "References," retaining only the main paragraphs containing descriptions of chemical experiments and reactions. Subsequently, the filtered JSON fragments are input into a large model, which combines text hierarchy, logical order, and semantic context to perform structured parsing of the document, extracting the natural language describing chemical reactions into independent single reaction units and recording their corresponding page numbers, paragraph levels, and bounding box coordinates. For multiple text objects that are vertically consecutive and semantically adjacent on the same page, the system aggregates them based on their bounding box coordinates and syntactic relationships, calculating a unified spatial range using the circumscribed rectangle method to achieve the locational traceability of cross-line chemical reaction text. For image data, the visual large model Qwen-VL is used to perform semantic recognition and classification on the output image objects, filtering out valid images containing chemical molecular structure diagrams or chemical reaction process diagrams.

[0164] The trained text parsing model is used to process the text modal data to generate preliminary structured chemical reaction information. At the same time, a visual model is used to identify the molecular structure and its corresponding bbox position in the molecular structure diagram. After cropping the image, the text information is identified by the OCR model to construct a compound reference dictionary, thereby eliminating the reference ambiguity in the text and image data.

[0165] Each reaction unit is analyzed one by one using a pre-trained text model to generate structured data. For image data, the MolCoref molecular coreference recognition model is further called to detect and locate the bounding boxes of chemical identifiers and molecular structures in the image, and the image is cropped based on the detection results. The cropped molecular structure image is converted into the corresponding SMILES sequence by the MolScribe molecular structure analysis module, and then its standardized compound name is obtained by querying through the PubChem data interface. Subsequently, a reference mapping dictionary is established based on the identifiers in the image and the identified compound information.

[0166] Finally, the references in the structured data are replaced with a reference dictionary to obtain the complete structured data.

[0167] S11: Associate the complete structured data with the same data in Y, and finally output the chemical information about Z in the associated result as the extraction result.

[0168] Example:

[0169] Taking patent document CN106146355A as an example, MinerU parsing can obtain content_list text modal data containing location information, for example... Figure 6 As shown, and modal data from 10 images, for example... Figure 7 As shown;

[0170] For the content_list file, the large model is combined with the hierarchy, that is, for Figure 6 Information processing is performed, and the semantic context is broken down into independent unit reaction texts, i.e., independent embodiment paragraphs. Each embodiment paragraph is input into the text parsing model, which outputs BIO tags for the reaction role recognition task. The BIO tags are then processed into structured data. For example, after processing, patent document CN106146355A will yield {"Reactants":"Syringic acid","Products":"Acetylsyringic acid","Solvents":"Dichloromethane","Catalysts":"Triethylamine","Time":"12h","Yield":"95%"};

[0171] For image modal data (such as...) Figure 7 As shown), the image modal data is input into the MolCoref parsing model. After generating Coref pairs, the identifiers are identified using Paddle OCR, and the molecular structures are identified as 'smiles' using MolScribe and then queried using PubChem. Finally, the following dictionary of references is generated:

[0172] {

[0173] "(3)":"Acetyl syringic acid",

[0174] "(1)":"1,3-DI-Boc-2-methylisothiourea",

[0175] "(2)":"N,N'-Bis(Boc)-N”-(4-hydroxybutyl)guanidine",

[0176] "(4)":"4-[bis[(2-methylpropan-2-yl)oxycarbonylamino]...",

[0177] }

[0178] Finally, based on the reference dictionary, potential references in the parsed text description are eliminated, and the final structured data is obtained based on the positional information attached to the content_list. [{"Reactants":"Syringic acid","Products":"Acetylsyringic acid","Solvents":"Dichloromethane","Catalysts":"Triethylamine","Time":"12h","Yield":"95%","bbox":[105,723,894,827],"page_idx":7},{"Reactants":"Acetylsyringic acid"] Acid; 4-guanidinobutanol; DIC","Products":"Compound(4)*4-[bis[(2-methylpropan-2-yl)oxycarbonylamino]...*","Solvents":"Anhydrous dichloromethane","Catalysts":"DPTs","Temperature":"-5℃;room temperature","Time":"1h;12h","Yield":"63.5%","bbox":[168,528,826,619],"page_idx":8}...).

[0179] Experimental content and results

[0180] This section primarily evaluates the performance of the text parsing model. The invention uses the publicly available dataset ChEMU, which contains 1500 manually labeled organic reaction texts (the dataset used for evaluation was not used to train our model). The labels include dozens of entity names such as reaction products, starting materials, reagents, catalysts, and temperatures. For the tasks of extracting experimental verbs and determining reaction types, GPT is used to generate preliminary labels, which are then manually reviewed.

[0181] For the tasks of chemical reaction role recognition and experimental operation verb extraction, the BIO annotation system is used to evaluate the label prediction for each token. Assume there are N entities in the test set (consisting of consecutive B-labels and I-labels), and the number of entities correctly identified by the model is N. correct The model predicts a total of N entities. pred An entity is considered correctly identified if and only if all of its token labels (including boundary B-labels and internal I-labels) and role types are correctly predicted. Statistical analysis based on prediction results:

[0182] TP: The number of entities N correctly identified by the model correct ;

[0183] FP: The number of entities N incorrectly predicted by the model. pred -N correct ;

[0184] FN: The number of real entities not recognized by the model (NN) correct ;

[0185] Evaluation indicators can then be calculated based on this:

[0186]

[0187] For the reaction type identification task, a multi-classification task evaluation metric is used to calculate the model's classification accuracy across different reaction types:

[0188]

[0189] As shown in Table 1 below:

[0190] Table 1. Text parsing model training parameter settings

[0191] Parameter name Detailed settings illustrate pre-trained encoding model Bert-base Used to encode text and generate feature vectors Feature representation dimension d 768 The vector dimension mapped to each token batch size 32 Number of samples input to the model each time for training Number of training rounds 50 Maximum number of training iterations Learning rate 1e-5 Optimizer initial learning rate loss function weight λ1 0.5 Controlling chemical reaction role identification task weights loss function weight λ2 0.3 Control experimental operation verb extraction task weights <![CDATA[Loss function weight λ3]]> 0.2 Controlling reaction type to determine task weight

[0192] To verify the effectiveness of this invention, it was compared and evaluated with several representative baseline models under the same dataset and training configuration: the general pre-trained model BERT, the chemical reaction extraction method ChemRxnExtractor, and the best model from the ChEMU competition, Melaxtech. The experimental results for the chemical reaction role recognition task are as follows:

[0193] Table 2. Experimental Results of the Chemical Reaction Role Recognition Task

[0194] Methods / Indicators Precision Recall F1 BERT, a general pre-trained model 0.955 0.948 0.949 ChemRxnExtractor 0.943 0.952 0.961 Melaxtech 0.957 0.951 0.954 ours 0.972 0.954 0.977

[0195] For the experimental verb extraction and response type judgment tasks, the general pre-trained model BERT was mainly used for comparison, and the results are as follows:

[0196] Table 3. Experimental Results of Verb Extraction and Response Type Classification Tasks

[0197]

[0198] The experimental results demonstrate that the proposed multi-task joint learning method exhibits significant performance advantages across all three tasks in chemical reaction text parsing. In the chemical reaction element identification task, the proposed method achieves an F1 score of 0.977, representing improvements of 2.3%, 2.8%, and 1.6% compared to Melaxtech, BERT, and ChemRxnExtractor, respectively. In the experimental verb extraction task, the proposed method achieves an F1 score of 0.938, a 2.7% improvement over BERT. In the reaction type judgment task, the proposed method achieves an accuracy of 0.927, an 8.4 percentage point improvement over BERT. These experimental results fully validate the effectiveness of the multi-task joint learning framework: by learning a general chemical text representation through a shared encoding layer, promoting knowledge interaction between tasks through a cross-task attention layer, and focusing on specific task features through task-specific layers, the model achieves mutual promotion and collaborative optimization among tasks. Compared to baseline models that learn on a single task, our method achieves performance improvements across multiple tasks and achieves a better balance between precision and recall, validating the significant advantages and practical application value of the multi-task joint learning framework in chemical reaction text parsing tasks.

[0199] Regarding the end-to-end extraction of structured data from PDF documents, this invention introduces a unified PDF parsing and multimodal recognition system within its overall framework. By integrating a text parsing module and a molecular structure recognition model, it achieves collaborative processing of text, images, and chemical structural formulas in PDF files, automatically extracting chemical reaction paragraphs, chemical entities, reaction roles, and corresponding structural diagram information. Simultaneously, this invention constructs a traceable structured data link, retaining text page numbers and position offsets in the extraction results, enabling reversible tracing from reaction fragments to the original document, thereby ensuring the verifiability and data credibility of the structured results.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for extracting and structuring multimodal chemical reaction information based on a large model, characterized in that: Includes the following steps: S01: Select the publicly available original dataset of chemical reactions, where each original data point of a chemical reaction is a text description of the chemical reaction and is labeled with the chemical reaction type; The original chemical reaction dataset is structured to obtain a basic sample set. The structured processing involves using a large model prompt template Script = {SystemPrompt, UserPrompt, SourceText, RawLabels} to label each sample in the original chemical reaction dataset with task labels. The task labels include chemical reaction role recognition task labels, experimental operation verb extraction task labels, and reaction type task labels. The chemical reaction role recognition task labels and experimental operation verb extraction task labels are both labeled using the BIO sequence labeling method. S02: Based on T = {SystemPrompt,UserPrompt,SeedInput,StrategyInput}, LLM is used to augment the basic sample set with task labels to obtain the training set D; S03: Construct a multi-task parsing model M, which includes a chemical text shared coding layer based on Transformer, a task-specific coding layer based on feedforward network, a cross-task cross-attention layer based on attention mechanism, and a task-specific decoding layer based on conditional random field. S04: Select the i-th training sample from D, and perform word segmentation on i to obtain the token sequence [w1, w2, ..., w n ], w n This represents the nth token, and then [w1, w2, ..., w n As input to the Transformer-based chemical text shared coding layer, the output is the hidden embedding matrix of i. and Where L represents the total number of shared coding layers for chemical text based on Transformer, and d represents the dimension of the hidden embedding matrix; S05: Will The input is each task-specific encoding layer based on a feedforward network, and the output is a task-specific representation H for i. i,t H i,t The calculation expression is as follows: Where t represents a specific task, W i,t b represents the weight of the i-th training sample under t. i,t This represents the bias term of the i-th training sample under t; the specific tasks include chemical reaction role recognition task, experimental operation verb extraction task, and reaction type task; S06: H i,t Input a cross-task attention layer based on an attention mechanism, and output H. i,t Enhanced representation The calculation formula is as follows: Where j represents another arbitrary task of i, α tj V represents the attention weight of t on j. j Represents the value vector of j; When t is a chemical reaction role recognition task When t is the experimental operation verb extraction task. When t is a chemical reaction role recognition task S07: Will The input is a task-specific decoding layer based on a conditional random field, and the output is the predicted value of the task label for each token in i. The formula for calculating the predicted value of each task label is as follows: The formula for calculating the predicted value of the chemical reaction role recognition task label is: Among them, y n W represents the predicted label corresponding to the nth token in the input text sequence. ER b represents the weight matrix of the output layer of the chemical reaction role recognition task. ER This represents the bias vector for the task. This indicates that the CRF layer starts from the previous label y. i-1 Move to the current label y i The transition probability matrix parameters, where Z represents the normalization factor; The formula for calculating the predicted value of the labels in the experimental verb extraction task is as follows: Among them, W EA b represents the weight matrix of the output layer of the experimental verb extraction task. EA This represents the bias vector for the task. The formula for calculating the predicted value of the reaction type task label is: Among them, W RT b represents the weight matrix of the output layer for reaction-type tasks. RT This represents the bias vector for the task. Traverse all samples in D to obtain the predicted probability of the chemical reaction role recognition task label, the predicted probability of the experimental operation verb extraction task label, and the predicted probability of the reaction type task label for each sample. S08: Construct the joint loss function of M The calculation formula is as follows: in, This indicates a loss of chemical reaction role recognition. This indicates the loss from extracting experimental operation verbs. The loss represents the reaction type, and λ1, λ2, and λ3 are the loss weighting coefficients. The Adam optimizer is used to train M, and the parameters of M are updated through gradient descent backpropagation. Training is not stopped when changes occur or the maximum number of iterations is reached, thus obtaining a well-trained M'. S09: Select a PDF document Y containing the chemical information Z to be extracted, and use MinerU to parse Y page by page to obtain a JSON file containing location information. The JSON file containing location information is recorded in a uniform format, and then the JSON file is separated into text modal data and image modal data according to the type contained in the format. S10: Use M' to parse the text modal data to generate preliminary structured chemical reaction information, use a large visual model to perform image recognition on the image modal data, and construct a compound reference mapping dictionary; replace the references in the preliminary structured chemical reaction information with the corresponding compounds through the compound reference mapping dictionary to obtain complete structured data; S11: Associate the complete structured data with the same data in Y, and finally output the chemical information about Z in the associated result as the extraction result.

2. The method for extracting and structuring multimodal chemical reaction information based on a large model as described in claim 1, characterized in that: The contents of the training set D obtained in S02 are as follows: S021: A predefined type instruction template, in which output constraints are embedded, wherein the output constraints are to output the corresponding chemical text description and corresponding task tags in a structured form. S022: Randomly draw seed data x from the base sample set with task labels. seed , will x seed Inputting t into the LLM yields a new base sample x. gen x gen With BIO sequence task tags and reaction type task tags, x gen The expression is as follows: x gen =Prompt(x seed ,t) Among them, Prompt(·) represents a type instruction template; S023: Repeat S022, and obtain a new basic sample set X after several iterations. gen X gen Together with the basic sample set, they constitute the training set D.

3. The method for extracting and structuring multimodal chemical reaction information based on a large model as described in claim 2, characterized in that: The hidden embedding matrix of i obtained in S04 The steps are as follows: S041: [w1,w2,...,w n As input to the Transformer-based chemical text shared encoding layer, it is mapped to word vector E through the Transformer embedding layer. i =[e1,e2,...,e n ], where e n =Embed(w n ), indicating w n The corresponding embedding vector, Embed(·) represents the embedding operation; then in E i E is obtained by embedding location information. i,p =[e1+p1,e2+p2,...,e n +p n ], where p n Indicates w n The corresponding position vector; S042: E i,p After passing through the L-layer encoding layer of the Transformer, we get The position information for each token is generated using sine and cosine functions, as detailed below: Here, k represents the dimension index (0 ≤ k < d / 2), and d represents the dimension of the hidden embedding matrix.

4. The method for extracting and structuring multimodal chemical reaction information based on a large model as described in claim 3, characterized in that: H is obtained in S06 i,t Enhanced representation The steps are as follows: S061: Compute the query representation Q of t using an attention mechanism. t , key represents K t Sum value represents V t : Q t =W Q H i,t ,K t =W K H i,t ,V t =W V H i,t Among them, W Q W K W V Each represents the corresponding weight; S062: Calculate the attention score A between task t and task j. tj The calculation expression is as follows: Where, d t The dimension representing task t. The transpose of the key matrix of task j; S63: Subsequently, regarding A tj Perform a softmax operation to obtain the normalized attention weights α. tj The calculation expression is as follows: Here, exp(·) represents the exponential function with the natural constant e as the base; S64: Using α tj The value of task j represents V. j We perform a weighted summation to obtain the enhanced representation of task t.

5. The method for extracting and structuring multimodal chemical reaction information based on a large model as described in claim 4, characterized in that: The joint loss function of M is constructed in S08. The steps are as follows: S081: The loss of chemical reaction role recognition, a sequence labeling loss based on CRF, is calculated as follows: in, Enhanced representation of chemical reaction role recognition, Indicates the true label, A scoring function representing the true label; S082: The loss of experimental operation verb extraction, a sequence labeling loss based on cross-entropy, is calculated as follows: Where m represents the length of the text sequence, c l Indicates a real BIO tag, This indicates the probability that the token belongs to a real tag; S083: The reaction type loss adopts a sentence-level multi-class cross-entropy loss, and the calculation formula is as follows: Where K represents the number of reaction types, y k One-hot encoding representing the true reaction type. This indicates the probability that the reaction belongs to the k-th class. S084: Total Loss Function The calculation formula is as follows: Here, λ1, λ2, λ3 ∈ [0, 1] represent the balance coefficients.

Images