Chemical reaction information extraction method and device, electronic equipment and storage medium

Abstract

The application provides a chemical reaction information extraction method and device, electronic equipment and a storage medium. The chemical reaction information extraction method comprises: obtaining a chemical reaction image and a text instruction; inputting the chemical reaction image and the text instruction into a chemical reaction information extraction model to obtain chemical reaction information corresponding to the text instruction; wherein the chemical reaction information extraction model comprises a visual encoder and a reaction information generator; based on the visual encoder, visual tokens of the chemical reaction image are extracted, and a compression operation is performed on the visual tokens to obtain visual features; and based on the reaction information generator and the visual features, the chemical reaction information corresponding to the text instruction is generated. Through the technical scheme provided by the application, after the visual tokens are extracted, the compression operation is performed on the visual tokens, which can significantly reduce the calculation redundancy of the background area, significantly reduce the calculation and storage costs, and retain the semantic information.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for extracting chemical reaction information. Background Technology

[0002] With the rapid development of artificial intelligence technology, visual language models have demonstrated outstanding capabilities in general-domain image understanding and complex reasoning tasks. However, directly applying them to the highly specialized and structured field of chemistry, especially in visual understanding tasks related to chemical reaction mechanisms, still faces significant challenges. The dilemma of visual representation of chemical information: In chemical research, the structural information of molecules and reactions is usually presented in the form of images, which is the most intuitive medium of communication. However, current mainstream chemical AI research relies excessively on text representations, such as those using the Simplified Molecular Input Line Entry System (SMILES) and Self-Referencing Embedded Strings (SELFIES). These representation methods inevitably lose the spatial geometric information of molecules and the visual context of reactions during the encoding process.

[0003] While some work has attempted to apply general visual language models to the field of chemistry, these typically directly utilize existing architectures, resulting in two major bottlenecks: computational inefficiency due to visual token redundancy. Chemical molecular images inherently exhibit sparsity, with information-dense regions (molecular structures) comprising only a small fraction of the entire image, while the majority consists of blank background. Existing models (such as ChemVLM) uniformly segment the entire image into numerous blocks (e.g., 1280 visual tokens) when processing high-resolution chemical images, with the vast majority of tokens corresponding to non-informative background regions. This leads to severe computational redundancy, resulting in slow training and inference speeds, high computational costs, and significantly hindering their application and research on large chemical datasets. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for extracting chemical reaction information, in order to overcome the deficiencies in the prior art.

[0005] This invention provides a method for extracting chemical reaction information, comprising: Acquire images and text commands related to chemical reactions; The chemical reaction image and the text command are input into the chemical reaction information extraction model to obtain the chemical reaction information corresponding to the text command; wherein, the chemical reaction information extraction model includes a visual encoder and a reaction information generator; The step of inputting the chemical reaction image and the text command into the chemical reaction information extraction model to obtain the chemical reaction information corresponding to the text command includes: Based on the visual encoder, visual tokens of the chemical reaction image are extracted, and compression is performed on the visual tokens to obtain visual features; Based on the reaction information generator and the visual features, the chemical reaction information corresponding to the text command is generated.

[0006] According to a chemical reaction information extraction method provided by the present invention, the visual encoder includes an image block embedding layer and a plurality of sequentially connected transformer blocks; The step of extracting visual tokens from the chemical reaction image based on the visual encoder and performing compression on the visual tokens to obtain visual features includes: Based on the image embedding layer, the chemical reaction image is segmented to obtain multiple reaction sub-images, a thumbnail image of the chemical reaction image is generated, and the thumbnail image and the multiple reaction sub-images are linearly projected into a visual token sequence; wherein, the visual token sequence includes multiple visual tokens; Each of the transformer blocks calculates the importance score of each visual token in the input visual token sequence. Based on the importance score, a compression operation is performed on the visual tokens in the visual token sequence until the compression operation of the last transformer block is completed, thereby obtaining the visual feature.

[0007] According to a method for extracting chemical reaction information provided by the present invention, the step of performing a compression operation on visual tokens in the visual token sequence based on the importance score includes: Based on the current transformer block, the score variance is calculated using the importance scores of each visual token in the current input visual token sequence; The fractional variance is compared with a preset threshold to obtain the comparison result; Based on the comparison results, a compression operation is performed on each of the visual tokens; wherein the compression operation includes a token pruning operation and a token merging operation; Based on the visual token after compression, a bias term is calculated such that the next transformer block of the current transformer block calculates the importance score based on the bias term.

[0008] According to a method for extracting chemical reaction information provided by the present invention, the step of performing a compression operation on each of the visual tokens based on the comparison result includes: If the variance is greater than the preset threshold, the visual tokens in the currently input visual token sequence are divided into a first token group and a second token group, and the visual tokens in the first token group and the visual tokens in the second token group are combined to form a token pair. Calculate the similarity between the two visual tokens in the token pair; The token pairs are sorted in descending order based on the similarity to obtain a first sorting result, and the target token pair is determined based on the first sorting result. Based on the similarity, a token merging operation is performed on the two visual tokens in the target token pair; If the variance is less than or equal to the preset threshold, the visual tokens in the current input visual token sequence are sorted in descending order based on the importance score to obtain a second sorting result; Based on the second permutation result, the target visual token is determined, and in the currently input visual token sequence, token pruning operation is performed on other visual tokens besides the target visual token.

[0009] According to a chemical reaction information extraction method provided by the present invention, the reaction information generator includes a text feature extraction module, a projection layer, and a reaction information generation module; The step of generating the chemical reaction information corresponding to the text instruction based on the reaction information generator and the visual features includes: Based on the text feature extraction module, the text features of the text instruction are extracted; Based on the projection layer, the text features are aligned with the visual features; Based on the reaction information generation module, the chemical reaction information is generated by aligning the text features and the visual features.

[0010] According to a chemical reaction information extraction method provided by the present invention, the training process of the chemical reaction information extraction model includes: Obtain a multi-task training set; wherein, the multi-task training set includes training samples corresponding to multiple tasks, and the training samples include sample images and sample instructions; The training samples are input into the initial chemical reaction information extraction model for iterative training until the training is completed, thereby obtaining the chemical reaction information extraction model; wherein, the initial chemical reaction information extraction model includes a visual encoder and a reaction information generator.

[0011] According to the chemical reaction information extraction method provided by the present invention, the multiple tasks include at least two of the following: molecular recognition task, reaction recognition task, reaction type classification task, property prediction task, and reaction prediction task.

[0012] The present invention also provides a chemical reaction information extraction device, comprising: The first acquisition module is configured to acquire chemical reaction images and text commands; The extraction module is configured to input the chemical reaction image and the text command into a chemical reaction information extraction model to obtain chemical reaction information corresponding to the text command; wherein, the chemical reaction information extraction model includes a visual encoder and a reaction information generator; The extraction module includes: The extraction submodule is configured to extract visual tokens from the chemical reaction image based on the visual encoder, and perform compression operations on the visual tokens to obtain visual features; The generation submodule is configured to generate the chemical reaction information corresponding to the text instruction based on the reaction information generator and the visual features.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the chemical reaction information extraction method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the chemical reaction information extraction method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the chemical reaction information extraction method as described above.

[0016] The chemical reaction information extraction method, apparatus, electronic device, and storage medium provided by this invention pre-construct a chemical reaction information extraction model. After extracting a visual token, the visual encoder in the model compresses the token, significantly reducing computational redundancy in the background region while focusing on and preserving key information about the molecular structure. This also significantly reduces computational and storage costs while retaining semantic information. Simultaneously, the reaction information generator understands text instructions and, combined with the visual features output by the visual encoder, directly generates text-level output from pixel-level input. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the chemical reaction information extraction method provided by the present invention.

[0019] Figure 2 This is a schematic diagram of the input and output of the chemical reaction information extraction model provided in the embodiments of the present invention.

[0020] Figure 3 This is a schematic diagram of the chemical reaction information extraction model provided by the present invention.

[0021] Figure 4 This is a schematic diagram of a visual response recognition and prediction task performed on the ChemRxn-V benchmark in an embodiment of the present invention.

[0022] Figure 5 This is a schematic diagram of the chemical reaction information extraction device provided by the present invention.

[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] Figure 1 This is a flowchart illustrating a method for extracting chemical reaction information according to an exemplary embodiment. For example... Figure 1 As shown in an exemplary embodiment, the method for extracting chemical reaction information includes steps 110 to 120, which are described in detail below.

[0026] Step 110: Obtain chemical reaction images and text instructions.

[0027] In embodiments of the present invention, such as Figure 2 As shown, the system retrieves a chemical reaction image and a text command provided by the user. The text command could be something like, "Please describe the mechanism of this reaction."

[0028] Step 120: Input the chemical reaction image and the text command into the chemical reaction information extraction model to obtain the chemical reaction information corresponding to the text command; wherein, the chemical reaction information extraction model includes a visual encoder and a reaction information generator; The step of inputting the chemical reaction image and the text command into the chemical reaction information extraction model to obtain the chemical reaction information corresponding to the text command includes: Based on the visual encoder, visual tokens of the chemical reaction image are extracted, and compression is performed on the visual tokens to obtain visual features; Based on the reaction information generator and the visual features, the chemical reaction information corresponding to the text command is generated.

[0029] In this embodiment of the invention, a chemical reaction information extraction model is pre-trained and deployed, such as... Figure 3 As shown, the chemical reaction information extraction model includes a visual encoder and a reaction information generator.

[0030] First, the visual encoder extracts visual tokens from the chemical reaction image. A visual token is a basic unit that maps the pixel information of the original chemical reaction image into a discrete, processable vector sequence. After obtaining the visual token, a compression operation is performed. By compressing high-resolution pixels into an information-dense, compressed visual feature sequence, computational redundancy in the background region is significantly reduced, while focusing on and preserving key information about the molecular structure. This also significantly reduces computational and storage costs while retaining semantic information.

[0031] Textual instructions and compressed visual features generate chemical reaction information through a reaction information generator. By sharing the same embedding space with visual tokens and language tokens, the model can perform multimodal reasoning directly on the same sequence.

[0032] The model outputs accurate and fluent natural language text describing chemical reactions, including reaction type identification, changes in the reaction process, and a complete description of the reaction mechanism. Changes in the reaction process include the identification of changes in the central atom and the breaking and formation of chemical bonds. Depending on the text instructions, the chemical reaction information can be a string of SMILES for reactants and products, the reaction type name, a natural language description of the reaction mechanism (e.g., "This reaction is a Diels-Alder cycloaddition reaction. The π electrons of the conjugated diene flow to the double bond of the dienophile, forming a six-membered ring transition state, followed by the simultaneous formation of two new carbon-carbon σ bonds…"), a string of SMILES for predicted products, or executable code that generates a molecular image.

[0033] Chemical reaction information is generated through inference within the model, without the need for external knowledge bases or rules, achieving a direct mapping from pixels to professional text.

[0034] In this embodiment of the invention, a chemical reaction information extraction model is pre-constructed. After extracting visual tokens, the visual encoder in the model compresses these tokens, significantly reducing computational redundancy in the background region while focusing on and preserving key information about the molecular structure. This also significantly reduces computational and storage costs while retaining semantic information. Simultaneously, the reaction information generator understands textual instructions and, combined with the visual features output by the visual encoder, directly generates text-level output from pixel-level input.

[0035] In an exemplary embodiment of the present invention, the visual encoder includes an image block embedding layer and a plurality of transformer blocks connected in sequence; The step of extracting visual tokens from the chemical reaction image based on the visual encoder and performing compression on the visual tokens to obtain visual features includes: Based on the image embedding layer, the chemical reaction image is segmented to obtain multiple reaction sub-images, a thumbnail image of the chemical reaction image is generated, and the thumbnail image and the multiple reaction sub-images are linearly projected into a visual token sequence; wherein, the visual token sequence includes multiple visual tokens; Each of the transformer blocks calculates the importance score of each visual token in the input visual token sequence. Based on the importance score, a compression operation is performed on the visual tokens in the visual token sequence until the compression operation of the last transformer block is completed, thereby obtaining the visual feature.

[0036] In this embodiment of the invention, the model first receives a chemical reaction image. For high-resolution input, the patch embedding layer processes it using a dynamic resolution strategy, that is, segmenting the chemical reaction image into multiple reaction sub-images of standard resolution (e.g., 448×448 pixels), while simultaneously generating a globally downsampled thumbnail image. All reaction sub-images and the thumbnail image are linearly projected into a series of visual token sequences. Where N is the initial total number of tokens. The dynamic resolution strategy ensures that the model can focus on both local details and preserve the global context.

[0037] The visual encoder includes multiple transformer blocks connected in sequence. The number of transformer blocks can be adjusted as needed, but there must be at least four transformer blocks.

[0038] The visual token sequence output from the image embedding layer is input to the first transformer block. The first transformer block calculates the importance score of each visual token in the input sequence, and then performs compression on each visual token based on the importance score. The compressed visual token sequence is then input to the second transformer block, which performs the same compression operation as the first. This compressed visual token sequence is then input to the third transformer block, and so on, until the last transformer block has completed its compression operation and outputs the visual features. Because each transformer block performs compression on the received visual token sequence, the visual token sequences received by each transformer block are different.

[0039] Specifically, in the visual encoder's first... In each transformer block, the visual token sequence input to that transformer block... Importance assessment is performed using an attention-based adaptive token sampler (ATS) to compute each visual token. Importance score.

[0040] First, calculate the self-attention matrix. : ; in, These represent the query and key matrix in the l-th exchange block, respectively. represents the scaling factor, and T represents the transpose.

[0041] Then, the importance score for each visual token is calculated. : .

[0042] The importance score formula takes into account the salience of visual tokens in the attention map (the degree of correlation with the [CLS] token) and the value norm of their own features, and can accurately identify irrelevant tokens representing the background and key tokens representing molecular structures.

[0043] In an exemplary embodiment of the present invention, performing a compression operation on the visual tokens in the visual token sequence based on the importance score includes: Based on the current transformer block, the score variance is calculated using the importance scores of each visual token in the current input visual token sequence; The fractional variance is compared with a preset threshold to obtain the comparison result; Based on the comparison results, a compression operation is performed on each of the visual tokens; wherein the compression operation includes a token pruning operation and a token merging operation; Based on the visual token after compression, a bias term is calculated such that the next transformer block of the current transformer block calculates the importance score based on the bias term.

[0044] In this embodiment of the invention, in the current transformer block, the fractional variance of the importance scores of all visual tokens in the input visual token sequence is calculated. Extensive prior experimental verification determined that a preset threshold was in place. , such as a preset threshold Set as Fractional variance With preset threshold The comparison is performed to obtain the comparison results, and dynamic decisions are made based on the comparison results to determine which compression operation to perform.

[0045] Specifically, when fractional variance Less than or equal to the preset threshold At that time, token pruning is performed when the fractional variance... Greater than the preset threshold When performing a token merging operation, it can be represented as: .

[0046] In this embodiment of the invention, a lower fractional variance means that the importance of all visual tokens is similar, which usually corresponds to a large area of ​​uniform background. In this case, a pruning strategy is used to directly discard redundant information. A higher fractional variance means that the importance of visual tokens varies significantly, which usually corresponds to information-dense complex molecular structures. In this case, a merging strategy is used to extract and fuse structural information.

[0047] To compress visual tokens without losing key information, a proportional attention mechanism is introduced. This involves maintaining a row vector. Elements in a row vector Recorded visual tokens The original number of tokens represented. After a token merge or token pruning operation, This will be updated accordingly. For example, in the l-th layer exchange block, a row vector with the same length as the current visual token sequence is maintained. Each element... It records an important piece of information: how many visual tokens from the original input were merged (or pruned) to form the current i-th visual token.

[0048] In the attention matrix calculation of the next layer (l+1) exchange block, When added as a bias term to the attention matrix, it can be represented as: ; By enhancing the bias term, visual tokens that represent more original information can receive higher weight in attention allocation, ensuring the fidelity of the information flow.

[0049] Visual token sequence after compression Sent to the Each transformer block. This process iterates layer by layer within the visual encoder, gradually reducing the visual token sequence and ultimately outputting a highly condensed and information-rich visual feature representation, laying an efficient and accurate foundation for subsequent generation of mechanistic descriptions.

[0050] In an exemplary embodiment of the present invention, performing a compression operation on each of the visual tokens based on the comparison result includes: If the variance is greater than the preset threshold, the visual tokens in the currently input visual token sequence are divided into a first token group and a second token group, and the visual tokens in the first token group and the visual tokens in the second token group are combined to form a token pair. Calculate the similarity between the two visual tokens in the token pair; The token pairs are sorted in descending order based on the similarity to obtain a first sorting result, and the target token pair is determined based on the first sorting result. Based on the similarity, a token merging operation is performed on the two visual tokens in the target token pair; If the variance is less than or equal to the preset threshold, the visual tokens in the current input visual token sequence are sorted in descending order based on the importance score to obtain a second sorting result; Based on the second permutation result, the target visual token is determined, and in the currently input visual token sequence, token pruning operation is performed on other visual tokens besides the target visual token.

[0051] In this embodiment of the invention, when a token merging operation is triggered, the Bipartite Graph Soft Matching (BSM) algorithm is used for processing. Specifically, the currently input visual token sequence is... The visual tokens are randomly divided into two first token groups of as equal size as possible. Second token group The first token group Each visual token is associated with the second token group. All visual tokens are paired, and the cosine similarity between the two visual tokens in each pair is calculated. Visual tokens in Find the most similar partner among them.

[0052] Sort the token pairs in descending order based on similarity and select the pairs with the highest similarity. 1. Select 1 target token pair as the target token pair. Use similarity as the weight for each target token pair. A weighted average is performed, and the results are merged into a new visual token. Unmatched visual tokens remain unchanged.

[0053] Through the technical solutions of the embodiments of the present invention, visual tokens that are spatially discontinuous can be intelligently merged without relying on spatial adjacency, as long as their features are similar (such as different parts belonging to the same functional group), thereby forming a more complete representation of the molecular structure.

[0054] When the token pruning operation is triggered, the Top-K selection algorithm is used to prune the current input visual token sequence. All visual tokens are sorted in descending order of their importance score, and only the top ones are retained. The most important target visual tokens are used to form a compressed visual token sequence. . It is a preset, fixed value, ensuring the controllability and predictability of computational load. K can be different in token pruning and token merging operations. This operation can efficiently filter out over 80% of background tokens.

[0055] In an exemplary embodiment of the present invention, the reaction information generator includes a text feature extraction module, a projection layer, and a reaction information generation module; The step of generating the chemical reaction information corresponding to the text instruction based on the reaction information generator and the visual features includes: Based on the text feature extraction module, the text features of the text instruction are extracted; Based on the projection layer, the text features are aligned with the visual features; Based on the reaction information generation module, the chemical reaction information is generated by aligning the text features and the visual features.

[0056] In this embodiment of the invention, the text feature extraction module extracts text features from the text instructions, and the compressed visual features are aligned with the text feature space through a projection layer. The reaction information generation module integrates the visual and text features and generates the final output, i.e., the chemical reaction information, through an autoregressive approach. In this embodiment of the invention, the reaction information generation module can be based on a large language model.

[0057] In an exemplary embodiment of the present invention, the training process of the chemical reaction information extraction model includes: Obtain a multi-task training set; wherein, the multi-task training set includes training samples corresponding to multiple tasks, and the training samples include sample images and sample instructions; The training samples are input into the initial chemical reaction information extraction model for iterative training until the training is completed, thereby obtaining the chemical reaction information extraction model; wherein, the initial chemical reaction information extraction model includes a visual encoder and a reaction information generator.

[0058] In this embodiment of the invention, constructing a high-quality, large-scale, and diverse training dataset is fundamental to successfully training a chemical reaction information extraction model. To enable the model to possess comprehensive chemical understanding and reasoning capabilities, a large-scale dataset covering molecular and reaction-level tasks, namely a multi-task training set, was constructed.

[0059] Molecular and reaction data were systematically obtained from multiple authoritative publicly available chemical databases, including: ORDERly dataset: Provides a large amount of well-labeled chemical reaction data, suitable for advanced tasks such as reaction identification and reaction prediction; ChEBI-20-MM dataset: Contains a large number of chemical entities and their annotations, suitable for molecular recognition tasks; MolGrapher, MolScribe, and EDU-CHEMC datasets: provide paired data of molecular images and corresponding SMILES, with EDU-CHEMC specifically containing challenging handwritten molecular images.

[0060] For data containing only the string "SMILES", the corresponding molecular image is generated through subsequent rendering; for data that already contains images, it is directly included in the multi-task training set and its annotation quality is ensured.

[0061] Standardized image rendering of chemical structures, i.e., using cheminformatics tools RDKit and Indigo to convert SMILES strings into high-quality molecular and reaction images.

[0062] A diversified rendering strategy is implemented to enhance the robustness and generalization ability of the model. By defining a series of variable rendering parameters and randomly combining them each time a model is generated, sufficient diversity in visual features is ensured in the generated molecular and reaction images. Specific parameters include: Molecular representation: bond thickness and color, whether atom labels are displayed, background color and transparency; Layout style: molecular orientation, layout algorithm (e.g., KK layout), overall size and scale; Image attributes: resolution (from 224×224 to 800×800), image format (PNG, SVG); For reaction images, ensure that reactants, reagents, solvents and products have a clear spatial arrangement in the image to facilitate the model's understanding of the reaction process.

[0063] Rich and structured text annotations are constructed for each molecular and reaction image to support the model in learning different chemical understanding capabilities. For molecular images, the annotation is its SMILES string; for reaction images, the annotation may include: SMILES strings of reactants, reagents, and products; reaction type labels (such as "nucleophilic substitution reaction"); and natural language descriptions of the reaction mechanism (for mechanism description generation tasks).

[0064] Specifically, for the molecular recognition task: the corresponding SMILES strings are labeled to train the model's ability to convert images into structured text; For reaction identification tasks: the labeling uses the format of reactant > reagent.solvent > product, each component is represented by SMILES, and multiple molecules are separated by periods; For the reaction type classification task: label the corresponding reaction type (such as "nucleophilic substitution", "Diesel-Alder reaction", etc.).

[0065] For property prediction tasks: RDKit is used to calculate a series of physicochemical properties of molecules as numerical labels, such as MW (Molecular Weight), LogP (Partition Coefficient), TPSA (Topological Polar Surface Area), HBD (Hydrogen Bond Donor), HBA (Hydrogen Bond Acceptor), RB (Rotatable Bonds), and QED (Quantitative Estimate of Drug-likeness). For reaction prediction tasks: only reactant images are provided, labeled as the corresponding products SMILES, to train the model's chemical reasoning ability; For the task of generating mechanism descriptions: construct natural language descriptions to explain in detail the key steps in the reaction process, such as bond breaking and formation, electron transfer, and intermediate formation; For handwritten molecular images (such as those from the EDU-CHEMC dataset), we developed dedicated parsing code to convert their native format (SSML) into SMILES strings, forming high-quality handwritten image-SMILES paired data, specifically to improve the model's ability to handle non-standard inputs; For molecular image generation tasks, executable code is used as an intermediary representation, replacing the traditional <|image|> tag with the corresponding Python rendering code. The code directly contains the SMILES string and drawing instructions, making the generated results directly verifiable and executable.

[0066] Through the systematic data construction process described above, a large-scale, multi-task chemical vision-language dataset containing approximately 1.25 million samples was ultimately formed, covering various chemical understanding tasks from the molecular level to the reaction level. This laid a solid foundation for training high-performance chemical vision-language models.

[0067] Task Definition: The final dataset covers multiple tasks, including molecular recognition, reaction recognition, reaction type classification, property prediction, reaction prediction, and mechanism description generation.

[0068] As mentioned above, the various tasks in the embodiments of the present invention include at least two of the following: molecular recognition task, reaction recognition task, reaction type classification task, property prediction task, and reaction prediction task.

[0069] Based on the aforementioned processing, the multi-task training set contains image-text pairs for the following tasks: Molecular recognition task: Input a molecular image, output the corresponding SMILES string; Reaction recognition task: Input a complete reaction image and output the SMILES strings for reactants, reagents, and products; Reaction type classification task: Input reaction image, output reaction type label (e.g., nucleophilic substitution); Property prediction task: Input a molecular image and output molecular properties (such as molecular weight, LogP). Reaction prediction task: Input reactant images, directly predict and output the SMILES string of products. This task is key to evaluating the model's chemical reasoning ability; Mechanism description generation task: Input reaction image and text instructions (such as "describe the mechanism of this reaction"), output reaction mechanism described in natural language; In the above image-text pairs, the image serves as the sample image, and the text serves as the sample instruction, forming a training sample. The real chemical reaction information corresponding to each image-text pair serves as the corresponding sample label.

[0070] In this embodiment of the invention, the initial chemical response information extraction model includes a visual encoder and a response information generator. The initial chemical response information extraction model adopts the standard ViT-MLPLLM architecture, where the visual encoder is responsible for image processing and the response information generator is responsible for text understanding and generation.

[0071] Specifically, the visual encoder uses a multimodal visual language model (such as InternVL) as its backbone network for training. The response information generator uses a large language model as its backbone network for training.

[0072] Using a constructed multi-task dataset, the initial chemical reaction information extraction model was fine-tuned with full parameter supervision. The training objective was to minimize the loss between the model's predictions (such as generated SMILES, reaction type labels, and mechanism description text) and the true sample labels. Thanks to the efficiency improvements brought by visual token optimization, the training and inference times of the model when processing high-resolution chemical images were significantly reduced, and the computational resource consumption was greatly reduced, making it possible to train a high-performance chemical visual-language model with limited computing power. Through multi-task learning, the model simultaneously optimizes multiple objectives such as molecule recognition, reaction prediction, and mechanism description, thereby learning a general chemical visual-language representation.

[0073] Based on the aforementioned training, the core of the model is a unified chemical visual language model that generates text-level output directly from pixel-level input in an end-to-end manner. The model implicitly learns chemical knowledge and makes inferences through its internal large language model components, rather than relying on external knowledge bases or manually constructed rules.

[0074] In this embodiment of the invention, to enable the model to generate images, executable code is used as an intermediate representation. The model learns to generate Python code containing the string "SMILES" and plotting instructions, rather than directly generating image pixels. This approach allows the generated molecular images to be directly verified and obtained by executing the code, without relying on external parsing tools.

[0075] In this embodiment of the invention, after the above-described efficient architecture design and large-scale multi-task training, the chemical reaction information extraction model constructed by this invention exhibits excellent performance and practicality in multiple dimensions. Its output capability and effect evaluation are specifically reflected in the following four aspects: High inference speed and significantly reduced computational overhead: Thanks to the core adaptive visual token optimization, the computational load of the model is significantly reduced when processing high-resolution chemical images. As shown in Table 1 below, under the same hardware environment, the chemical reaction information extraction model (Ours) of this invention achieves an inference throughput of 11.84 samples / second, which is significantly higher than the general model InternVL2.5-4B (9.11 samples / second) and the chemical domain model ChemVLM-8B (7.41 samples / second).

[0076] Table 1

[0077] More importantly, as shown in Table 1, the average number of input tokens (including visual and text tokens) for the chemical reaction information extraction model was successfully reduced to 108, which is only about 1 / 8 of the comparison model (~894 tokens). This dramatic reduction in the number of tokens directly translates into a leap in training speed, with full parameter fine-tuning taking only 15 hours, far less than the baseline model, making it possible to conduct cutting-edge research with limited computing power.

[0078] Superior Molecular Visual Understanding Capabilities: The chemical reaction information extraction model demonstrates its accurate visual perception capabilities in fundamental molecular recognition and property prediction tasks. Molecular Recognition: In the ChemOCR benchmark test, the chemical reaction information extraction model achieved an average Tanimoto similarity (Avg. Sim.) of 91.2% and a Tanimoto hit rate (Tani@1.0) of 77.4%, comprehensively outperforming all comparable models, including GPT-4o and the ChemVLM series, reaching a state-of-the-art (SOTA) level. Particularly noteworthy is that the chemical reaction information extraction model achieves performance comparable to specialized OCR models (such as DECIMER) in molecular recognition tasks, demonstrating the significant potential of general-purpose VLMs for specialized tasks. Property Prediction: On the img2property task, the chemical reaction information extraction model has a lower mean squared error (MSE) than all baseline models in predicting most properties such as molecular weight (MW), topological polar surface area (TPSA), and hydrogen bond donor / acceptor (HBD / HBA). The error is about half that of the second-ranked ChemMLLM, demonstrating its ultra-high accuracy in regressing molecular properties from images.

[0079] Exceptional chemical reaction-level visual reasoning ability: such as Figure 4 As shown in Table 2, the chemical reaction information extraction model demonstrates groundbreaking performance on the novel ChemRxn-V visual reaction benchmark. Reaction Recognition and Prediction: As shown in Table 2, in the reaction recognition task, the chemical reaction information extraction model achieves an average similarity (Avg. Sim.) of 93.4% and an exact match (EM) rate of 67.9%. In the more challenging reaction prediction task (i.e., predicting products solely from reactant images), the model achieves an average similarity of 78.9% and a Tani@1.0 hit rate of 52.4%. In stark contrast, all comparison models (including GPT-4o, ChemVLM, etc.) perform extremely poorly on this task (Tani@1.0's highest hit rate is only 1.4%), fully demonstrating that the chemical reaction information extraction model has achieved a qualitative leap in its ability to perform chemical logical reasoning through visual signals.

[0080] Table 2

[0081] Table 2 Performance comparison on ChemRxn-V benchmark (unit: %) Mechanism Description Generation: such as Figure 3 As shown, the chemical reaction information extraction model can receive reaction images and natural language instructions (e.g., "Describe the reaction mechanism shown in the figure below") and output professional and fluent text describing the mechanism. For example, for an esterification reaction, the model can generate a description such as "The oxygen atom in the carboxyl group nucleophilically attacks the carbonyl carbon atom under acid catalysis, followed by proton transfer and the elimination of a water molecule, forming an ester bond..." This end-to-end pixel-to-text generation capability provides readable and interpretable "thought chain" data for chemical reaction prediction, serving as a key bridge connecting visual perception and chemical knowledge reasoning.

[0082] A novel and practical paradigm for molecular image generation: This invention employs an innovative paradigm mediated by executable code for molecular image generation tasks. Instead of directly generating pixel images that are difficult to evaluate, the model generates Python code that can call libraries such as RDKit. This code directly contains the product's SMILES string and plotting instructions.

[0083] In summary, the chemical reaction information extraction model successfully achieved the goals of being "smaller, faster, and stronger" through the collaborative design of an efficient model architecture and complex chemical tasks. It serves not only as an efficient chemical image recognition tool but also as a chemical research assistant with deep reasoning capabilities, providing a novel vision-based solution for understanding and predicting chemical reactions by generating reliable mechanistic descriptions.

[0084] The chemical reaction information extraction device provided by the present invention will be described below. The chemical reaction information extraction device described below can be referred to in correspondence with the chemical reaction information extraction method described above. It should be noted that the device provided in the following embodiments and the method provided in the above embodiments belong to the same concept, and the specific way in which each module and unit performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0085] In one exemplary embodiment of the present invention, please refer to Figure 5 , Figure 5 A chemical reaction information extraction device according to an exemplary embodiment includes the following modules: The first acquisition module 510 is configured to acquire chemical reaction images and text commands; The extraction module 520 is configured to input the chemical reaction image and the text instruction into a chemical reaction information extraction model to obtain chemical reaction information corresponding to the text instruction; wherein, the chemical reaction information extraction model includes a visual encoder and a reaction information generator; The extraction module 520 includes: The extraction submodule is configured to extract visual tokens from the chemical reaction image based on the visual encoder, and perform compression operations on the visual tokens to obtain visual features; The generation submodule is configured to generate the chemical reaction information corresponding to the text instruction based on the reaction information generator and the visual features.

[0086] In an exemplary embodiment of the present invention, the visual encoder includes an image block embedding layer and a plurality of transformer blocks connected in sequence; Extraction submodules, including: The projection unit is configured to segment the chemical reaction image based on the image embedding layer to obtain multiple reaction sub-images, generate a thumbnail image of the chemical reaction image, and linearly project the thumbnail image and the multiple reaction sub-images into a visual token sequence; wherein the visual token sequence includes multiple visual tokens; The compression unit is configured to calculate the importance score of each visual token in the input visual token sequence through each of the transformer blocks, and perform a compression operation on the visual tokens in the visual token sequence based on the importance score, until the compression operation of the last transformer block is completed, thereby obtaining the visual feature.

[0087] In an exemplary embodiment of the present invention, the compression unit includes: The first computational subunit is configured to calculate the score variance based on the current transformer block, using the importance scores of each visual token in the currently input visual token sequence. The comparison subunit is configured to compare the fractional variance with a preset threshold to obtain a comparison result; An execution subunit is configured to perform a compression operation on each of the visual tokens based on the comparison result; wherein the compression operation includes a token pruning operation and a token merging operation; The second computational subunit is configured to compute a bias term based on the visual token after compression, such that the next transformer block of the current transformer block computes the importance score based on the bias term.

[0088] In an exemplary embodiment of the present invention, the execution subunit includes: The partitioning structure is configured such that if the variance is greater than the preset threshold, the visual tokens in the currently input visual token sequence are divided into a first token group and a second token group, and the visual tokens in the first token group and the visual tokens in the second token group are combined to form token pairs. A computational structure is configured to calculate the similarity between two visual tokens in the token pair; The structure is determined and configured to sort the token pairs in descending order based on the similarity to obtain a first sorting result, and the target token pair is determined based on the first sorting result. The first execution structure is configured to perform a token merging operation on the two visual tokens in the target token pair based on the similarity. The arrangement structure is configured such that if the variance is less than or equal to the preset threshold, the visual tokens in the currently input visual token sequence are arranged in descending order based on the importance score to obtain a second arrangement result; The second execution structure is configured to determine the target visual token based on the second permutation result, and to perform token pruning operations on other visual tokens besides the target visual token in the currently input visual token sequence.

[0089] In an exemplary embodiment of the present invention, the reaction information generator includes a text feature extraction module, a projection layer, and a reaction information generation module; Generate submodules, including: The extraction unit is configured to extract the text features of the text instruction based on the text feature extraction module; The alignment unit is configured to align the text features with the visual features based on the projection layer; The generation unit is configured to generate the chemical reaction information based on the reaction information generation module, using the aligned text features and the visual features.

[0090] In one exemplary embodiment of the present invention, a chemical reaction information extraction device includes: The second acquisition module is configured to acquire a multi-task training set; wherein, the multi-task training set includes training samples corresponding to multiple tasks, and the training samples include sample images and sample instructions; The iterative training module is configured to input training samples into the initial chemical reaction information extraction model for iterative training until the training is completed, thereby obtaining the chemical reaction information extraction model; wherein, the initial chemical reaction information extraction model includes a visual encoder and a reaction information generator.

[0091] In an exemplary embodiment of the present invention, the multiple tasks include at least two of the following: molecular recognition task, reaction recognition task, reaction type classification task, property prediction task, and reaction prediction task.

[0092] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a chemical reaction information extraction method, which includes: acquiring chemical reaction images and text instructions; The chemical reaction image and the text command are input into the chemical reaction information extraction model to obtain the chemical reaction information corresponding to the text command; wherein, the chemical reaction information extraction model includes a visual encoder and a reaction information generator; The step of inputting the chemical reaction image and the text command into the chemical reaction information extraction model to obtain the chemical reaction information corresponding to the text command includes: Based on the visual encoder, visual tokens of the chemical reaction image are extracted, and compression is performed on the visual tokens to obtain visual features; Based on the reaction information generator and the visual features, the chemical reaction information corresponding to the text command is generated.

[0093] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the chemical reaction information extraction method provided by the above methods, the method including: acquiring chemical reaction images and text instructions; The chemical reaction image and the text command are input into the chemical reaction information extraction model to obtain the chemical reaction information corresponding to the text command; wherein, the chemical reaction information extraction model includes a visual encoder and a reaction information generator; The step of inputting the chemical reaction image and the text command into the chemical reaction information extraction model to obtain the chemical reaction information corresponding to the text command includes: Based on the visual encoder, visual tokens of the chemical reaction image are extracted, and compression is performed on the visual tokens to obtain visual features; Based on the reaction information generator and the visual features, the chemical reaction information corresponding to the text command is generated.

[0095] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the chemical reaction information extraction method provided by the above methods, the method comprising: acquiring chemical reaction images and text instructions; The chemical reaction image and the text command are input into the chemical reaction information extraction model to obtain the chemical reaction information corresponding to the text command; wherein, the chemical reaction information extraction model includes a visual encoder and a reaction information generator; The step of inputting the chemical reaction image and the text command into the chemical reaction information extraction model to obtain the chemical reaction information corresponding to the text command includes: Based on the visual encoder, visual tokens of the chemical reaction image are extracted, and compression is performed on the visual tokens to obtain visual features; Based on the reaction information generator and the visual features, the chemical reaction information corresponding to the text command is generated.

[0096] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting chemical reaction information, characterized in that, include: Acquire images and text commands related to chemical reactions; The chemical reaction image and the text command are input into the chemical reaction information extraction model to obtain the chemical reaction information corresponding to the text command; wherein, the chemical reaction information extraction model includes a visual encoder and a reaction information generator; The step of inputting the chemical reaction image and the text command into the chemical reaction information extraction model to obtain the chemical reaction information corresponding to the text command includes: Based on the visual encoder, visual tokens of the chemical reaction image are extracted, and compression is performed on the visual tokens to obtain visual features; Based on the reaction information generator and the visual features, the chemical reaction information corresponding to the text command is generated.

2. The method for extracting chemical reaction information according to claim 1, characterized in that, The visual encoder includes an image block embedding layer and multiple transformer blocks connected in sequence; The step of extracting visual tokens from the chemical reaction image based on the visual encoder and performing compression on the visual tokens to obtain visual features includes: Based on the image embedding layer, the chemical reaction image is segmented to obtain multiple reaction sub-images, a thumbnail image of the chemical reaction image is generated, and the thumbnail image and the multiple reaction sub-images are linearly projected into a visual token sequence; wherein, the visual token sequence includes multiple visual tokens; Each of the transformer blocks calculates the importance score of each visual token in the input visual token sequence. Based on the importance score, a compression operation is performed on the visual tokens in the visual token sequence until the compression operation of the last transformer block is completed, thereby obtaining the visual feature.

3. The method for extracting chemical reaction information according to claim 2, characterized in that, The step of performing compression on visual tokens in the visual token sequence based on the importance score includes: Based on the current transformer block, the score variance is calculated using the importance scores of each visual token in the current input visual token sequence; The fractional variance is compared with a preset threshold to obtain the comparison result; Based on the comparison results, a compression operation is performed on each of the visual tokens; wherein the compression operation includes a token pruning operation and a token merging operation; Based on the visual token after compression, a bias term is calculated such that the next transformer block of the current transformer block calculates the importance score based on the bias term.

4. The method for extracting chemical reaction information according to claim 3, characterized in that, The compression operation performed on each of the visual tokens based on the comparison result includes: If the variance is greater than the preset threshold, the visual tokens in the currently input visual token sequence are divided into a first token group and a second token group, and the visual tokens in the first token group and the visual tokens in the second token group are combined to form a token pair. Calculate the similarity between the two visual tokens in the token pair; The token pairs are sorted in descending order based on the similarity to obtain a first sorting result, and the target token pair is determined based on the first sorting result. Based on the similarity, a token merging operation is performed on the two visual tokens in the target token pair; If the variance is less than or equal to the preset threshold, the visual tokens in the current input visual token sequence are sorted in descending order based on the importance score to obtain a second sorting result; Based on the second permutation result, the target visual token is determined, and in the currently input visual token sequence, token pruning operation is performed on other visual tokens besides the target visual token.

5. The method for extracting chemical reaction information according to claim 1, characterized in that, The reaction information generator includes a text feature extraction module, a projection layer, and a reaction information generation module; The step of generating the chemical reaction information corresponding to the text instruction based on the reaction information generator and the visual features includes: Based on the text feature extraction module, the text features of the text instruction are extracted; Based on the projection layer, the text features are aligned with the visual features; Based on the reaction information generation module, the chemical reaction information is generated by aligning the text features and the visual features.

6. The method for extracting chemical reaction information according to any one of claims 1 to 5, characterized in that, The training process of the chemical reaction information extraction model includes: Obtain a multi-task training set; wherein, the multi-task training set includes training samples corresponding to multiple tasks, and the training samples include sample images and sample instructions; The training samples are input into the initial chemical reaction information extraction model for iterative training until the training is completed, thereby obtaining the chemical reaction information extraction model; wherein, the initial chemical reaction information extraction model includes a visual encoder and a reaction information generator.

7. The method for extracting chemical reaction information according to claim 6, characterized in that, The multiple tasks include at least two of the following: molecular recognition task, reaction recognition task, reaction type classification task, property prediction task, and reaction prediction task.

8. A chemical reaction information extraction device, characterized in that, include: The first acquisition module is configured to acquire chemical reaction images and text commands; The extraction module is configured to input the chemical reaction image and the text command into a chemical reaction information extraction model to obtain chemical reaction information corresponding to the text command; wherein, the chemical reaction information extraction model includes a visual encoder and a reaction information generator; The extraction module includes: The extraction submodule is configured to extract visual tokens from the chemical reaction image based on the visual encoder, and perform compression operations on the visual tokens to obtain visual features; The generation submodule is configured to generate the chemical reaction information corresponding to the text instruction based on the reaction information generator and the visual features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the chemical reaction information extraction method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the chemical reaction information extraction method as described in any one of claims 1 to 7.

Images