A Spectral Data Mining and Reconstruction System and Method Based on Multimodal Information Extraction

By extracting and reconstructing catalytic reaction information from infrared spectral images and text using a multimodal information extraction system, the problems of incomplete and inefficient data acquisition in existing technologies have been solved. A high-quality infrared spectral knowledge base has been built, which has promoted in-depth research on catalytic reactions.

CN123116189APending Publication Date: 2026-11-17DALIAN INSTITUTE OF CHEMICAL PHYSICS CHINESE ACADEMY OF SCIENCES
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510634537.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing infrared spectroscopy data acquisition methods mainly rely on single-modal information processing, resulting in incomplete data acquisition, low efficiency, and a lack of systematic multimodal data reconstruction methods, which limits the in-depth development of catalytic reaction research.

Method used

A spectroscopic data mining system employing multimodal information extraction, combined with machine vision and natural language processing technologies, extracts and reconstructs catalytic reaction information from infrared spectral images and related texts. This includes modules for spectral information target detection, information region character recognition, infrared spectral data reconstruction, and multimodal information integration and knowledge base construction.

Benefits of technology

It achieves efficient and accurate collaborative extraction of multimodal information, constructs a structured infrared spectroscopy knowledge base, improves data acquisition efficiency, and promotes the study of catalytic reaction mechanisms and the optimized design of catalysts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123116189A_ABST
    Figure CN123116189A_ABST
Patent Text Reader

Abstract

This invention provides a spectroscopic data mining and reconstruction system based on multimodal information extraction, belonging to the technical field of energy catalytic reactions. It includes: a spectral information target detection module, an information region character recognition module, an infrared spectral data reconstruction module, a catalytic reaction spectral text information extraction module, and a multimodal information integration and knowledge base construction module. The system extracts infrared spectra of catalytic reactions and related descriptive text from published literature, and preprocesses the extracted catalytic reaction infrared spectra and descriptive text. The preprocessed information is then used to detect targets in information regions through the spectral information target detection module, and a spectral region character recognition model is constructed through the information region character recognition module to recognize characters in the spectral regions. Furthermore, the catalytic reaction spectral text information extraction module extracts text information from the catalytic reaction spectral data, and finally, the multimodal information integration and knowledge base construction module integrates and constructs the multimodal information and knowledge base.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the technical field of energy catalytic reactions, and more particularly to a spectroscopic data mining and reconstruction system and method based on multimodal information extraction. Background Technology

[0002] Infrared spectroscopy, with its unique molecular fingerprinting capabilities, has become a key analytical tool for characterizing catalyst surface structures, elucidating reaction mechanisms, and monitoring adsorption processes. With the development of intelligent computing technology, the demand for deeper and more intelligent analysis and research using infrared spectroscopy is increasing. Constructing a high-quality infrared spectral database is fundamental to achieving accurate analysis; however, current experimental data acquisition faces numerous limitations, primarily manifested in over-reliance on theoretical simulations and a lack of effective integration of multimodal data. This significantly hinders the in-depth development of related research.

[0003] While some methods for processing and analyzing infrared spectral data exist in the current technology, most of them suffer from the following shortcomings:

[0004] Single-modal information processing: Traditional data acquisition methods mainly focus on processing single-modal information, such as extracting data only from images or text, while ignoring the rich correlation information between images and text. This results in the inability to comprehensively and accurately obtain complete information about catalytic reactions, thus affecting a deeper understanding of the catalytic reaction mechanism.

[0005] Low data acquisition efficiency: Manually extracting infrared spectral data and related information from scientific literature is not only time-consuming and labor-intensive, but also prone to errors. Especially when faced with massive amounts of literature, this inefficient data acquisition method cannot meet the needs of large-scale data analysis and intelligent computing.

[0006] Lack of systematic data reconstruction methods: Existing data processing techniques often only provide local or fragmented data, lacking the ability to systematically integrate and reconstruct spectral data from images with key information from text. This makes it difficult for researchers to build a complete knowledge base, thus limiting the further development of data-driven catalytic reaction research. Summary of the Invention

[0007] In response to the technical problems mentioned in the background section, this invention provides a spectral data mining and reconstruction system and method based on multimodal information extraction. This invention efficiently and accurately extracts and reconstructs infrared spectral data and related catalytic reaction information from scientific literature by combining machine vision and natural language processing technologies. The method first performs target detection on regions containing key information in the spectral image. Then, a spectral region character recognition module efficiently converts information from image mode to text or data mode. Next, an infrared spectral data reconstruction module acquires and reconstructs the spectral data. Simultaneously, important entity information in the associated text is identified and extracted. Finally, a multimodal information integration and knowledge base construction module establishes a structured morphological knowledge base for the information and data extracted from images and text.

[0008] The technical means employed in this invention are as follows:

[0009] A spectroscopic data mining and reconstruction system based on multimodal information extraction includes: a spectral information target detection module, an information region character recognition module, an infrared spectral data reconstruction module, a catalytic reaction spectrum text information extraction module, and a multimodal information integration and knowledge base construction module;

[0010] The infrared spectra and related texts describing the catalytic reaction are extracted from publicly available literature and preprocessed. The preprocessed information is then used to detect targets in the information region through the spectral information target detection module. A TroOCR-LSTM-MultiAtt spectral region character recognition model is then constructed through the information region character recognition module to recognize characters in the spectral region. Subsequently, the catalytic reaction spectrum text information extraction module extracts the catalytic reaction spectrum text information. Finally, the multimodal information integration and knowledge base construction module integrates the multimodal information and constructs the knowledge base.

[0011] Furthermore, the spectral information target detection module uses YOLO target detection technology to identify and segment key information regions in infrared spectral images, including the following steps:

[0012] Step 11: Perform standardization preprocessing on the infrared spectral images to construct a literature image dataset;

[0013] Step 12: Input the document image dataset into the IR-YOLOv5-DC3 model based on YOLOv5 improvement; the improved IR-YOLOv5-DC3 model is trained using an infrared spectral annotation dataset, and the improved IR-YOLOv5-DC3 model includes: a feature extraction backbone network, a multi-scale feature fusion network, a detection output network, and a dynamic convolution DC3 module;

[0014] Step 13: Output the bounding box coordinates and detection confidence of the target region.

[0015] Furthermore, the information region character recognition module uses the TrOCR-LSTM-MultiAtt spectral region character recognition method to recognize and extract character information from infrared spectral images, including the following steps:

[0016] Step 21: Extract the infrared spectrum of the catalytic reaction from the published literature and input it into the IR-YOLOv5-DC3 spectrum information target detection module. Collect all the output results of the spectrum information target detection module and establish the dataset used by the information region character recognition module.

[0017] Step 22: Use the visual Transformer encoder to extract features from the preprocessed image, and introduce a BiLSTM module based on multi-head attention feature fusion in the Transformer decoder part. Input the output feature vector of the decoder into the BiLSTM module to further learn sequence features and obtain a new query vector feature Q.

[0018] Step 23: Using the high-dimensional multimodal fusion feature vector output by the Transformer decoder after fusing image features and input text sequence features as the key vector and value vector, input them into the multi-head attention model to learn the multi-head attention weights:

[0019]

[0020] Where Q represents the query vector, K represents the key vector, V represents the value vector, and d k This represents the dimension of the key vector.

[0021] 4. A spectroscopic data mining and reconstruction system based on multimodal information extraction according to claim 1, characterized in that the infrared spectral data reconstruction module realizes the extraction and data reconstruction of the internal region of the infrared spectral image through processing and analysis of the infrared spectral image, including the following steps:

[0022] Step 31: Perform color recognition and segmentation processing on the infrared spectral image; use target detection to accurately locate and extract the legend region in the spectral image, enhance the edge range of the legend's color bands using an edge detection algorithm, and determine the RGB value range of each color band of the legend, denoted as RGBrange_legends=[L1,L2,…,L…]. n ], where S1, S2, ..., S nThese represent the RBG intervals for each color identified by the color clustering algorithm. The format of each color interval is [(R1,G1,B1),(R′1,G′1,B′1)], where R, G, and B are the values ​​of the three corresponding color channels (Red, Green, Blue).

[0023] Step 32: Randomly select three in-situ spectral image segments with fixed wavenumber ranges (i.e., the horizontal axis) from the original infrared spectral image obtained from scientific literature. Then, use an adaptive color clustering algorithm to cluster and identify the RGB intervals corresponding to multiple color spectra, denoted as RGBrange_spec = [S1, S2, ..., S...]. n Its content format is consistent with RGBrange_legends;

[0024] Step 33: Calculate the average RGB value for each RGB interval range, denoted as [eL1, eL2, ..., eL... n ] and [eS1,eS2,…,eS n ], where eL i (LR) i LG i ,LB i ), eS i Represents (SR) i ,SG i ,SB i ), that is, the average value of the i-th RGBrange_legends interval and the RGBrange_spec interval, which is a three-dimensional point information;

[0025] Step 34: Construct the Euclidean distance algorithm for eL i With eS i The similarity calculation involves matching the spectral colors with the colors of the legend regions extracted from the infrared spectra of the catalytic reaction extracted from published literature. The calculation formula is as follows:

[0026]

[0027] Where, d i The result of the similarity calculation;

[0028] Step 35: Based on the RGB interval pixel extraction algorithm, the RGB values ​​of each pixel on the infrared spectrum of the catalytic reaction are extracted by traversing the published literature. Each pixel is classified into the pixel RGB interval corresponding to each original legend area according to the interval in which the RGB value is located. All pixels corresponding to each legend RGB interval are extracted and the background is filled with white pixels. That is, all pixels except those that match the RGB interval are adjusted to white to obtain a single spectral curve, thus achieving effective separation of in-situ multi-infrared spectral data.

[0029] The data reconstruction stage of the internal region of the infrared spectral image includes the following steps:

[0030] Step 36: Align the extracted pixel coordinates according to the wavenumber range;

[0031] Step 37: Then normalize the intensity coordinates to the range of 0-100. The normalization formula is:

[0032]

[0033] Among them, y i Represents the original intensity value at coordinate point, y max This represents the highest intensity value of the in-situ spectrum;

[0034] Step 38: To solve the problem of coordinate point redundancy caused by the width of the spectral curve, calculate the arithmetic mean of the ordinates of the same x-coordinate points as the final ordinate under that x-coordinate.

[0035] Step 39: Smooth the infrared spectrum using the Savitzky-Gore filter method and extract the final spectral data;

[0036] The smoothing formula is:

[0037]

[0038] Where m represents the window size, and k represents the index of the element in the convolution kernel. h represents the smoothed data of the i-th point. k This represents the value of the j-th element in the convolution kernel, where n is the value of the j-th element. i+k This represents the data corresponding to the index in the original data.

[0039] Furthermore, the catalytic reaction spectrum text information extraction module utilizes the IR-SciBERT-DCEM-CRF catalytic reaction spectrum text named entity recognition model to achieve semantic understanding and key information extraction from the catalytic reaction spectrum text, including the following steps:

[0040] Step 41: Preprocess the input catalytic reaction spectrum text;

[0041] Step 42: Introduce a dynamic context enhancement module, using SciBERT-base pre-trained weights as initialization, and fine-tune the training with a self-built catalytic reaction text entity recognition dataset to construct the IR-SciBERT-DCEM-CRF catalytic reaction spectrum text entity recognition model. Dynamically fuse the context features of domain keywords through a gated attention mechanism.

[0042] Step 43: Set anchor words to automatically extract high-frequency domain terms from the self-built catalytic reaction text entity recognition dataset;

[0043] Step 44: Calculate the similarity between the word embeddings output by SciBERT and the anchor word embeddings to generate dynamic weights:

[0044]

[0045] Where σ represents the sigmoid activation function, This represents the vector representation of the current token, which is the output vector of SciBERT, k. j denoted as plotted word vector, W represents trainable weight parameters, and b represents bias terms that are automatically adjusted to scalars or vectors;

[0046] Step 45: Concatenate the weighted contextual features with the original SciBERT output, then input them into the CRF layer to finally extract the target entity from the catalytic reaction text sentence from the original scientific paper.

[0047] Furthermore, the multimodal information integration and knowledge base construction module, based on an information integration method with a unified data format and storage structure, realizes the alignment and integration of data extracted from images and text, including the following steps:

[0048] Step 51: Determine the standardized and unified format of infrared spectrum data and catalytic reaction information. Key information includes in-situ infrared spectroscopy experimental data, peak wavenumber, peak product, and reaction conditions.

[0049] Step 52: Align the spectral data reconstructed from infrared spectral images from scientific literature with the key information extracted from the text, mainly by matching whether the peak wavenumber information is equal.

[0050] Step 53: Store the integrated data in a structured knowledge base.

[0051] This invention also includes a method for reconstructing spectroscopic data based on multimodal information extraction, comprising the following steps:

[0052] Step A: Extract the infrared spectra of the catalytic reaction and related text descriptions from published literature, and preprocess the extracted infrared spectra of the catalytic reaction and related text descriptions.

[0053] Step B: The preprocessed information is used to detect targets in the information region through the spectral information target detection module; the targets in the information region include: legend region, peak wavenumber, peak product, and wavenumber range;

[0054] Step C: Construct a TrOCR-LSTM-MultiAtt spectral region character recognition model using the information region character recognition module to achieve spectral region character recognition;

[0055] Step D: Extract text information from the catalytic reaction spectrum using the catalytic reaction spectrum text information extraction module;

[0056] Step E: Multimodal information integration and knowledge base construction are achieved through the multimodal information integration and knowledge base construction module.

[0057] Further, in step A, the preprocessing includes: text preprocessing and image preprocessing; the text preprocessing includes: removing irrelevant information, removing hyphens, sentence segmentation and word segmentation; the image preprocessing includes: size adjustment and grayscale processing.

[0058] Compared with the prior art, the present invention has the following advantages:

[0059] Efficient collaborative extraction of multimodal information: By combining machine vision and natural language processing technologies, experimental data and key information are collaboratively extracted from images and texts in infrared spectroscopy-related scientific research literature, overcoming the limitations of traditional methods and improving the comprehensiveness and accuracy of data acquisition.

[0060] Improve data acquisition efficiency: Utilize advanced information extraction algorithms and models to achieve an automated and efficient data extraction process, significantly reducing manual intervention, improving data acquisition efficiency, and supporting large-scale data-driven research.

[0061] Constructing a systematic database: Integrating and standardizing the extracted spectral data and text information to build a structured infrared spectroscopy knowledge base, providing a high-quality data foundation for in-depth research in the field of energy catalytic reactions, and promoting the intelligent development of this field.

[0062] Facilitating research on catalytic reaction mechanisms: By providing complete and accurate infrared spectral data and related reaction information, it helps researchers better understand catalytic reaction mechanisms, providing strong data support for the optimized design of catalysts, the optimization of reaction processes, and the development of new reactions.

[0063] In summary, this invention addresses the shortcomings of existing infrared spectral data acquisition and analysis technologies by proposing an innovative method for multimodal information extraction and data reconstruction. This method aims to meet the growing demand for high-quality data in the field of energy catalytic reactions and promote the development of scientific research and technological applications in this field to a higher level. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Fig. 1 This is a schematic diagram of the overall system flow of the present invention.

[0066] Fig. 2 This is a schematic diagram of the infrared spectral data reconstruction module of the present invention. Detailed Implementation

[0067] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0068] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0069] like Figs. 1-2 As shown, the present invention provides a spectroscopic data mining and reconstruction system based on multimodal information extraction, including: a spectral information target detection module, an information region character recognition module, an infrared spectral data reconstruction module, a catalytic reaction spectrum text information extraction module, and a multimodal information integration and knowledge base construction module;

[0070] The infrared spectra and related texts describing the catalytic reaction are extracted from publicly available literature and preprocessed. The preprocessed information is then used to detect targets in the information region through the spectral information target detection module. A TroOCR-LSTM-MultiAtt spectral region character recognition model is then constructed through the information region character recognition module to recognize characters in the spectral region. Subsequently, the catalytic reaction spectrum text information extraction module extracts the catalytic reaction spectrum text information. Finally, the multimodal information integration and knowledge base construction module integrates the multimodal information and constructs the knowledge base.

[0071] In a preferred embodiment, the spectral information target detection module uses YOLO target detection technology to identify and segment key information regions in infrared spectral images, including the following steps:

[0072] Step 11: Perform standardization preprocessing on the infrared spectral images to construct a literature image dataset;

[0073] Step 12: Input the document image dataset into the IR-YOLOv5-DC3 model based on YOLOv5 improvement; the improved IR-YOLOv5-DC3 model is trained using an infrared spectral annotation dataset, and the improved IR-YOLOv5-DC3 model includes: a feature extraction backbone network, a multi-scale feature fusion network, a detection output network, and a dynamic convolution DC3 module;

[0074] In this application, as a preferred embodiment, the Labelimg tool is used to annotate spectral image information regions. YOLOv5 includes a feature extraction backbone network, a multi-scale feature fusion network, and a detection output network. The feature extraction backbone network is used for image feature learning, the multi-scale feature fusion network fuses the multi-scale features generated by the backbone network to enhance the feature map, and the detection output network aggregates the feature map and generates detection boxes. Based on the initial architecture of the YOLOv5 model, a dynamic convolution DC3 module is used to replace the original C3 structure. Dynamic feature learning is achieved by calculating dynamic kernel weights, which can adaptively adjust the receptive field of the 3×3 / 5×5 convolution kernel. Further fine-tuning training is performed using a self-built infrared spectral annotation dataset to construct the IR-YOLOv5-DC3 model, realizing the detection capability of multi-scale regions (such as dense annotations and wide charts) in document images.

[0075] Step 13: Output the bounding box coordinates and detection confidence of the target region. The detection output of IR-YOLOv5-DC3 predicts bounding boxes through multi-scale feature maps. Each grid point generates 3 anchor boxes and outputs a tensor containing the target coordinates (the actual position is calculated by combining the anchor boxes after Sigmoid decoding), confidence, and class probability. After confidence thresholding and NMS non-maximum suppression, the final output is a detection result list in the format [x1, y1, x2, y2, conf, cls], which includes bounding box coordinates, confidence, and class index. The entire process uses Sigmoid decoupling to support multi-label detection, balancing accuracy and real-time performance.

[0076] In a preferred embodiment of this application, the information region character recognition module uses the TrOCR-LSTM-MultiAtt spectral region character recognition method to recognize and extract character information from infrared spectral images, including the following steps:

[0077] Step 21: Extract the infrared spectrum of the catalytic reaction from the published literature and input it into the IR-YOLOv5-DC3 spectrum information target detection module. Collect all the output results of the spectrum information target detection module (which are several image regions containing important information) and establish the dataset used by the information region character recognition module.

[0078] Step 22: Use the visual Transformer encoder to extract features from the preprocessed image, and introduce a BiLSTM module based on multi-head attention feature fusion in the Transformer decoder part. Input the output feature vector of the decoder into the BiLSTM module to further learn sequence features and obtain a new query vector feature Q.

[0079] Step 23: Using the high-dimensional multimodal fusion feature vector output by the Transformer decoder after fusing image features and input text sequence features as the key vector and value vector, input them into the multi-head attention model to learn the multi-head attention weights:

[0080]

[0081] Where Q represents the query vector, K represents the key vector, V represents the value vector, and d k This represents the dimension of the key vector.

[0082] In a preferred embodiment, the infrared spectral data reconstruction module extracts and reconstructs the internal regions of the infrared spectral image through processing and analysis, including the following steps:

[0083] Step 31: Perform color recognition and segmentation processing on the infrared spectral image; use target detection to accurately locate and extract the legend region in the spectral image, enhance the edge range of the legend's color bands using an edge detection algorithm, and determine the RGB value range of each color band of the legend, denoted as RGBrange_legends=[L1,L2,…,L…]. n ], where S1, S2, ..., S n Let R1, G1, B1, and B2 represent the RGB intervals for each color identified by the color clustering algorithm. Each color interval is represented in the format [(R1, G1, B1), (R...]. ′ 1,G ′ 1,B ′ 1)], R, G, and B are the values ​​of the three corresponding color channels (Red, Green, Blue);

[0084] Step 32: Randomly select three in-situ spectral image segments with fixed wavenumber ranges (i.e., the horizontal axis) from the original infrared spectral image obtained from scientific literature. Then, use an adaptive color clustering algorithm to cluster and identify the RGB intervals corresponding to multiple color spectra, denoted as RGBrange_spec = [S1, S2, ..., S...]. n Its content format is consistent with RGBrange_legends;

[0085] Step 33: Calculate the average RGB value for each RGB interval range, denoted as [eL1, eL2, ..., eL... n ] and [eS1,eS2,…,eS n ], where eL i (LR) i LG i ,LB i ), eS i Represents (SR) i ,SG i ,SB i ), that is, the average value of the i-th RGBrange_legends interval and the RGBrange_spec interval, which is a three-dimensional point information;

[0086] Step 34: Construct the Euclidean distance algorithm for eL i With eS i The similarity calculation involves matching the spectral colors with the colors of the legend regions extracted from the infrared spectra of the catalytic reaction extracted from published literature. The calculation formula is as follows:

[0087]

[0088] Where, d i The result of the similarity calculation;

[0089] Step 35: Based on the RGB interval pixel extraction algorithm, the RGB values ​​of each pixel on the infrared spectrum of the catalytic reaction are extracted by traversing the published literature. Each pixel is classified into the pixel RGB interval corresponding to each original legend area according to the interval in which the RGB value is located. All pixels corresponding to each legend RGB interval are extracted and the background is filled with white pixels. That is, all pixels except those that match the RGB interval are adjusted to white to obtain a single spectral curve, thus achieving effective separation of in-situ multi-infrared spectral data.

[0090] The data reconstruction stage of the internal region of the infrared spectral image includes the following steps:

[0091] Step 36: Align the extracted pixel coordinates according to the wavenumber range;

[0092] Step 37: Then normalize the intensity coordinates to the range of 0-100. The normalization formula is:

[0093]

[0094] Among them, y i Represents the original intensity value at coordinate point, y max This represents the highest intensity value of the in-situ spectrum;

[0095] Step 38: To solve the problem of coordinate point redundancy caused by the width of the spectral curve, calculate the arithmetic mean of the ordinates of the same x-coordinate points as the final ordinate under that x-coordinate.

[0096] Step 39: Smooth the infrared spectrum using the Savitzky-Gore filter method and extract the final spectral data;

[0097] The smoothing formula is:

[0098]

[0099] Where m represents the window size, and k represents the index of the element in the convolution kernel. h represents the smoothed data of the i-th point. k This represents the value of the j-th element in the convolution kernel, where n is the value of the j-th element. i+k This represents the data corresponding to the index in the original data.

[0100] Preferably, the catalytic reaction spectrum text information extraction module uses the IR-SciBERT-DCEM-CRF catalytic reaction spectrum text named entity recognition model to achieve semantic understanding and key information extraction of the catalytic reaction spectrum text, including the following steps:

[0101] Step 41: Preprocess the input catalytic reaction spectrum text;

[0102] Step 42: Introduce a dynamic context enhancement module, using SciBERT-base pre-trained weights as initialization, and fine-tune the training with a self-built catalytic reaction text entity recognition dataset to construct the IR-SciBERT-DCEM-CRF catalytic reaction spectrum text entity recognition model. Dynamically fuse the context features of domain keywords through a gated attention mechanism.

[0103] Traditional SciBERT-CRF models learn text features using a Transformer encoder and are pre-trained on a large number of scientific literature texts from different fields. The feature vectors output by the Transformer encoder are input into a Conditional Random Field (CRF) model, ultimately outputting the label category for each word. Based on the PDFminer tool, text is automatically extracted from PDF-formatted scientific literature. Preprocessing steps include text cleaning, sentence segmentation, and word segmentation. Annotation is performed using the BIO annotation method (B represents the beginning of an entity, I represents the interior of an entity, and O represents irrelevant characters), establishing a catalytic reaction text entity recognition dataset. A Dynamic Context Enhancement (DCEM) module is introduced into the traditional SciBERT-CRF framework. Initialized with SciBERT-base pre-trained weights, and fine-tuned using a self-built catalytic reaction text entity recognition dataset, an IR-SciBERT-DCEM-CRF catalytic reaction spectral text entity recognition model is constructed. This model dynamically fuses contextual features from domain keywords through a gated attention mechanism.

[0104] Step 43: Set the plotting words and automatically extract high-frequency domain terms from the self-built catalytic reaction text entity recognition dataset (i.e., high-frequency words contained in a large number of text objects targeted by this system, usually containing words related to catalytic reactions, including reactants, reaction conditions and peak positions).

[0105] Step 44: Calculate the similarity between the word embeddings output by SciBERT and the anchor word embeddings to generate dynamic weights:

[0106]

[0107] Where σ represents the sigmoid activation function, This represents the vector representation of the current token, which is the output vector of SciBERT, k. j denoted as plotted word vector, W represents trainable weight parameters, and b represents bias terms that are automatically adjusted to scalars or vectors;

[0108] Step 45: Concatenate the weighted contextual features with the original SciBERT output, then input them into the CRF layer to finally extract the target entity from the catalytic reaction text sentence from the original scientific paper.

[0109] Preferably, the multimodal information integration and knowledge base construction module uses an information integration method based on a unified data format and storage structure to align and integrate data extracted from images and text, including the following steps:

[0110] Step 51: Determine the standardized and unified format of infrared spectrum data and catalytic reaction information. Key information includes in-situ infrared spectroscopy experimental data, peak wavenumber, peak product, and reaction conditions.

[0111] Step 52: Align the spectral data reconstructed from infrared spectral images from scientific literature with the key information extracted from the text. The main method is to match the peak position and wavenumber information (correspondence between peak position and reaction product).

[0112] Step 53: Store the integrated data in a structured knowledge base.

[0113] This invention also includes a spectroscopic data mining and reconstruction method based on multimodal information extraction, comprising the following steps:

[0114] Step A: Extract the infrared spectrum of the catalytic reaction and related text descriptions from published literature, and preprocess the extracted infrared spectrum of the catalytic reaction and related text descriptions; the preprocessing includes: text preprocessing and image preprocessing; the text preprocessing includes: removing irrelevant information, removing hyphens, sentence segmentation and word segmentation; the image preprocessing includes: size adjustment and grayscale processing.

[0115] Step B: The preprocessed information is used to detect targets in the information region through the spectral information target detection module; the targets in the information region include: legend region, peak wavenumber, peak product, and wavenumber range;

[0116] Step C: Construct a TrOCR-LSTM-MultiAtt spectral region character recognition model using the information region character recognition module to achieve spectral region character recognition;

[0117] Step D: Extract text information from the catalytic reaction spectrum using the catalytic reaction spectrum text information extraction module;

[0118] Step E: Multimodal information integration and knowledge base construction are achieved through the multimodal information integration and knowledge base construction module.

[0119] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spectroscopic data mining and reconstruction system based on multimodal information extraction, characterized in that, include: The system includes a target detection module for spectral information, a character recognition module for information regions, an infrared spectral data reconstruction module, a text information extraction module for catalytic reaction spectra, and a multimodal information integration and knowledge base construction module. The infrared spectra and related texts describing the catalytic reaction are extracted from publicly available literature and preprocessed. The preprocessed information is then used to detect targets in the information region through the spectral information target detection module. A TroOCR-LSTM-MultiAtt spectral region character recognition model is then constructed through the information region character recognition module to recognize characters in the spectral region. Subsequently, the catalytic reaction spectrum text information extraction module extracts the catalytic reaction spectrum text information. Finally, the multimodal information integration and knowledge base construction module integrates the multimodal information and constructs the knowledge base.

2. The spectroscopic data mining and reconstruction system based on multimodal information extraction according to claim 1, characterized in that, The spectral information target detection module uses YOLO target detection technology to identify and segment key information regions in infrared spectral images, including the following steps: Step 11: Perform standardization preprocessing on the infrared spectral images to construct a literature image dataset; Step 12: Input the document image dataset into the IR-YOLOv5-DC3 model based on YOLOv5 improvement; the improved IR-YOLOv5-DC3 model is trained using an infrared spectral annotation dataset, and the improved IR-YOLOv5-DC3 model includes: a feature extraction backbone network, a multi-scale feature fusion network, a detection output network, and a dynamic convolution DC3 module; Step 13: Output the bounding box coordinates and detection confidence of the target region.

3. The spectroscopic data mining and reconstruction system based on multimodal information extraction according to claim 1, characterized in that, The information region character recognition module uses the TrOCR-LSTM-MultiAtt spectral region character recognition method to recognize and extract character information from infrared spectral images, including the following steps: Step 21: Extract the infrared spectrum of the catalytic reaction from the published literature and input it into the IR-YOLOv5-DC3 spectrum information target detection module. Collect all the output results of the spectrum information target detection module and establish the dataset used by the information region character recognition module. Step 22: Use the visual Transformer encoder to extract features from the preprocessed image, and introduce a BiLSTM module based on multi-head attention feature fusion in the Transformer decoder part. Input the output feature vector of the decoder into the BiLSTM module to further learn sequence features and obtain a new query vector feature Q. Step 23: Using the high-dimensional multimodal fusion feature vector output by the Transformer decoder after fusing image features and input text sequence features as the key vector and value vector, input them into the multi-head attention model to learn the multi-head attention weights: Where Q represents the query vector, K represents the key vector, V represents the value vector, and d k This represents the dimension of the key vector.

4. The spectroscopic data mining and reconstruction system based on multimodal information extraction according to claim 1, characterized in that, The infrared spectral data reconstruction module processes and analyzes the infrared spectral image to extract the internal regions of the image and reconstruct the data, including the following steps: Step 31: Perform color recognition and segmentation processing on the infrared spectral image; use target detection to accurately locate and extract the legend region in the spectral image, enhance the edge range of the legend's color bands using an edge detection algorithm, and determine the RGB value range of each color band of the legend, denoted as RGBrange_legends=[L1,L2,…,L…]. n ], where S1, S2, ..., S n These represent the RBG intervals for each color identified by the color clustering algorithm. The format of each color interval is [(R1,G1,B1),(R′1,G′1,B′1)], where R, G, and B are the values ​​of the three corresponding color channels (Red, Green, Blue). Step 32: Randomly select three in-situ spectral image segments with fixed wavenumber ranges (i.e., the horizontal axis) from the original infrared spectral image obtained from scientific literature. Then, use an adaptive color clustering algorithm to cluster and identify the RGB intervals corresponding to multiple color spectra, denoted as RGBrange_spec = [S1, S2, ..., S...]. n Its content format is consistent with RGBrange_legends; Step 33: Calculate the average RGB value for each RGB interval range, denoted as [eL1, eL2, ..., eL... n ] and [eS1,eS2,…,eS n ], where eL i (LR) i LG i ,LB i ), eS i Represents (SR) i ,SG i ,SB i ), that is, the average value of the i-th RGBrange_legends interval and the RGBrange_spec interval, which is a three-dimensional point information; Step 34: Construct the Euclidean distance algorithm for eL i With eS i The similarity calculation involves matching the spectral colors with the colors of the legend regions extracted from the infrared spectra of the catalytic reaction extracted from published literature. The calculation formula is as follows: Where, d i The result of the similarity calculation; Step 35: Based on the RGB interval pixel extraction algorithm, the RGB values ​​of each pixel on the infrared spectrum of the catalytic reaction are extracted by traversing the published literature. Each pixel is classified into the pixel RGB interval corresponding to each original legend area according to the interval in which the RGB value is located. All pixels corresponding to each legend RGB interval are extracted and the background is filled with white pixels. That is, all pixels except those that match the RGB interval are adjusted to white to obtain a single spectral curve, thus achieving effective separation of in-situ multi-infrared spectral data. The data reconstruction stage of the internal region of the infrared spectral image includes the following steps: Step 36: Align the extracted pixel coordinates according to the wavenumber range; Step 37: Then normalize the intensity coordinates to the range of 0-100. The normalization formula is: Among them, y i Represents the original intensity value at coordinate point, y max This represents the highest intensity value of the in-situ spectrum; Step 38: To solve the problem of coordinate point redundancy caused by the width of the spectral curve, calculate the arithmetic mean of the ordinates of the same x-coordinate points as the final ordinate under that x-coordinate. Step 39: Smooth the infrared spectrum using the Savitzky-Gore filter method and extract the final spectral data; The smoothing formula is: Where m represents the window size, and k represents the index of the element in the convolution kernel. h represents the smoothed data of the i-th point. k This represents the value of the j-th element in the convolution kernel, where n is the value of the j-th element. i+k This represents the data corresponding to the index in the original data.

5. The spectroscopic data mining and reconstruction system based on multimodal information extraction according to claim 1, characterized in that, The catalytic reaction spectrum text information extraction module uses the IR-SciBERT-DCEM-CRF catalytic reaction spectrum text named entity recognition model to achieve semantic understanding and key information extraction of the catalytic reaction spectrum text, including the following steps: Step 41: Preprocess the input catalytic reaction spectrum text; Step 42: Introduce a dynamic context enhancement module, using SciBERT-base pre-trained weights as initialization, and fine-tune the training with a self-built catalytic reaction text entity recognition dataset to construct the IR-SciBERT-DCEM-CRF catalytic reaction spectrum text entity recognition model. Dynamically fuse the context features of domain keywords through a gated attention mechanism. Step 43: Set anchor words to automatically extract high-frequency domain terms from the self-built catalytic reaction text entity recognition dataset; Step 44: Calculate the similarity between the word embeddings output by SciBERT and the anchor word embeddings to generate dynamic weights: Where σ represents the sigmoid activation function, This represents the vector representation of the current token, which is the output vector of SciBERT, k. j denoted as plotted word vector, W represents trainable weight parameters, and b represents bias terms that are automatically adjusted to scalars or vectors; Step 45: Concatenate the weighted contextual features with the original SciBERT output, then input them into the CRF layer to finally extract the target entity from the catalytic reaction text sentence from the original scientific paper.

6. The spectroscopic data mining and reconstruction system based on multimodal information extraction according to claim 1, characterized in that, The multimodal information integration and knowledge base construction module, based on a unified data format and storage structure, implements the alignment and integration of data extracted from images and text, including the following steps: Step 51: Determine the standardized and unified format of infrared spectrum data and catalytic reaction information. Key information includes in-situ infrared spectroscopy experimental data, peak wavenumber, peak product, and reaction conditions. Step 52: Align the spectral data reconstructed from infrared spectral images from scientific literature with the key information extracted from the text, mainly by matching whether the peak wavenumber information is equal. Step 53: Store the integrated data in a structured knowledge base.

7. A method for reconstructing spectroscopic data based on multimodal information extraction, using the system described in claims 1-6, characterized in that, Includes the following steps: Step A: Extract the infrared spectra of the catalytic reaction and related text descriptions from published literature, and preprocess the extracted infrared spectra of the catalytic reaction and related text descriptions. Step B: The preprocessed information is used to detect targets in the information region through the spectral information target detection module; the targets in the information region include: legend region, peak wavenumber, peak product, and wavenumber range; Step C: Construct a TrOCR-LSTM-MultiAtt spectral region character recognition model using the information region character recognition module to achieve spectral region character recognition; Step D: Extract text information from the catalytic reaction spectrum using the catalytic reaction spectrum text information extraction module; Step E: Multimodal information integration and knowledge base construction are achieved through the multimodal information integration and knowledge base construction module.

8. The method for spectroscopic data mining and reconstruction based on multimodal information extraction according to claim 7, characterized in that, In step A, the preprocessing includes: text preprocessing and image preprocessing; the text preprocessing includes: removing irrelevant information, removing hyphens, sentence segmentation and word segmentation; the image preprocessing includes: size adjustment and grayscale processing.