Material data processing method and system based on image representation

Abstract

The disclosure provides a material data processing method and system based on image expression, belonging to the field of data processing. The material data processing method comprises: performing semantic analysis on a target image in a document based on a pre-trained model to generate a descriptive explanation text; embedding the descriptive explanation text as placeholder content in the body of the document to replace the target image; performing structured information extraction based on the full text containing the descriptive explanation text; based on embedding matching and relative error calculation of numerical values, respectively outputting precision score and integrity score of the extraction result, and synthesizing into overall score; when the overall score is higher than a preset threshold, write the extraction record to the knowledge base, and record the metadata for tracing. The material data processing method and system provided by the disclosure can avoid the key data loss caused by imperfect image processing in traditional methods.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a material data processing method and system based on image representation. Background Technology

[0002] In materials science research, many key conclusions are presented in graphical form, such as line graphs, coordinate graphs, and composite subplots; many performance indicators are not directly presented in the text but are implicit in the figures. Current practices mainly fall into two categories: one is manual reading and recording by professionals, one article at a time; the other is directly inputting the entire document's text and images into a multimodal / large language model for batch extraction. The former is inefficient, suffers from significant subjective variability, and is difficult to scale; the latter, while possessing some automation capabilities, often suffers from incomplete extraction results, omission of key points, high operational and annotation costs, and difficulty in establishing stable calibration standards.

[0003] While recent research has explored text and data conversion using large language models, it has focused primarily on the main text and tables, and remains insufficient in the systematic and one-time extraction of structured elements embedded in images. Especially in PCT, TPD, magnification / cyclic performance, and stress-strain graphs, key information often requires joint geometric and semantic analysis at the curve level. Relying solely on end-to-end multimodal inference is susceptible to model illusions, context truncation, and style domain shifts, resulting in unverifiable results. Summary of the Invention

[0004] The purpose of this disclosure is to provide a material data processing method and system based on image representation, which can extract key information from the image representation in a document.

[0005] To achieve the above objectives, the technical solution provided in this disclosure is as follows:

[0006] In a first aspect, this disclosure provides a material data processing method based on image representation, which includes:

[0007] Semantic analysis is performed on the target image in the document based on a pre-trained model to generate descriptive explanatory text. The descriptive explanatory text is then embedded as placeholder content into the main text of the document to replace the target image. Based on the full text containing the descriptive explanatory text, structured information extraction is performed. Based on embedding matching and numerical relative error calculation, the accuracy score and completeness score of the extraction results are output respectively and synthesized into an overall score. When the overall score is higher than a preset threshold, the extraction record is written into the knowledge base, and metadata for traceability is recorded.

[0008] In one or more embodiments, semantic analysis is performed on target images in a document to generate descriptive explanatory text, including: parsing the document to obtain text, figure titles, and images in the document, and filtering target images containing material characterization information based on figure titles and / or context; performing semantic classification and layout understanding on the target images to identify sub-figures, legends, curve mapping relationships, and coordinate systems; obtaining key points and segment information of curves from the target images, and generating descriptive explanatory text for the target images according to a preset template.

[0009] In one or more embodiments, filtering target images containing material characterization information based on image titles and / or context includes: keyword matching and / or regular expression matching based on image titles and context; binary classification of image titles and context based on a classifier to determine whether they contain material characterization information; and filtering target images containing material characterization information based on image number references, the correspondence between image titles and images.

[0010] In one or more embodiments, the target image is semantically classified and its layout understood to identify sub-images, legends, curve mapping relationships, and coordinate systems. This includes determining sub-images in the target image based on the title and / or context and establishing a sub-image index; identifying the number of curves and legend items within each sub-image and determining the correspondence between the legend and the curves; determining whether the coordinate system is a single y-axis or a dual y-axis, and whether the coordinate axis scale is linear or logarithmic, and extracting the units and numerical ranges of each coordinate axis; extracting test condition labels related to the curves, and generating an image semantic description at the sub-image granularity that includes sub-image indexes, curve identifiers, legend mappings, coordinate system information, coordinate unit ranges, and test condition labels, for subsequent key point and segment information acquisition and descriptive explanation text generation.

[0011] In one or more embodiments, the key points and segment information include at least one of plateau region, inflection point, and peak position; wherein, the plateau region is determined by a slope threshold and minimum length constraint obtained by linear fitting of a sliding window; the inflection point is determined by curvature abrupt change and / or first derivative rate of change threshold; and the peak position is determined by local extremum detection combined with minimum peak width and minimum peak height constraints.

[0012] In one or more embodiments, generating descriptive explanatory text for the target image based on a preset template includes: using the preset template to transcribe the obtained curve, coordinate system semantics, and annotation information into structured natural language fragments, wherein the natural language fragments include field key names, source image indexes, and page location information.

[0013] In one or more embodiments, the output fields of the structured information extraction include at least one of the following: material formulation, material type, elemental composition and content, structural or morphological characteristics, preparation conditions, testing conditions, and performance indicators.

[0014] In one or more embodiments, based on embedding matching and numerical relative error calculation, the accuracy score and completeness score of the extraction results are output respectively and synthesized into an overall score, including: using text embedding matching to pair the output fields of the structured information extraction with the fields of the reference set; calculating the relative error of the matched numerical fields to obtain the accuracy score, calculating the completeness score for the missing / redundant fields, and synthesizing the accuracy score and completeness score into an overall score according to a preset weight.

[0015] In one or more embodiments, the material data processing method further includes: when the overall score is lower than a preset threshold, manually verifying the extracted records; updating the extraction template and model parameters based on the verification results, and marking the extracted records as revised for model retraining or fine-tuning.

[0016] Secondly, this disclosure provides a material data processing system based on image representation, comprising:

[0017] The document comprises a parsing module for performing semantic analysis on target images in a document based on a pre-trained model, generating descriptive explanatory text; an embedding module for embedding the descriptive explanatory text as placeholder content into the original text of the document to replace the target image; an extraction module for performing structured information extraction based on the full text containing the descriptive explanatory text; an evaluation module for outputting an accuracy score and a completeness score of the extraction results based on embedding matching and numerical relative error calculation, and combining them into an overall score; and a writing module for writing the extraction record into a knowledge base and recording metadata for traceability when the overall score is higher than a preset threshold.

[0018] The material data processing method and system based on image representation disclosed herein transforms image information that is difficult to directly parse into calculable textual content by performing semantic analysis on the target image in the document and generating descriptive explanatory text, and then embedding the text into the original text to replace the image. Then, structured information extraction is performed based on the complete text and explanatory context, which can not only capture core elements such as material formulation and performance indicators, but also avoid the loss of key data caused by imperfect image processing in traditional methods. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a material data processing method in one embodiment of the present disclosure;

[0021] Figure 2 This is a schematic diagram illustrating the generation of descriptive explanatory text in one embodiment of this disclosure;

[0022] Figure 3 This is a score comparison chart of the material data processing method in one embodiment of the present disclosure compared to the conventional method;

[0023] Figure 4 This is a schematic diagram of a material data processing system in one embodiment of the present disclosure;

[0024] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions in this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0026] In materials science research, research conclusions are often presented visually through various images, such as performance curves, coordinate graphs, or composite subgraphs. However, existing technologies have significant shortcomings in information acquisition: manual interpretation is inefficient, relies on personal experience, and is prone to omissions; while relying directly on multimodal models to process entire documents in batches can easily lead to incomplete extraction results, missing key points, field mismatches, and difficulties in quantitative evaluation. Especially when multiple curves, multiple coordinate systems, or different experimental conditions exist simultaneously in a single image, existing methods generally cannot achieve comprehensive, accurate, and traceable extraction, resulting in low efficiency in the utilization of research data.

[0027] Based on an in-depth analysis of these issues, this disclosure proposes a novel approach. Its core principle is to make explicit the structured information implicit in the image, so that subsequent information extraction no longer relies on complex and uncontrollable end-to-end processing, but instead uses a unified parsing and evaluation based on intermediate representations. Specifically, this method first understands the target image through semantic analysis, transing curves, coordinate systems, conditional labels, and key points or segment information into descriptive explanatory text with clear logical relationships, and embedding this explanatory text into the original text as placeholder content. This approach integrates image semantics with textual context, allowing image information to naturally participate in the full-text information extraction process.

[0028] Building upon this foundation, the entire document, including text and placeholder explanations, undergoes structured extraction. The units of measurement for numerical fields are standardized and unified to ensure direct comparison of extraction results across different documents. Simultaneously, a quality assessment process is introduced, utilizing semantic matching and numerical bias calculation to output quantifiable indicators across both accuracy and completeness dimensions, establishing a unified evaluation standard. Finally, results meeting the requirements are written into the knowledge base, recording metadata such as document identifiers, figure numbers, model versions, and timestamps, thus establishing a traceable research data mechanism.

[0029] Please refer to Figure 1 The diagram shown is a flowchart of a material data processing method based on image representation according to an embodiment of this disclosure. The material data processing method specifically includes the following steps:

[0030] S101: Based on a pre-trained model, perform semantic analysis on the target images in the document to generate descriptive explanatory text.

[0031] The core of step S101 lies in transcribing the structured information in the target image that is useful but difficult for a text extractor to understand directly into a readable, searchable, and auditable explanatory text. To this end, the system uses a pre-trained model as its backbone to perform semantic understanding of the target image, identifying the type of scientific graph (e.g., line graph, coordinate graph, or composite subgraph), the correspondence between each curve and its legend, whether the coordinate system is single or double y-axis, whether the axis scale is linear or logarithmic, the units and numerical ranges of each axis, and the experimental conditions associated with the curves (temperature, pressure, rate, potential, etc.). In this process, a visual-language model handles the overall semantic judgment, while a layout model and OCR handle the reading of titles, scales, and units. Lightweight geometric algorithms can also be used to assist in locating the curve's direction and key inflection points. After semantic fusion, the above multi-source results are organized into descriptive explanatory text, i.e., an explanation written in natural language.

[0032] In one exemplary embodiment, semantic analysis is performed on target images in a document to generate descriptive explanatory text. Specifically, this includes: parsing the document to obtain the text, figure titles, and images in the document, and filtering target images containing material characterization information based on the figure titles and / or context; performing semantic classification and layout understanding on the target images to identify sub-figures, legends, curve mapping relationships, and coordinate systems; obtaining key points and segment information of curves from the target images, and generating descriptive explanatory text for the target images according to a preset template.

[0033] First, select the graphs that are truly relevant to the material representation from the document, then gradually analyze their semantic structure, and finally transform the curve elements, coordinate system, condition labels, and key point values ​​into descriptive text fragments, thus providing a foundation for subsequent structured extraction.

[0034] In practice, the first step is to parse the document to obtain text, figure titles, and images, and then filter out target images based on the figure titles and / or context. Research papers typically contain a large number of images, but not all figures are relevant to material characterization; therefore, keyword matching, regular expression retrieval, or lightweight classifiers are needed for filtering. For example, when terms such as "PCT (pressure-composition-temperature)," "TPD," "desorption," or "isotherm" appear in the figure title or adjacent text, it can be preliminarily determined that the figure may contain key curve information for hydrogen storage materials. This avoids irrelevant images from entering subsequent processes, reduces computational overhead, and improves the relevance of the extraction results.

[0035] After acquiring the target image, semantic classification and layout understanding are required. The purpose of this step is to understand the image's basic structure and organization. For example, an image may contain multiple subimages, distinguished by letter labels (a), (b), and (c). Subimage boundaries need to be identified and indexed for accurate referencing in subsequent results. Simultaneously, the legend is a crucial clue for understanding curve attribution; therefore, the content of the legend items must be identified and a correspondence established with the number of curves. More complex cases involve images with dual y-axes or logarithmic coordinates, which must be correctly parsed; otherwise, the curve data cannot be accurately reconstructed. For example, in a dual y-axis image, the left axis might represent temperature, and the right axis pressure; mismatches will distort subsequently extracted values. Through this process, the overall semantic skeleton of the image is established.

[0036] Subsequently, it is necessary to extract key points and segment information of the curve from the target image. Scientific images often carry characteristic values ​​from experiments, such as plateau intervals, inflection point positions, and peak sizes, which are often the direct basis for research conclusions. For example, in the PCT curve, the plateau interval determines the hydrogen storage capacity of the alloy; in the TPD curve, the peak temperature reflects the energy barrier of the desorption reaction. By detecting local extrema, abrupt slope changes, or approximately horizontal segments of the curve, this key point and segment information can be captured, and experimental condition labels (such as 300K, 1 bar) can be added to make the semantics of the results more complete.

[0037] Finally, a descriptive explanatory text for the target image is generated based on a preset template. The purpose of this preset template is to express all the previously parsed elements in a standardized text format, making it both consistent with natural language reading habits and possessing clear structural identifiers (such as...). Figure 2 (As shown). Such a description is not only directly understandable to humans, but can also be identified as specific fields by subsequent text extractors and further converted into structured data formats such as JSON.

[0038] In one exemplary embodiment, the method of filtering target images containing material characterization information based on image titles and / or context includes: keyword matching and / or regular expression matching based on image titles and context; binary classification of image titles and context based on a classifier to determine whether they contain material characterization information; and filtering target images containing material characterization information based on image number references, the correspondence between image titles and images.

[0039] A single research paper often contains a large number of images, including experimental characterization curves, performance graphs, as well as device photographs, schematic diagrams, or flowcharts. Without targeted image selection, subsequent image semantic analysis, keypoint detection, and data extraction can become redundant or even erroneous.

[0040] In practical implementation, the first step is to use keyword matching and regular expression matching based on the figure title and context. The figure title and adjacent paragraphs in a research paper often directly indicate the type of information the figure displays. For example, keywords such as "PCT curve," "TPD spectrum," "isotherm," and "electrochemical performance" can directly reveal that the figure is a material performance characterization curve. By establishing a glossary of commonly used materials science terms and regular expression patterns, potential target images can be quickly identified. For instance, when the figure title is detected as "Hydrogen desorption isotherms of alloy A at various temperatures," the keywords "isotherm" and "desorption" trigger a match, and the figure is then marked as a candidate target image.

[0041] Secondly, a classifier is introduced to perform binary classification of the image title and context to further determine whether the image contains material characterization information. While keyword matching is efficient, it contains noise; for example, a flowchart might contain words like "temperature," but this doesn't necessarily mean it's a characterization curve. Using lightweight machine learning or deep learning classifiers, binary judgment can be performed based on the overall semantic context. For instance, a small text classification model can be trained to classify an input like "Figure 2. Pressure-composition isotherms of Mg-based alloys" as a characterization image, while an input like "Figure 3. Flowchart of the synthesis process" would be classified as a non-characterization image. This approach effectively improves the accuracy of the selection and reduces false positives.

[0042] The third mechanism verifies the validity of a figure based on the correspondence between figure citations, figure titles, and images. In research papers, citations such as "as shown in Fig. 4(b)" often appear in the text, and corresponding figures are also listed below the figures, such as "Figure 4(b)...". By establishing a three-way correspondence between text citations, figure titles, and image files, the validity of the target image can be further confirmed. For example, if the text states "the plateau pressure can be observed in Fig. 5," and... Figure 5 If a figure's title includes "Pressure-composition isotherms," ​​then it can be confirmed that the figure is directly related to key performance indicators and should be considered a priority target image. This cross-validation mechanism ensures that the selection results not only rely on a single source but also take into account the context of the text, thereby enhancing robustness.

[0043] To illustrate with a concrete example: In a research paper on hydrogen storage alloys, the first step is to analyze the figure title, "Figure 2. PCT curves of alloy X at 300, 350 and 400 K." The keywords "PCT" and "curves" directly match the candidate criteria. The classifier then further determines that the semantics of this section highly match the material performance characterization, confirming that the figure should be included in the target set. The text itself states, "as shown in Fig. 2, the plateau pressure shifts with temperature," linking the figure title, context, and textual references. Ultimately, this confirms that the figure is a material characterization image and marks it for further processing.

[0044] The purpose of this screening step is to improve the relevance and efficiency of subsequent processing. Image semantic analysis and key point extraction are only valuable when truly relevant representation images enter the subsequent process, avoiding wasting computational resources on irrelevant images. At the same time, it can improve the overall system accuracy and reduce misjudgments caused by image noise or irrelevant images.

[0045] In one exemplary embodiment, semantic classification and layout understanding are performed on the target image to identify sub-images, legends, curve mapping relationships, and coordinate systems. Specifically, this includes: determining sub-images in the target image based on the title and / or context and establishing a sub-image index; identifying the number of curves and legend items within each sub-image and determining the correspondence between the legend and the curves; determining whether the coordinate system is a single y-axis or a dual y-axis, and whether the coordinate axis scale is linear or logarithmic, and extracting the units and numerical ranges of each coordinate axis; extracting test condition labels related to the curves, and generating an image semantic description at the sub-image granularity that includes the sub-image index, curve identifier, legend mapping, coordinate system information, coordinate unit range, and test condition labels, for subsequent key point and segment information acquisition and descriptive explanation text generation.

[0046] In practical implementation, the first step is to determine the sub-image structure within the target image using the image title and context, and then establish a sub-image index. Research papers often place multiple sets of experimental results under the same image, distinguishing them using labels such as (a), (b), and (c). During parsing, these sub-image regions need to be identified and assigned index numbers so that subsequent results can be accurately traced. For example, Figure 3 It may contain 3 subplots, each showing the adsorption curves under different temperature conditions. Without a subplot index, it is impossible to accurately label which experimental condition each curve belongs to.

[0047] After identifying the subplots, the next step is to identify the number of curves and the legend entries, and establish the correspondence between them. Scientific images typically contain multiple curves, each representing an experimental variable, and the colors, line types, or markers in the legend are crucial for identification. It is necessary to distinguish the curve colors or symbols and map them one-to-one with the legend entries to clearly identify each curve. For example, in a scaling factor plot, curve A corresponds to condition 1C, and curve B corresponds to condition 2C. If this mapping is incorrect, subsequent key data points will be misattributed, thus affecting the conclusions.

[0048] Next, it's necessary to determine the coordinate system used in the image, which is a prerequisite for understanding the meaning of the curve. Some scientific graphs use a single y-axis, while others use two y-axes; some are linear coordinates, while others are logarithmic coordinates. This information is analyzed, and the units and numerical ranges of each coordinate axis are extracted. For example, a hydrogen adsorption isotherm graph might use logarithmic pressure as the horizontal axis and hydrogen adsorption capacity (wt.%) as the vertical axis. If it's incorrectly identified as a linear coordinate system, it will affect the judgment of plateau intervals and inflection points. Furthermore, the units (e.g., bar, Pa, wt.%) and numerical ranges (e.g., 10⁻¹⁰–10⁻¹⁰) of the coordinate axes are also important. 3 The bar needs to be accurately extracted to ensure the effectiveness of numerical normalization and comparison.

[0049] Besides curves and coordinate systems, test condition labels are also an important component of image semantics. These labels often appear in image titles, legends, or annotations, such as "300K," "0.1M HCl," and "Scan rate 50 mV / s." Binding these condition labels to curves ensures that the key points extracted later are not just abstract numerical values, but also carry a clear experimental condition context.

[0050] Finally, an image semantic description is generated at the subgraph level. This description includes not only the subgraph index but also curve labels, legend mappings, coordinate system information, coordinate unit ranges, and test condition labels. Such a semantic description can be directly understood by humans and can also be used by downstream modules for keypoint extraction and descriptive explanatory text generation.

[0051] In one exemplary embodiment, the key points and segment information include at least one of a plateau region, an inflection point, and a peak position; wherein, the plateau region is determined by a slope threshold and a minimum length constraint obtained through linear fitting of a sliding window; the inflection point is determined by a curvature abrupt change and / or a first derivative rate of change threshold; and the peak position is determined by local extremum detection combined with minimum peak width and minimum peak height constraints.

[0052] A plateau region refers to a nearly horizontal interval in a curve. For example, in a PCT curve, the plateau region reflects the hydrogen absorption and desorption process of an alloy under constant pressure. To identify such intervals, a sliding window linear fitting method can be used to linearly approximate the curve within different window ranges. If the slope of the fitted curve is lower than a preset threshold, and the length of the interval exceeds the minimum length constraint, it can be identified as a plateau region. For instance, if an isothermal adsorption curve is nearly flat in the 1-2 bar range, with a slope close to zero, and the interval span is significantly larger than the noise window, this portion can be labeled as a plateau region.

[0053] Inflection points often appear where the trend of a curve changes significantly, such as the beginning or end of an adsorption / desorption curve. Mathematically, these points often correspond to abrupt changes in curvature or where the rate of change of the first derivative exceeds a certain threshold. The locations of abrupt changes from positive to negative or from negative to positive can be identified by calculating the curvature of the curve at different points, or by comparing the differences in slopes between adjacent intervals.

[0054] Peak positions typically appear in experimental curves such as TPD (temperature programmed desorption), DSC (differential scanning calorimetry), or electrochemical impedance spectroscopy, representing the maximum response of a process. Peak positions can be identified using local extremum detection methods, combined with constraints on minimum peak width and minimum peak height to avoid misidentifying noise or minor perturbations as peak positions. By introducing dual constraints on peak width and peak height, spurious small peaks caused by experimental noise can be filtered out, thus ensuring the stability of peak position extraction.

[0055] In one exemplary embodiment, generating descriptive explanatory text for the target image based on a preset template specifically includes: using the preset template, transcribing the obtained curve, coordinate system semantics, and annotation information into structured natural language fragments, wherein the natural language fragments include field key names, source image indexes, and page location information.

[0056] In terms of implementation, the first step is to design a pre-defined template, similar to a formatted fill-in-the-blank sentence, reserving fixed positions for different information elements. For example, the template could include fields such as "figure index," "curve label," "axis label and unit," "coordinate system type," "keypoint value," "experimental condition label," and "page location information." Having already identified and parsed these elements in the previous steps, the parsing results are simply filled into the corresponding positions in the template to generate a complete and standardized natural language description. This semi-structured text ensures both the accuracy of the field content and the coherence of natural language.

[0057] By using fixed field keys, each descriptive fragment can be traced back to its specific source image index and page location information, ensuring that the original evidence can be found during subsequent audits, reviews, or knowledge base searches. This is particularly important in scientific research data processing scenarios, because experimental results in materials science often need to be repeatedly verified and compared horizontally. Without this locating mechanism, the reliability and reusability of the data would be greatly reduced.

[0058] By using a standardized template, the explanatory texts extracted from different papers maintain a relatively consistent structure, avoiding information gaps or inconsistent expressions caused by free generation. Each explanatory text includes source information such as figure numbers and page numbers, along with field keys, enabling rapid location of the original image and achieving two-way verification of data and source. Furthermore, since the explanatory texts are already expressed in blocks within the template, the extraction module can directly locate and parse the numerical fields and units, reducing error rates and processing costs.

[0059] S102: The descriptive explanatory text is embedded as placeholder content into the body of the document to replace the target image.

[0060] By placing the descriptive explanatory text generated in the previous step back into the document context, key information that originally existed only in the image is integrated with the main text in a readable, searchable, and auditable form. At the original (or adjacent) location of the target image, a structured natural language fragment replaces the image's role from the processing perspective, allowing subsequent extraction and evaluation to be conducted using the same set of textual information, eliminating the need for repeated image recognition.

[0061] In implementation, reversible placeholder embedding methods can be selected depending on the medium. For PDF documents, a placeholder segment can be inserted by covering or adjacent to the page bounding box of the target figure with a text layer, keeping the original figure as a hidden / lower-level object and adding a unique identifier (such as FigID, page number, bbox) to achieve logical replacement and physical preservation. For HTML / Markdown documents, the figure or image link can be replaced with a detail expand block or paragraph text, retaining the original figure's revisitable link. For DOCX / LaTeX documents, a placeholder segment can be inserted at the figure title-figure body using content controls / packages, and an anchor point can be added at "See Figure x" in the text. Regardless of the method chosen, the placeholder text contains field key names and source location information (such as figure index, page position, confidence level, etc.).

[0062] S103: Perform structured information extraction based on the full text containing the descriptive explanation text.

[0063] Step S10 involves extracting elements scattered throughout the main text, figure titles, and image explanations into structured fields (such as material formulation, type, elements and content, preparation / testing conditions, performance indicators, and uncertainties) within the merged text context and based on a defined domain pattern. Unlike traditional methods that directly extract data from raw images or scattered text, S103 leverages the graph semantics of the previous step to incorporate the most easily mismatched curve-legend-coordinate-conditional relationships, along with figure numbers and page positions, into a single parsable text stream. This allows downstream processes to focus solely on the standard text-to-structure process, eliminating the need for multimodal inference and reducing extraction difficulty and errors.

[0064] In terms of implementation, this can be achieved by loading field definitions and value constraints customized for materials literature (such as Formula, Type, Elements, Content, Structure / Morphology, Synthesis / Processing, TestCondition [temperature / pressure / rate / potential / medium / flow rate], Performance [capacity / platform pressure / hysteresis / peak position / initial temperature, etc.], Uncertainty and source anchor points, etc.), and setting acceptable expression modes and value ranges for each field.

[0065] Subsequently, text retrieval and candidate segment localization (using BM25 / dense vector retrieval) are used to coarsely recall the most relevant sentences and segments to a certain field in the full text. On these candidates, a domain-fine-tuned extraction model or suggestive LLM is used to extract field name-value pairs, retaining adjacent units, conditional modifiers, and source citations (e.g., "Fig.3(b), p.12"). For numerical segments, their units and possible coordinate scale descriptions are extracted together; unit normalization can be completed in subsequent steps. For symbolic or enumerated segments (such as material type, carrier gas name, electrolyte formulation), further normalization is achieved through a thesaurus and alias table.

[0066] In one exemplary embodiment, the output fields of the structured information extraction include at least one of the following: material formulation, material type, elemental composition and content, structural or morphological characteristics, preparation conditions, testing conditions, and performance indicators.

[0067] In terms of implementation, synonym and alias mappings can be established based on domain vocabulary and expression variants to reduce the interference of natural language expression differences on recognition. Material formulations may appear as chemical formulas (such as LiFePO4, MgO) or as trade names or alloy codes (such as Alloy A, NCM811). During extraction, the original expression, standardized formulation, and resolvable formula (element-valence-stoichiometry) can be retained simultaneously, facilitating subsequent elemental closure verification and family clustering.

[0068] Material type determination is commonly found in titles or first paragraphs. These fields are often semantically enumerable. During extraction, a dual-channel identification approach combining a pattern dictionary and dense vector similarity can be used to reduce ambiguity. When extracting elemental composition and content, attention must be paid to the consistency of units and measurement standards. The same article may simultaneously present at.%, wt.%, and mol ratios. The original values ​​and units can be captured together and converted to a uniform scale (e.g., wt.%) during the subsequent standardization stage, while preserving the conversion chain and uncertainty.

[0069] Structural or morphological characteristics are typical fields with dense evidence but loose descriptions. Common expressions in the literature include "nanofacial sheets," "hollow spheres," "porous networks," and "particle size 200–300 nm." When extracting these terms, it is important to grasp the keywords and also to link quantitative descriptions with characterization methods, such as "SEM: average particle size 250 nm."

[0070] Preparation conditions and testing conditions are the most easily confused parts between upstream and downstream processes. This solution adopts a process-characterization dichotomy when extracting data: the former focuses on temperature, time, atmosphere, precursors, annealing / quenching, etc.; the latter focuses on temperature, pressure, electrolyte formulation, scan rate, carrier gas flow rate, pH, etc. during testing, and prioritizes reading the condition labels bound to the curve from the descriptive explanation text to ensure that the curve-level indicators correspond one-to-one with their context.

[0071] Performance metrics directly relate to key points and segments on the image side, such as PCT plateau pressure and hysteresis width, TPD peak position and onset temperature, electrochemical specific capacity / rate / cycle retention, mechanical yield strength / Young's modulus, and impedance spectroscopy R_ct. When these metrics are extracted, the numerical value, unit, and source anchor (figure number, page number, bbox) are captured simultaneously to support subsequent quality assessment and traceability.

[0072] S104: Based on embedding matching and numerical relative error calculation, output the accuracy score and completeness score of the extraction results respectively, and synthesize them into an overall score.

[0073] First, semantic embedding is used to pair the extracted fields with the fields in the reference set (manually annotated or authoritative standards) one by one. Then, the error of numerical fields is quantified into relative error. After obtaining evidence at the field level of whether the alignment is correct, how large the numerical deviation is, and which fields are missing or redundant, the results are summarized into accuracy score and completeness score, and finally synthesized into an overall score according to preset weights.

[0074] The reason for using embedding matching instead of relying solely on keywords is that literature often uses multiple ways of writing the same concept, and semantic similarity in the embedding space can more robustly align synonyms and near-synonyms; while relative error provides a comparable numerical deviation measure across units and scales, which can be measured on the same scale.

[0075] In practice, matching pairs are first constructed for candidate fields. Each extracted entry can contain information such as field name, field value, unit, source anchor (e.g., Fig. 3(b), p12), and confidence level. The control set entries also exist with the same structure. The field name and its context phrase are jointly encoded into a vector, and the cosine similarity with the same type of field in the control set is calculated. Combined with the consistency of the source anchor (whether the figure number / page number is from the same source) and unit compatibility, a comprehensive matching score is formed.

[0076] For successfully matched numerical fields, unit normalization is performed first, followed by calculation of relative error. For interval or set values ​​(e.g., platform 1.5~2.0 bar, particle size 200~300 nm), interval overlap rate or weighted error at endpoints / midpoints can be used for scoring. For text fields, embedding similarity or controlled vocabulary consistency mapping is used for scoring. Field-level scores are then aggregated by importance to form an accuracy score (measuring whether the alignment is correct). Simultaneously, the percentage of fields successfully selected and passing the threshold is calculated as a completeness score (measuring how many were selected). Both scores are then combined according to business-defined weights to form an overall score, reflecting the combined level of accuracy and completeness.

[0077] S105: When the overall score is higher than a preset threshold, the extracted records are written into the knowledge base, and metadata for traceability is recorded.

[0078] Once the overall score exceeds a preset threshold, it is written into the knowledge base in a standardized, versioned, and traceable manner; at the same time, the context required to reproduce the result (including source, location, model and hints, time and status, etc.) is synchronously archived in the form of metadata.

[0079] In terms of implementation, the knowledge base can be either a columnar data warehouse oriented towards query analysis (facilitating statistics and modeling) or a graph database oriented towards entities and relationships (facilitating cross-paper and cross-image correlation retrieval). Each extracted record is written as a serializable object: the main body consists of structured fields (such as material formula, composition and content, test conditions, performance indicators and units), accompanied by a set of essential metadata keys. Metadata typically includes at least the document identifier (such as DOI), page number, figure number and page bounding box, descriptive explanatory text snapshot, extraction JSON snapshot, standardized unit caliber, hash summaries of text and images, model and hint version number, parameter configuration signature, processing timestamp, overall score and sub-score, review status and responsible person, and serial numbers of upstream and downstream tasks. To ensure replayability, the original image and the preprocessed intermediate products (such as subgraph segmentation results) can be linked using object storage chaining.

[0080] In one exemplary embodiment, the material data processing method further includes: when the overall score is lower than a preset threshold, manually verifying the extracted records; updating the extraction template and model parameters based on the verification results, and marking the extracted records as revised for model retraining or fine-tuning.

[0081] When the overall score falls below a preset threshold (the threshold can be set in conjunction with business tolerance and different thresholds can be set for different fields), the result will not be discarded directly. Instead, the record along with its evidence chain will be pushed to the verification workbench. The workbench will display the full text of the sample, the figure number and its position on the page, placeholder descriptive explanatory text, a structured JSON draft, and the accuracy / completeness.

[0082] At the implementation level, low-scoring samples are automatically categorized: those with obvious deviations due to unit normalization or formatting issues are prioritized for the rule priority queue, where they can be quickly corrected using applicable templates and rules; samples involving semantic mismatches or visual judgment failures are prioritized for the model priority queue, requiring accurate field values ​​and binding relationships to be provided by a human. Human-reviewed changes and original values ​​can be displayed side-by-side, recording the differences before / after modification, the modifier, timestamp, and notes; all changes are synchronized to a versioned knowledge base, marked as "revised," and a change event is generated for subsequent auditing. On the model side, revised samples are collected as training pairs and incorporated into the hard example set for the next round of training.

[0083] Figure 3 This is a score comparison chart showing the method of this disclosure compared to traditional direct extraction methods. Figure 3 It can be seen that the image-based material data processing method provided in this disclosure is able to extract key information from images better than traditional methods.

[0084] In summary, the image-based material data processing method provided in this disclosure transforms image information that is difficult to directly parse into calculable textual content by performing semantic analysis on the target image in the document and generating descriptive explanatory text, and then embedding the text into the original text to replace the image. Then, structured information extraction is performed based on the complete text and explanatory context, which can not only capture core elements such as material formulation and performance indicators, but also avoid the loss of key data caused by imperfect image processing in traditional methods.

[0085] Please refer to Figure 4 As shown, based on the same inventive concept as the aforementioned image-based material data processing method, this disclosure provides a material data processing system 400, which includes: a parsing module 401, an embedding module 402, an extraction module 403, an evaluation module 404, and a writing module 405.

[0086] The parsing module 401 performs semantic analysis on the target image in the document based on a pre-trained model to generate descriptive explanatory text. The embedding module 402 embeds the descriptive explanatory text as placeholder content into the original text of the document to replace the target image. The extraction module 403 performs structured information extraction based on the full text containing the descriptive explanatory text. The evaluation module 404 outputs an accuracy score and a completeness score for the extraction results based on embedding matching and numerical relative error calculation, and combines them into an overall score. The writing module 405 writes the extraction record to a knowledge base and records metadata for traceability when the overall score is higher than a preset threshold.

[0087] Please refer to Figure 5 As shown, embodiments of this disclosure also provide an electronic device 500, which includes at least one processor 501, a memory 502 (e.g., non-volatile memory), a main memory 503, and a communication interface 504, and the at least one processor 501, memory 502, main memory 503, and communication interface 504 are connected together via an internal bus 505. The at least one processor 501 is configured to invoke at least one program instruction stored or encoded in the memory 502 to cause the at least one processor 501 to perform various operations and functions of the image-based material data processing method described in the various embodiments of this specification.

[0088] In the embodiments of this specification, electronic device 500 may include, but is not limited to: personal computer, server computer, workstation, desktop computer, laptop computer, notebook computer, mobile electronic device, smartphone, tablet computer, cellular phone, personal digital assistant (PDA), handheld device, messaging device, wearable electronic device, consumer electronic device, etc.

[0089] This disclosure also provides a computer-readable medium carrying computer-executable instructions, which, when executed by a processor, can be used to implement various operations and functions of the image-based material data processing methods described in the various embodiments of this specification.

[0090] The computer-readable medium in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0091] In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0092] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, systems, and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] It will be apparent to those skilled in the art that this disclosure is not limited to the details of the exemplary embodiments described above, and that this disclosure can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of this disclosure is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this disclosure. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0095] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for processing material data based on image representation, characterized in that, include: Parse the document to obtain the text, figure titles and images in the document, and filter target images containing material characterization information based on the figure titles and / or context; Based on a pre-trained model, semantic classification and layout understanding are performed on the target image to identify sub-images, legends, curve mapping relationships, and coordinate systems. Key points and segment information of curves are obtained from the target image, and descriptive explanatory text of the target image is generated according to a preset template. The preset template is used to transcribe the obtained curve, coordinate system semantics, and annotation information into structured natural language fragments, which include field key names, source image indexes, and page location information. The descriptive explanatory text is embedded as placeholder content into the body of the document to replace the target image; Based on the full text containing the descriptive explanation text, perform structured information extraction; Text embedding matching is used to pair the output fields of the structured information extraction with the fields of the reference set. The relative error of the matched numerical fields is calculated to obtain the accuracy score, and the completeness score is calculated for the missing / redundant fields. The accuracy score and the completeness score are combined into an overall score according to a preset weight. When the overall score is higher than a preset threshold, the extracted records are written into the knowledge base, and metadata for traceability is recorded.

2. The material data processing method according to claim 1, characterized in that, Target images containing material characterization information are filtered based on the image title and / or context, including: Keyword matching and / or regular expression matching based on graph title and context; The image title and its context are classified using a classifier to determine whether they contain material characterization information. Based on the correspondence between figure number references, figure titles and images, target images containing material characterization information are selected.

3. The material data processing method according to claim 1, characterized in that, The target image is subjected to semantic classification and layout understanding to identify sub-images, legends, curve mapping relationships, and coordinate systems, including: Identify sub-graphs in the target image based on the title and / or context, and build a sub-graph index; Identify the number of curves and legend items within each subplot, and determine the correspondence between the legend and the curves; Determine whether the coordinate system is a single y-axis or a double y-axis, and whether the coordinate axis scale is linear or logarithmic, and extract the unit and numerical range of each coordinate axis; Extract test condition labels related to the curve, and generate image semantic descriptions at the sub-graph level, including sub-graph index, curve identifier, legend mapping, coordinate system information, coordinate unit range, and test condition labels, for subsequent key point and segment information acquisition and descriptive explanation text generation.

4. The material data processing method according to claim 1, characterized in that, The key points and segment information include at least one of plateau region, inflection point, and peak position; wherein, the plateau region is determined by a slope threshold and minimum length constraint obtained by linear fitting of a sliding window; the inflection point is determined by curvature abrupt change and / or first derivative rate of change threshold; and the peak position is determined by local extremum detection combined with minimum peak width and minimum peak height constraints.

5. The material data processing method according to claim 1, characterized in that, The output fields of the structured information extraction include at least one of the following: material formulation, material type, elemental composition and content, structural or morphological characteristics, preparation conditions, testing conditions, and performance indicators.

6. The material data processing method according to claim 1, characterized in that, The material data processing method further includes: When the overall score is lower than the preset threshold, the extracted records are manually verified. The extraction template and model parameters are updated based on the verification results, and the extraction record is marked as revised for use in model retraining or fine-tuning.

7. A material data processing system based on image representation, used in the material data processing method according to any one of claims 1 to 6, characterized in that, include: The parsing module is used to perform semantic analysis on target images in a document based on a pre-trained model and generate descriptive explanatory text. An embedding module is used to embed the descriptive explanatory text as placeholder content into the original text of the document to replace the target image; The extraction module is used to perform structured information extraction based on the full text containing the descriptive explanation text; The evaluation module is used to output the accuracy score and completeness score of the extraction results based on embedding matching and numerical relative error calculation, and then combine them into an overall score. The writing module is used to write the extracted records into the knowledge base and record metadata for traceability when the overall score is higher than a preset threshold.

Images