Digital regularization and graph conversion method for large model training

Abstract

The invention discloses a digital regularization and graph conversion method for large model training, which belongs to the technical field of image processing, and comprises the following steps: acquiring a target document, carrying out preprocessing and layout analysis on the target document, and extracting title text block information, body text block information and document picture data; based on multi-scale density analysis, determining a column layout of each page in the target document and a column to which each text block belongs, and establishing a chapter structure; judging the chapter attribution of the text block through two-stage decision, and judging the reference attribution of the document picture based on the reference relationship and the document picture data; reconstructing a document structure, and sequentially generating graphic structured data; and constructing a multi-modal knowledge graph. According to the method, the logic structure of the complex typesetting document is identified and reconstructed, the structured data is extracted, the data utilization rate and the information depth of the complex document are improved, and a high-quality training basis is provided for a large model.

Description

Technical Field

[0001] This invention relates to a method for digital normalization and image transformation for training large models, belonging to the field of image processing technology. Background Technology

[0002] With the breakthroughs achieved by large language models in natural language processing tasks, higher demands are being placed on the quality and scale of training data. Academic papers, industry analysis reports, and technical white papers, which contain high-value knowledge, often have complex layouts, including multi-column layouts and interwoven text and images. Digital preprocessing of these documents, converting them from unstructured formats into machine-readable, semantically rich structured data, is a prerequisite for their effective integration into the large model training process.

[0003] Chinese patent CN120496081A discloses a PDF text extraction method based on image segmentation and OCR, including the following steps: S1, image processing of the PDF file to be parsed; S2, constructing a column determination model to determine whether the image-processed PDF file has columns; S3, constructing an image segmentation model to segment the PDF file with columns, and sequentially calling the OCR algorithm interface to extract text information according to the coordinates and concatenating them in order; S4, filtering headers and footers based on text coordinates; S5, finally performing regular expression filtering and table information filtering to obtain the text data.

[0004] Although existing technologies can segment PDF files with columns and extract and sequentially stitch text information, the stitching of text information relies on simple rules, making it difficult to obtain high-value data that can be used for training large models from complex layouts with multiple columns and paragraph breaks. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a digital regularization and graph transformation method for large-scale model training. This method accurately identifies complex column layouts through multi-scale density analysis, achieves precise reconstruction of document logical structure and intelligent extraction of graphical data, constructs a multimodal knowledge graph, and provides structured data that meets the requirements of large-scale model training.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for digitization and graph transformation for training large models includes:

[0008] S1. Obtain the target document, perform preprocessing and layout analysis on the target document, and extract title text block information, body text block information and document image data;

[0009] S2. Based on multi-scale density analysis, determine the column layout of each page in the target document and the column to which each text block belongs, and establish the chapter structure;

[0010] S3. Determine the chapter affiliation of the main text block through a two-stage decision-making process, and determine the affiliation of the document image based on the citation relationship and document image data;

[0011] S4. Reconstruct the document structure based on chapter and citation affiliation, and generate graphical structured data sequentially;

[0012] S5. Based on the title text block information, body text block information, document image data, document structure and graphic structured data, construct a multimodal knowledge graph.

[0013] Furthermore, the steps to determine the column layout of each page in the target document and the column to which each text block belongs include:

[0014] Set up multi-scale sliding windows based on different proportions of the page width;

[0015] Each sliding window is slid across the page with a preset step size, and the weighted text coverage density of each sliding window under each step size condition is calculated.

[0016] Based on the weighted text coverage density, a density curve corresponding to each sliding window is generated using a linear interpolation algorithm, and a comprehensive density curve is obtained by fusion.

[0017] Calculate the density gradient magnitude at each location on the composite density curve, and select candidate locations for column boundaries;

[0018] A neighborhood is established centered on each candidate position of the column boundary, and the column boundary is determined using a nonmaximum suppression algorithm.

[0019] Based on the column boundaries, the column layout of each page is determined, each column is sequentially numbered, and the column to which each text block belongs is determined.

[0020] Furthermore, the steps for calculating the weighted text coverage density of each sliding window under each step size condition include:

[0021] Get all text blocks that overlap with the current sliding window, and treat them as overlapping text blocks;

[0022] Calculate the Euclidean distance between the center point of each overlapping text block and the center point of the current sliding window, and use it as the sliding center distance;

[0023] Calculate the ratio of the horizontal projection length of each overlapping text block within the current sliding window to the width of the sliding window, and obtain the coverage density values ​​of multiple text blocks corresponding to the current sliding window;

[0024] Based on the distance between the sliding centers, a weighting coefficient is set, and the coverage density values ​​of the multiple text blocks are summed in a weighted manner to obtain the weighted text coverage density value at the current sliding window position.

[0025] Furthermore, the steps to establish a chapter structure include:

[0026] Insert a chapter start tag with the highest level label at the beginning of the target document to create a root chapter;

[0027] Get the title number of each title text block;

[0028] Construct a standard delimiter dataset, determine whether the title number of each title text block contains a preset delimiter in the standard delimiter dataset, and perform a difference analysis on the title number based on the determination result;

[0029] Based on the difference analysis results, the hierarchical structure of each title text block is obtained, and the chapter structure is established.

[0030] Furthermore, the steps for differentiating the title numbers based on the judgment results include:

[0031] If the title number is parsable and contains a preset delimiter, split the title number according to the delimiter to obtain the title code identifier list and text block level;

[0032] Generate the direct parent number based on the coded identifier list;

[0033] Within the processed title text blocks, the title text block hierarchy is determined by backtracking to find title text blocks with the same title number as their direct parent.

[0034] If the title number of the title text block cannot be parsed or does not contain a preset separator, obtain the number of characters of the title number, and calculate the hierarchical confidence score of each title text block based on the number of characters of the title number, the title font size and the title bolding status.

[0035] The hierarchical confidence scores are grouped into multiple clusters using a clustering algorithm. The median point of the hierarchical confidence scores of adjacent cluster centers is calculated as a dynamic threshold for dividing different levels.

[0036] The hierarchical confidence score of each title text block is compared with the dynamic threshold, and the hierarchical structure of each title text block is determined based on the comparison result.

[0037] Furthermore, the steps for determining the chapter affiliation of a text block through a two-stage decision-making process include:

[0038] Starting from the coordinates of the lower left corner of the main text block, the search backtracks in reverse order according to the column number to determine the candidate chapter to which the main text block belongs;

[0039] Based on the candidate chapter determination results, the semantic similarity between the main text block and the title is calculated and it is determined whether it exceeds a preset threshold to further determine the chapter affiliation of the main text block.

[0040] Furthermore, the steps for determining the attribution of document images based on citation relationships and document image data include:

[0041] Based on document image data, search for text blocks that explicitly reference document images throughout the entire document;

[0042] If the search is successful, the text block that is first referenced will be used as the reference to the document image. Otherwise, the distance between the center coordinates of the document image and the center coordinates of each text block on the same page will be calculated, and the text block with the closest distance will be used as the reference to the document image.

[0043] Furthermore, the steps for reconstructing the document structure based on chapter and citation attribution include:

[0044] The main text blocks are grouped according to their chapter affiliation, and the content of each main text block is converted into a semantic vector.

[0045] Based on the grouping results, local semantic clustering is performed on the semantic vectors of text blocks belonging to the same chapter and on the same page, and the text blocks are sorted according to their coordinates.

[0046] For cases where chapters span multiple pages, the bottom text blocks of the same chapter and the top text blocks of the next page are obtained from consecutive pages, and the semantic similarity is calculated to obtain a consecutive text similarity score.

[0047] Get the last character of the text block at the bottom of the page, and set a symbolic logical score based on the last character;

[0048] Each text block is segmented into words. Based on the segmentation results and a pre-defined keyword dictionary, a keyword set for each text block is generated. The Jaccard similarity coefficient of the keyword set for each text block is calculated to obtain the keyword overlap score.

[0049] The weighted sum of the continuous text similarity score, symbolic logic score, and keyword overlap score is used to obtain the merging reliability score; based on the merging reliability score, continuous text blocks are determined and connected in an ordered manner.

[0050] Based on the reference attribution, each document's image is inserted into its corresponding text block, thus completing the document structure reconstruction.

[0051] Furthermore, the steps for generating graph structured data include:

[0052] Using a pre-trained image classification model, we can identify and extract line graphs from document images.

[0053] By training and calling the object detection model, the position and direction of the coordinate axes in each polyline graph are located, and the polyline graph is horizontally calibrated and the image is preprocessed.

[0054] Obtain the axis scale, axis unit, legend name, and color value of the adjacent color block; determine the axis scale range; and establish a mapping relationship between the legend name and the adjacent color block color value.

[0055] Based on the scale range, a data mapping model from coordinate axis pixel coordinates to data values ​​is constructed;

[0056] By using color space segmentation technology, curves of different colors in the line graph are separated; for each curve, key points are extracted along the trajectory, and the data values ​​corresponding to the key points are obtained through the data mapping model.

[0057] The data values ​​are associated with the legend name and coordinate axis units to generate structured data in the format of "dimension-value-unit". The corresponding image tags are then added to mark the inclusion relationship between the document images and the structured graphic data.

[0058] Furthermore, the steps for constructing a multimodal knowledge graph include:

[0059] Obtain the semantic vectors corresponding to the content of each text block and generate argument nodes; generate graphic nodes based on the image data of each document; and generate data nodes based on the graphic structured data.

[0060] Based on the chapter structure, establish the relationship edges between the discussion nodes;

[0061] Based on the attribution of document images, establish unidirectional edges pointing from argument nodes to graphic nodes;

[0062] Based on the image tag, create a one-way edge pointing from the graph node to the data node;

[0063] The argument nodes, graph nodes, data nodes, and all edges are integrated and serialized into a standardized graph structure data format to complete the construction of a multimodal knowledge graph.

[0064] The beneficial effects of this invention are as follows: By using a multi-scale sliding window and Gaussian weighted density calculation, column boundaries are accurately located, and the chapter affiliation of text and images is determined, thus reconstructing the document structure. At the same time, graphic structured data is obtained based on document images, integrating text, images, and structured data into a multimodal knowledge graph that can be understood by machines and is semantically interconnected, providing a high-quality data foundation for large model training. Attached Figure Description

[0065] Figure 1A flowchart of a digital normalization and graph transformation method for training large models;

[0066] Figure 2 This invention provides a flowchart for determining the column layout of each page in the target document and the column to which each text block belongs;

[0067] Figure 3 This is a flowchart illustrating the process of determining the candidate chapter to which a text block belongs in this invention.

[0068] Figure 4 A graphical structured data flow diagram is generated for this invention. Detailed Implementation

[0069] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0070] refer to Figures 1 to 4 As shown, this embodiment introduces a method for digitization regularization and graph transformation for large model training, including:

[0071] S1. Obtain the target document, perform preprocessing and layout analysis on the target document, and extract title text block information, body text block information, and document image data;

[0072] S2. Based on multi-scale density analysis, determine the column layout of each page in the target document and the column to which each text block belongs, and establish the chapter structure;

[0073] S3. Determine the chapter affiliation of the main text block through a two-stage decision-making process, and determine the affiliation of the document image based on the citation relationship and document image data;

[0074] S4. Reconstruct the document structure based on chapter and citation affiliation, and generate graphical structured data sequentially;

[0075] S5. Construct a multimodal knowledge graph based on title text block information, body text block information, document image data, document structure and graphical structured data.

[0076] Furthermore, to achieve accurate location and extraction of the target document content, the target document undergoes preprocessing and layout analysis, including:

[0077] The target document's pages are processed into images, saved as image formats, and then subjected to noise reduction, contrast enhancement, and resolution standardization.

[0078] Using pre-trained deep learning models, such as the YOLO model, to identify text and image regions in a document;

[0079] Use an optical character recognition tool to extract title text block information and body text block information from the text area; wherein, the title text block information includes at least: title number, title text content, title coordinates, title font size, and title bold status, and the body text block information includes at least: body text content and body text block coordinates;

[0080] Extract all document image files from the image region, obtain the coordinates of each document image through object detection algorithm, and extract the document image number and document image name by combining optical character recognition technology.

[0081] Furthermore, since messy documents often involve multiple columns mixed together, it is necessary to clarify the column layout of each page to reduce the interference of column layout on the reading order of reconstructed text blocks. The steps to determine the column layout of each page in the target document and the column to which each text block belongs include:

[0082] Multi-scale sliding windows are set based on different proportions of the page width, such as setting the sliding window width to 1 / 8, 1 / 4, and 1 / 2 of the page width, so as to capture both local boundary features and overall page layout features at the same time, and achieve adaptive recognition of different width column layouts.

[0083] Each sliding window slides on the page with a preset step size. For each sliding window, each slide based on the step size corresponds to a position. At each position, all text blocks that overlap with the current sliding window are obtained. The Euclidean distance between the center point of each text block and the center point of the sliding window is calculated based on the coordinates of the text blocks and the coordinates of the current sliding window.

[0084] Calculate the ratio of the horizontal projection length of each text block within the current sliding window to the width of the sliding window to obtain the coverage density values ​​of multiple text blocks corresponding to the current sliding window;

[0085] The weighted sum of the coverage density values ​​of each text block is used to calculate the weighted text coverage density value at the current sliding window position. To generate a smooth density distribution and reduce noise interference from irrelevant information such as headers and page numbers at the page edges, the weight distribution is calculated using a Gaussian function, with the center point of the sliding window as the origin, so that the weight values ​​decrease smoothly from the center of the window to the edge. The expression for the weight distribution calculation is as follows:

[0086]

[0087] in, This represents the weight of each position in the sliding window, where d is the Euclidean distance between the center point of the text block and the center point of the sliding window, and σ is a standard deviation parameter preset according to the window width, used to control the concentration of the weight distribution.

[0088] Multiple sets of discrete weighted text coverage density values ​​are obtained by sliding each sliding window on the page. Linear interpolation is used to generate density curves corresponding to each sliding window, which reflects the continuous distribution of text density as the window position changes.

[0089] Based on the density curves corresponding to each sliding window, a comprehensive density curve is obtained by fusion through wavelet transform, thus realizing the effective integration of multi-scale information.

[0090] Since column boundaries are usually physically represented as abrupt changes in text density, they will exhibit local maxima in the gradient magnitude of the composite density curve. Therefore, the gradient of the composite density curve is calculated to obtain the density gradient magnitude at each position, and the local extreme points where the density gradient magnitude exceeds a preset threshold are taken as candidate column boundary positions.

[0091] For documents with messy column layouts, the density gradient magnitude reflected by the overall density curve may appear as a continuous region rather than a single pole, leading to spatial clustering of candidate positions. Therefore, a neighborhood radius is set with each column boundary candidate position as the center to establish a neighborhood. Using a non-maximum suppression algorithm, only the position with the largest density gradient magnitude is retained as the column boundary within each neighborhood to eliminate redundant candidate points and ensure the uniqueness and accuracy of the column boundaries.

[0092] Based on the identified column boundaries, the area range of each column is calculated to determine the column layout of each page; each column is sequentially numbered from left to right, and the coordinates of the center point of the text block are calculated and compared with the area range of each column to determine the column to which each text block belongs;

[0093] Furthermore, the steps to establish a chapter structure include:

[0094] Insert a chapter start tag with the highest level label at the beginning of the target document to create a root chapter for the target document, ensuring that all content can be completely organized within the chapter structure;

[0095] Get the title number of each title text block;

[0096] Based on the paper templates of mainstream academic publishing institutions and the industry technical report specifications, a standard delimiter dataset is constructed, and it is determined whether each title number contains the preset delimiter in the standard delimiter dataset.

[0097] If the title number of the title text block is parsable and contains a preset delimiter, split the title number according to the delimiter;

[0098] Based on the segmentation results of each title number, obtain the title code identifier list and text block level. For example, for the title number "2.1.3", the three-level code identifiers "2", "1", and "3" are obtained by splitting it with the separator ".", forming a code identifier list with a length of 3. The title text block is set as a three-level title according to the length of the code identifier list.

[0099] Based on the list of coded identifiers, remove the last-level coded identifier from the current title number to generate the corresponding direct parent number;

[0100] Within the processed title text blocks, backtrack to find title text blocks with the same title number as their direct parent title number. If a successful search is found, the found title text block is designated as the parent title text block of the current title text block. At the same time, title text blocks with the same parent title text block and the same level are established as parallel titles. If no parent title text block is found, it indicates that the parent title is missing. In this case, the current title is assigned as a first-level title under the root chapter of the target document and marked as abnormal content awaiting manual review, in order to ensure the integrity and rationality of the hierarchical structure.

[0101] If the title number of a title text block is unparseable or does not contain a preset separator, it indicates that the hierarchical relationship cannot be directly inferred from the title number structure. The title number character count is obtained, and based on the title number character count, title font size, and title bolding status, a hierarchy confidence score (Score) is calculated for each title text block, as shown in the following expression:

[0102]

[0103] Where L is the number of characters in the title number, S is the normalized title font size, B is the binary label of the title in bold state, with 0 and 1 as the label values ​​for unbold and bold respectively, and α, β, γ are empirical weight coefficients. A multi-source dataset is constructed by obtaining a preset number of title samples from existing academic papers and research reports, and multiple linear regression is performed to generate the optimal combination of weight coefficients to determine the value of the empirical weight coefficients.

[0104] Based on the hierarchical confidence scores of each title text block, clustering algorithms such as K-means are used to group the hierarchical confidence scores into multiple clusters;

[0105] The midpoint of the hierarchical confidence score of adjacent cluster centers is calculated as the dynamic threshold for dividing different levels. For example, after clustering, three clusters are obtained by sorting them in descending order of hierarchical confidence scores, with center points P1, P2, and P3 respectively. The average hierarchical confidence score of P1 and P2 is calculated to obtain the dynamic threshold T1, and the average hierarchical confidence score of P2 and P3 is calculated to obtain the dynamic threshold T2. The three intervals divided by T1 and T2 are used to obtain the hierarchical confidence score range of the three levels.

[0106] The hierarchy confidence score of each title text block is compared with a dynamic threshold. Based on the range in which the hierarchy confidence score falls, the hierarchy of each title text block is determined. Then, a chapter start mark with a hierarchy label is inserted at each title text block to establish the chapter structure.

[0107] Furthermore, the steps for determining the chapter affiliation of a text block through a two-stage decision-making process include:

[0108] S310. Select the coordinates of the lower left corner of the main text block as the starting point;

[0109] S320. Search upwards along the vertical direction of the column containing the starting point from the starting point; if the chapter starting point is successfully found, proceed to step S350, otherwise proceed to step S330.

[0110] S330. Determine if the column containing the starting point is the first column of the page. If so, proceed to step S340. Otherwise, adjust the coordinates of the bottom left corner of the text block in the column preceding the main text block to the starting point, and jump to step S320 to continue execution.

[0111] S340. Determine if the starting point is the first page. If so, determine if the root chapter is the candidate chapter to which the main text block belongs and end the candidate chapter determination process, then proceed to step S360. Otherwise, adjust the coordinates of the bottom left corner of the last column of the bottom text block in the page preceding the starting point to the starting point, and jump to step S320 to continue execution.

[0112] S350. Based on the first chapter start mark found, determine the candidate chapter to which the main text block belongs, end the candidate chapter determination process, and proceed to step S360.

[0113] S360. Based on the determination result of the candidate chapter, the content of the main text block and the content of the title text block of the candidate chapter are converted into title semantic vectors respectively.

[0114] S370. Calculate the cosine similarity between the semantic vectors of the titles to obtain the semantic relevance score;

[0115] S380. If the semantic relevance score exceeds the preset threshold, confirm the candidate chapter to which the main text block belongs as the final chapter assignment; otherwise, mark the main text block as pending content and wait for manual review, and complete the chapter assignment determination for all main text blocks.

[0116] Furthermore, the steps for determining the attribution of document images based on citation relationships and document image data include:

[0117] Within the entire document, use regular expressions to perform pattern matching and search for text blocks that explicitly reference document image numbers;

[0118] If a text block referencing the document image number is successfully found, the text block that is first referenced will be designated as the document image's reference; otherwise, the distance between the center coordinates of the document image and the center coordinates of all text blocks on the same page will be calculated, and the text block with the closest distance will be designated as the document image's reference.

[0119] Furthermore, considering the thematic consistency and logical coherence of each chapter in the document, to avoid semantic confusion between different chapters, the document structure is reconstructed based on chapter affiliation and citation affiliation, including:

[0120] The main text blocks are grouped according to their chapter affiliation, and the content of each main text block is converted into a semantic vector.

[0121] Based on the grouping results, local semantic clustering is performed on the semantic vectors of text blocks belonging to the same chapter and on the same page. By calculating the cosine similarity between semantic vectors and setting a merging threshold, text blocks with cosine similarity exceeding the merging threshold are grouped into one cluster to ensure that text content with similar themes and strong content relevance is clustered. Then, the coordinates of the text blocks are sorted according to the sorting rules of top to bottom within columns and left to right between columns to eliminate the impact of page clutter on the reading order.

[0122] For cases where chapters span multiple pages, the bottom text blocks of the same chapter and the top text blocks of the next page are obtained from consecutive pages. The semantic similarity is calculated based on the corresponding semantic vectors of the bottom and top texts, and used as the consecutive text similarity score.

[0123] Get the last character of the text block at the bottom of the page. If the last character is not a punctuation mark or is a special punctuation mark, it is determined that the text block content is incomplete and has a need for continuation. The punctuation logic score is set to 1 point. Otherwise, the punctuation logic score is set to 0 points. The special punctuation marks include at least the following: comma, pause mark, semicolon, colon, and dash.

[0124] An initial keyword dictionary is constructed based on high-frequency terms from a publicly available academic terminology corpus. Keywords are also added manually to ensure that the keyword dictionary covers core terms of the same subject or document type. Based on a pre-trained Transformer architecture model, the target document is used as input, and a list of weights of the contribution of words and phrases to the topic is output. Words and phrases with weights exceeding a preset threshold are extracted and added to the keyword dictionary as keywords to improve the accuracy of keywords.

[0125] The text block at the bottom of the page and the text block at the top of the next page are segmented using a word segmentation tool. Keywords contained in the keyword dictionary are extracted from the word segmentation results to generate a keyword set for each text block.

[0126] Based on the keyword set, the number of overlapping keywords and the total number of keywords between two text blocks are obtained. The proportion of overlapping keywords to the total number of keywords is calculated to obtain the Jaccard similarity coefficient between the two keyword sets. The closer the Jaccard similarity coefficient is to 1, the higher the keyword overlap between the two text blocks and the stronger the theme consistency. Thus, the theme consistency between text blocks can be quantified as a keyword overlap score.

[0127] A weighted summation function is applied to the continuous text similarity score, symbolic logic score, and keyword overlap score to obtain the merging reliability score. When the merging reliability score reaches a threshold, the text block at the bottom of the page and the text block at the top of the next page are determined to be continuous text blocks and are connected in an ordered manner.

[0128] Based on the reference attribution, the images of each document are inserted into the corresponding text block to complete the document structure reconstruction.

[0129] Furthermore, to automatically extract line chart data from document images and generate high-quality training data that can be directly used by large models, structured graphical data is generated sequentially, including:

[0130] A pre-trained image classification model is used to automatically identify all document images and obtain line graphs from them.

[0131] For each line graph, the position and direction of the coordinate axes in each line graph are located by training and calling the object detection model;

[0132] The line graph is horizontally calibrated based on the coordinate axis position and orientation to eliminate the effects of rotation and tilt, and preprocessing operations are performed to enhance image contrast and filter noise reduction.

[0133] Obtain the axis scales, axis units, and legend names; determine the scale ranges for the X and Y axes; extract the color values ​​of the color blocks next to each legend; and establish a mapping relationship between the legend name and the color values ​​of the color blocks next to the legend.

[0134] Based on the scale range, a data mapping model from coordinate axis pixel coordinates to data values ​​is constructed using a linear interpolation algorithm;

[0135] By using color space segmentation technology, curves of different colors in the line graph are separated; for each curve, key points are extracted along the trajectory, and the pixel coordinates of each key point are converted into data values ​​through a data mapping model, thus completing the conversion from curve image to numerical value;

[0136] Associate data values ​​with legend names and axis units to generate structured data in the format of "dimension-value-unit", and add corresponding image tags to mark the inclusion relationship between document images and structured graphic data.

[0137] Furthermore, in order to transform text blocks and document image data into a machine-understandable, semantically interconnected knowledge network, the construction of a multimodal knowledge graph includes:

[0138] Obtain the semantic vectors corresponding to the content of each text block and generate argument nodes, generate graphic nodes based on the image data of each document, and generate data nodes based on the graphic structured data to realize the structured representation of heterogeneous information.

[0139] Based on the chapter structure, establish the relationship edges between the discussion nodes, describe the inclusion relationship between the discussion nodes, and thus capture the contextual semantics and logical structure;

[0140] Based on the attribution of document images, a one-way edge is established from the argument node to the graph node to provide semantic information of the text block associated with the graph node, thereby enhancing the interpretability and traceability of each document image and the structured data generated therefrom.

[0141] Based on the image tags, establish unidirectional edges from the graph nodes to the data nodes to clarify the source of the data nodes;

[0142] All argument nodes, graph nodes, data nodes, and edges are integrated and serialized into a standardized graph structure data format to complete the construction of a multimodal knowledge graph. This provides a contextual environment for information in each modality and a structured semantic knowledge base for training large models.

[0143] In summary, through multi-scale density analysis and dynamic threshold clustering, this invention identifies and reconstructs the fragmented logical structure of documents caused by multi-column layouts and text-image wrapping, ensuring the semantic coherence of text sequences. Simultaneously, it achieves automated extraction of structured data (dimension-value-unit) based on document images, supporting direct machine reading and learning. Ultimately, it constructs a multimodal knowledge graph integrating text, data, images, and their semantic relationships, transforming fragmented unstructured documents into training samples rich in verifiable reasoning chains. This improves the data utilization and information depth of complex documents, providing a high-quality training foundation for large models.

[0144] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for digital normalization and graph transformation for large model training, characterized in that, include: S1. Obtain the target document, perform preprocessing and layout analysis on the target document, and extract title text block information, body text block information and document image data; S2. Based on multi-scale density analysis, determine the column layout of each page in the target document and the column to which each text block belongs, and establish the chapter structure; S3. Determine the chapter affiliation of the main text block through a two-stage decision-making process, and determine the affiliation of the document image based on the citation relationship and document image data; S4. Reconstruct the document structure based on chapter and citation affiliation, and generate graphical structured data sequentially; S5. Based on the title text block information, body text block information, document image data, document structure and graphic structured data, construct a multimodal knowledge graph.

2. The method for digital normalization and graph transformation for large model training according to claim 1, characterized in that, The steps to determine the column layout of each page in the target document and the column to which each text block belongs include: Set up multi-scale sliding windows based on different proportions of the page width; Each sliding window is slid across the page with a preset step size, and the weighted text coverage density of each sliding window under each step size condition is calculated. Based on the weighted text coverage density, a density curve corresponding to each sliding window is generated using a linear interpolation algorithm, and a comprehensive density curve is obtained by fusion. Calculate the density gradient magnitude at each location on the composite density curve, and select candidate locations for column boundaries; A neighborhood is established centered on each candidate position of the column boundary, and the column boundary is determined using a nonmaximum suppression algorithm. Based on the column boundaries, the column layout of each page is determined, each column is sequentially numbered, and the column to which each text block belongs is determined.

3. The method for digital normalization and graph transformation for large model training according to claim 2, characterized in that, The steps for calculating the weighted text coverage density of each sliding window under each step size condition include: Get all text blocks that overlap with the current sliding window, and treat them as overlapping text blocks; Calculate the Euclidean distance between the center point of each overlapping text block and the center point of the current sliding window, and use it as the sliding center distance; Calculate the ratio of the horizontal projection length of each overlapping text block within the current sliding window to the width of the sliding window, and obtain the coverage density values ​​of multiple text blocks corresponding to the current sliding window; Based on the distance between the sliding centers, a weighting coefficient is set, and the coverage density values ​​of the multiple text blocks are summed in a weighted manner to obtain the weighted text coverage density value at the current sliding window position.

4. The method for digital normalization and graph transformation for large model training according to claim 1, characterized in that, The steps to establish a chapter structure include: Insert a chapter start tag with the highest level label at the beginning of the target document to create a root chapter; Get the title number of each title text block; Construct a standard delimiter dataset, determine whether the title number of each title text block contains a preset delimiter in the standard delimiter dataset, and perform a difference analysis on the title number based on the determination result; Based on the difference analysis results, the hierarchical structure of each title text block is obtained, and the chapter structure is established.

5. The method for digital normalization and graph transformation for large model training according to claim 4, characterized in that, The steps for differentiating title numbers based on the judgment results include: If the title number is parsable and contains a preset delimiter, split the title number according to the delimiter to obtain the title code identifier list and text block level; Generate the direct parent number based on the coded identifier list; Within the processed title text blocks, the title text block hierarchy is determined by backtracking to find title text blocks with the same title number as their direct parent. If the title number of the title text block cannot be parsed or does not contain a preset separator, obtain the number of characters of the title number, and calculate the hierarchical confidence score of each title text block based on the number of characters of the title number, the title font size and the title bolding status. The hierarchical confidence scores are grouped into multiple clusters using a clustering algorithm. The median point of the hierarchical confidence scores of adjacent cluster centers is calculated as a dynamic threshold for dividing different levels. The hierarchical confidence score of each title text block is compared with the dynamic threshold, and the hierarchical structure of each title text block is determined based on the comparison result.

6. The method for digital normalization and graph transformation for large model training according to claim 1, characterized in that, The steps for determining the chapter affiliation of a text block through a two-stage decision-making process include: Starting from the coordinates of the lower left corner of the main text block, the search backtracks in reverse order according to the column number to determine the candidate chapter to which the main text block belongs; Based on the candidate chapter determination results, the semantic similarity between the main text block and the title is calculated and it is determined whether it exceeds a preset threshold to further determine the chapter affiliation of the main text block.

7. The method for digital normalization and graph transformation for large model training according to claim 1, characterized in that, The steps for determining the attribution of document images based on reference relationships and document image data include: Based on document image data, search for text blocks that explicitly reference document images throughout the entire document; If the search is successful, the text block that is first referenced will be used as the reference to the document image. Otherwise, the distance between the center coordinates of the document image and the center coordinates of each text block on the same page will be calculated, and the text block with the closest distance will be used as the reference to the document image.

8. The method for digital normalization and graph transformation for large model training according to claim 1, characterized in that, The steps to reconstruct the document structure based on chapter and citation affiliation include: The main text blocks are grouped according to their chapter affiliation, and the content of each main text block is converted into a semantic vector. Based on the grouping results, local semantic clustering is performed on the semantic vectors of text blocks belonging to the same chapter and on the same page, and the text blocks are sorted according to their coordinates. For cases where chapters span multiple pages, the bottom text blocks of the same chapter and the top text blocks of the next page are obtained from consecutive pages, and the semantic similarity is calculated to obtain a consecutive text similarity score. Get the last character of the text block at the bottom of the page, and set a symbolic logical score based on the last character; Each text block is segmented into words. Based on the segmentation results and a pre-defined keyword dictionary, a keyword set for each text block is generated. The Jaccard similarity coefficient of the keyword set for each text block is calculated to obtain the keyword overlap score. The weighted sum of the continuous text similarity score, symbolic logic score, and keyword overlap score is used to obtain the merging reliability score; based on the merging reliability score, continuous text blocks are determined and connected in an ordered manner. Based on the reference attribution, each document's image is inserted into its corresponding text block, thus completing the document structure reconstruction.

9. The method for digital normalization and graph transformation for large model training according to claim 1, characterized in that, The steps for generating structured graph data include: Using a pre-trained image classification model, we can identify and extract line graphs from document images. By training and calling the object detection model, the position and direction of the coordinate axes in each polyline graph are located, and the polyline graph is horizontally calibrated and the image is preprocessed. Obtain the axis scale, axis unit, legend name, and color value of the adjacent color block; determine the axis scale range; and establish a mapping relationship between the legend name and the adjacent color block color value. Based on the scale range, a data mapping model from coordinate axis pixel coordinates to data values ​​is constructed; By using color space segmentation technology, curves of different colors in the line graph are separated; for each curve, key points are extracted along the trajectory, and the data values ​​corresponding to the key points are obtained through the data mapping model. The data values ​​are associated with the legend name and coordinate axis units to generate structured data in the format of "dimension-value-unit". The corresponding image tags are then added to mark the inclusion relationship between the document images and the structured graphic data.

10. The method for digital normalization and graph transformation for large model training according to claim 1, characterized in that, The steps to construct a multimodal knowledge graph include: Obtain the semantic vectors corresponding to the content of each text block and generate argument nodes; generate graphic nodes based on the image data of each document; and generate data nodes based on the graphic structured data. Based on the chapter structure, establish the relationship edges between the discussion nodes; Based on the attribution of document images, establish unidirectional edges pointing from argument nodes to graphic nodes; Based on the image tag, create a one-way edge pointing from the graph node to the data node; The argument nodes, graph nodes, data nodes, and all edges are integrated and serialized into a standardized graph structure data format to complete the construction of a multimodal knowledge graph.

Images