A document intelligent processing and knowledge base construction method, device and medium

Abstract

The application provides a document intelligent processing and knowledge base construction method, which comprises the following steps: receiving a multi-modal document and performing layout analysis, and dividing the document into several content regions; using an AI model to generate semantic labels for each region; based on the semantic labels, using a dynamic scheduling algorithm which combines business priority and estimated resource consumption to intelligently select and schedule processing models to extract the content of the regions and obtain original data; using an AI model to perform context semantic verification, correction and normalization on the original data, generate structured entity data, and analyze the semantic association relationship between entities; and finally, based on the entities and the association, a traceable domain knowledge base is constructed. The application solves the problems of uneven resource allocation, shallow content understanding and low knowledge level in multi-modal document processing, thereby realizing the construction of an automatic knowledge base.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and information technology, and in particular to a method, device, and medium for intelligent document processing and knowledge base construction. Background Technology

[0002] The efficiency and quality of knowledge acquisition and management directly determine the upper limit of the cognitive capabilities of intelligent systems. With the deepening of digital transformation, numerous high-value fields, such as nuclear power plant operation and maintenance, academic research, and financial analysis, have generated massive amounts of unstructured or semi-structured complex documents. These documents are typically multimodal, containing various information elements such as text, tables, formulas, and charts, and contain the entities and relationships needed to construct domain knowledge graphs. However, how to automatically and accurately extract and structure knowledge from such complex documents, and then build a domain knowledge base that can support intelligent question answering and decision support, remains a serious challenge, placing extremely high demands on the multimodal fusion capabilities, deep semantic understanding capabilities, and computational resource efficiency of information processing technologies.

[0003] Existing technologies already include solutions for document information extraction and processing. For example, patent CN202210066324.X discloses a visual data cleaning script cleaning process analysis system, whose core is to achieve the parsing and visualization of the table data cleaning process through a "program adapter + visualization generator" architecture. However, this type of solution is essentially for post-processing visualization of single, regular data structures. It has fundamental limitations when dealing with the knowledge processing of complex, multimodal documents such as nuclear power plant technical manuals and experimental reports, specifically in the following aspects:

[0004] First, there is a lack of multimodal deep fusion and joint analysis capabilities. Existing technologies typically only parse single modalities such as text or tables, lacking a collaborative analysis mechanism for the overall document layout and cross-modal semantic relationships. This makes it impossible to connect the equipment diagrams (image modality), the adjacent performance parameter tables (table modality), and the mathematical formulas describing their operating principles (formula modality) in nuclear power plant documents into a unified understanding, resulting in fragmented extracted information and difficulty in forming a complete equipment knowledge system.

[0005] Secondly, the semantic understanding and standardization capabilities are insufficient. Existing technologies mostly remain at the shallow processing level of character recognition and field extraction, lacking the ability to identify and normalize the deep semantics of data. For example, they cannot recognize that multiple expressions such as "core temperature," "reactor temperature," and "Tcore" are actually the same entity attribute, nor can they standardize numerical values ​​with different formats and units (such as "25℃" and "298K") into standard values, resulting in a large amount of redundancy and ambiguity in the subsequently constructed knowledge base and poor data consistency.

[0006] Third, there is a lack of intelligent resource scheduling mechanisms for large-scale processing. Existing solutions do not consider the differentiated requirements of computing resources (such as GPU memory and system memory) for different processing stages (such as layout analysis and model inference) during the document knowledge process. When processing the massive historical documents accumulated by nuclear power plants, uneven resource allocation can easily lead to processing bottlenecks, resulting in low system throughput and high response latency, making it difficult to meet the stringent requirements of engineering practice for processing efficiency.

[0007] Fourth, a closed-loop knowledge base has not been formed, resulting in a disconnect from application scenarios. Existing technologies often focus solely on "data extraction," failing to further correlate and integrate the extracted results into a structured knowledge base with rich semantic relationships. Furthermore, they lack mechanisms for connecting with downstream intelligent applications (such as question-answering systems and fault diagnosis engines). This prevents the processing results from being directly used for real-time operation and maintenance support and knowledge retrieval in nuclear power plants, and the technological value cannot be realized in the final application scenarios.

[0008] In summary, existing technologies have significant shortcomings in addressing the knowledge processing needs of complex multimodal documents in high-end equipment fields such as nuclear power plants, particularly in terms of multimodal fusion, semantic understanding, resource efficiency, and application closed-loop. Therefore, there is an urgent need in this field for an integrated solution that can achieve accurate, efficient, and automated document knowledge extraction and knowledge base construction, thereby bridging the entire chain from raw documents to structured knowledge and ultimately enabling intelligent applications. Summary of the Invention

[0009] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0010] According to a first aspect of the present invention, a method for intelligent document processing and knowledge base construction is provided, the method comprising the following steps:

[0011] S100, receiving at least one input multimodal document.

[0012] S200 performs layout analysis on each multimodal document and divides it into several content areas.

[0013] The S300 uses an artificial intelligence model to generate semantic tags for each content area.

[0014] S400, based on the semantic tags, select a processing model to extract the content of each content area and obtain the raw data.

[0015] S500, the original data is processed to generate structured entity data and extract the semantic relationships between entities.

[0016] S600, Based on the entity data and semantic relationships, construct a domain knowledge base.

[0017] According to a second aspect of the present invention, an electronic device is provided, including a processor and a memory; the processor executes the steps of the method described in the first aspect of the present invention by invoking a program or instructions stored in the memory.

[0018] According to a third aspect of the present invention, a computer-readable storage medium is provided that stores a program or instructions that cause a computer to perform the steps of the method described in the first aspect of the present invention.

[0019] The document intelligent processing and knowledge base construction method provided by this invention has at least the following technical effects:

[0020] (1) Achieved deep structuring and automated construction of multimodal document knowledge: This invention successfully transforms unstructured multimodal documents (such as PDF reports containing text, tables, and images) into a structured knowledge base rich in semantic relationships through a coherent process of "layout analysis -> semantic tagging -> content extraction -> semantic processing -> knowledge construction". This completely changes the inefficient traditional method of relying on manual reading, organization, and input, fundamentally automates the knowledge acquisition process, and greatly improves the efficiency and scale of knowledge base construction.

[0021] (2) Improved depth and accuracy of understanding complex document content: This method does not perform simple character recognition or field extraction, but rather uses an artificial intelligence model for deep semantic processing. By generating semantic tags to guide content extraction, and performing contextual semantic verification, correction, and normalization on the original data, high-quality "structured entity data" is generated and "semantic relationships" are extracted. This process significantly improves the accuracy and semantic consistency of information extraction from complex and heterogeneous documents, effectively reduces ambiguity and errors, and lays a solid foundation for building a high-quality knowledge base.

[0022] (3) It possesses broad domain adaptability and application value: Since the core steps of this method do not rely on hard-coded rules for a specific domain, but rather on the semantic understanding capabilities of artificial intelligence models, its application scope is not limited to specific types of documents or domains. This solution can adapt to a variety of complex multimodal documents, from nuclear power plant operation and maintenance manuals and financial analysis reports to academic papers. The structured domain knowledge base constructed can be directly used to support high-end applications such as intelligent question answering, decision support, and knowledge retrieval, forming a complete closed loop from raw data to knowledge services. It has strong technical versatility and high application value.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating a document intelligent processing and knowledge base construction method provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] This invention provides a method for intelligent document processing and knowledge base construction, such as... Figure 1 As shown, the method includes the following steps:

[0028] S100, receiving at least one input multimodal document.

[0029] In this embodiment of the invention, multimodal documents specifically refer to complex documents containing multimodal content in specific fields such as energy, engineering, and scientific research. Documents in the nuclear power plant field are the most representative, specifically including technical documents for nuclear power plants such as nuclear power plant equipment design manuals, nuclear reaction principle descriptions (including equipment structure formulas and principle texts), engineering records such as equipment operation and maintenance logs, construction process ledgers (including operation and maintenance parameter tables and fault description texts), and experimental reports such as nuclear material performance test reports and reactor operation parameter experimental data (including experimental curve images, test data tables, and analysis formulas). These documents generally possess the following characteristics:

[0030] Multimodal coexistence: It includes at least two of the following: text (technical parameter description, operation instructions), tables (operation parameter records, data comparison), formulas (nuclear reaction equations, energy calculation formulas), and images (equipment structure diagrams, experimental curve diagrams), forming a multi-dimensional information system of "text explanation - table quantification - formula derivation - image illustration", that is, it includes at least two of the following modalities: text, tables, formulas, and images.

[0031] Strong correlation: The information of each mode in the document corroborates each other. For example, the parameter "core temperature" mentioned in the text will correspond to specific values ​​in the table (such as 300℃), be used as a calculation variable in the formula (such as T=300℃ being substituted into the power calculation formula), and be reflected in the image through annotations (such as the "T=300℃" label in the core area), forming a close information correlation network.

[0032] S200 performs layout analysis on each multimodal document and divides it into several content areas.

[0033] In this embodiment of the invention, a layout model is used to analyze the layout of the input multimodal document, intelligently dividing it into several content regions. The layout model can employ a deep learning-based multi-task learning architecture and includes the following core components:

[0034] 1. Feature Encoder: Employs a hybrid architecture combining a pre-trained Convolutional Neural Network (CNN) and a Transformer to simultaneously extract visual, textual, and structural features from document images. The CNN branch is responsible for extracting visual features, while the Transformer branch handles text sequences and structural information.

[0035] 2. Multimodal feature fusion module: Dynamically integrates feature representations from different modalities using an attention mechanism to achieve cross-modal feature enhancement.

[0036] 3. Segmentation Decision Module: Based on multimodal fusion features, the segmentation confidence score for each location is calculated using a sliding window mechanism. In an illustrative embodiment, the layout model can divide the multimodal document into several content regions using the following multimodal segmentation confidence function:

[0037] H=λt×ft(L)+λi×fi(E)+λtb×ftb(G)+λf×ff(Sym);

[0038] Where H is the multimodal segmentation confidence score, ft(L) is the text semantic coherence function, calculated based on the text line spacing L, fi(E) is the image edge density function, calculated based on the image edge feature E, ftb(G) is the table structure integrity function, calculated based on the table alignment feature G, ff(Sym) is the formula symbol clustering function, calculated based on the symbol distribution feature Sym, λt, λi, λtb and λf are all modal weight coefficients, and λt+λi+λtb+λf=1.

[0039] Where ft(L) = (1 / n1) × ∑ n1 i1=1 exp(-σ(L i1 ) / Avg(L i1 ), where L i1This represents the line spacing feature of the i1th text line, where i1 ranges from 1 to n1, and n1 is the number of text lines within the sliding window, σ(L i ) and Avg (L i ) represent the standard deviation and mean of the line spacing feature, respectively, and exp() is the natural exponential function. ft(L) reflects the regularity of the text area's layout; the closer the value is to 1, the stronger the semantic coherence of the text.

[0040] fi(E) = (1 / n2) × ∑ n2 i2=1 Ep×G(p), where Ep represents the edge strength obtained by the Canny edge detection algorithm, G(p) is the Gaussian smoothing kernel function, n2 is the total number of pixels in the sliding window, and fi(E) is used to quantify the richness of image content.

[0041] ftb(G) = h1×As + h2×Ls + h3×Cs, where As is the cell alignment score, calculated by performing a Hough transform on the detected table lines; Ls is the line continuity score, calculated based on the line breakpoint detection results; and Cs is the cell regularity score, obtained by analyzing cell size consistency. h1 to h3 are the score weighting coefficients. h1 + h2 + h3 = 1.

[0042] ff(Sym) = (1 / n3) × ∑ n3 i3=1 (ρ(sym) i3 ,c) / max(d(sym i3 ,c)), where, sym i3 Let ρ(sym) represent the i3rd mathematical symbol, where i3 ranges from 1 to n3, n3 is the number of mathematical symbols within the sliding window, and c is the cluster center of the mathematical symbols within the sliding window, i.e., the mean point of the coordinates of all symbols. i3 c) is the symbol density function, d(sym) i3 c) is the distance function. Where ρ(sym) i3 c) = (1 / n3) × ∑ n3 j=1 exp(-||sym j -sym i3 || 2 / 2σ 2 σ is the bandwidth parameter, which controls the smoothness of the density estimation, and is typically taken as 0.5-1 times the average distance between symbols. 2 It is an L2 norm. d(sym) i3 c) = ||sym i3 -c|| 2 .

[0043] In this embodiment of the invention, a rectangular sliding window of a single scale is used, with the window width W set to 1 / 8 of the document width and the window height H set to 1 / 15 of the document height. The sliding step size S is set to 1 / 2 of the window height to ensure sufficient coverage of the document content and detection accuracy.

[0044] Furthermore, the specific steps for dividing the content area include:

[0045] Step 1: Sliding Scan and Confidence Curve Generation

[0046] (1) Using the preset window size (W×H) and sliding step size S, perform sequential sliding scan from the upper left corner to the lower right corner of the document page.

[0047] (2) At each window location, based on the multimodal content (text, image, table, formula) within the window, call the multimodal segmentation confidence function to calculate the confidence score of that window.

[0048] (3) Record the ordinate position (Y-axis coordinate) of the center point of each window and its corresponding confidence score H.

[0049] (4) After traversing the entire document, generate an original confidence curve Hraw(y) based on the distribution of the vertical coordinates of the window center point.

[0050] Step 2: Optimization of Confidence Curve

[0051] The original confidence curve Hraw(y) generated in step one is smoothed using Gaussian smoothing to suppress noise interference, resulting in the smoothed curve Hsmooth(y). The core of the smoothing process is to perform a convolution operation between a one-dimensional Gaussian kernel and the original curve.

[0052] Step 3: Location and Determination of Segmentation Boundaries

[0053] (1) Calculate the global statistic: Based on the smoothed confidence curve Hsmooth(y), calculate its mean μ H and standard deviation σ H .

[0054] (2) Setting a dynamic threshold: A dynamic boundary detection threshold T is set based on global statistics. The threshold calculation formula is: T = μ H –k×σ H Here, k is an adjustable parameter, selected between 1.5 and 3.0 based on the document noise level. The larger the k value, the fewer boundaries are detected, and vice versa.

[0055] (3) Detect candidate boundary points: Scan the Hsmooth(y) curve along the Y-axis to find all points y that satisfy Hsmooth(y) < T and are local minima. candidateThese points are the boundaries for segmenting the candidate document content.

[0056] (4) Boundary confirmation and redundancy removal: Redundancy removal is performed on the candidate boundary point set. If the vertical distance between multiple candidate points is less than the preset minimum region height (e.g., 2H), only the point with the lowest confidence score is retained as the final valid boundary.

[0057] (5) Generate content regions: Sort all the finally confirmed valid segmentation boundary points according to the Y coordinate. The area between these points is the divided content region. Each region is defined by the coordinates of its upper and lower boundaries.

[0058] Step 4: Region Type Labeling (Optional Post-processing)

[0059] All content regions obtained in step three, along with their internal multimodal features, are input again into the layout model or large language model to assign semantic labels, thus completing the entire process of "segmentation" and "labeling".

[0060] Through the steps described above based on a single-scale sliding window, this invention achieves efficient and accurate layout analysis and content segmentation of multimodal documents, laying a solid foundation for subsequent semantic understanding and information extraction.

[0061] Furthermore, the dynamic adjustment of the modality weight coefficients is achieved through a document type classifier, which identifies the type based on the macroscopic layout features of the document.

[0062] When processing academic papers, set λf > λtb > λt > λi, for example, λf = 0.4, λtb = 0.3, λt = 0.2, λi = 0.1.

[0063] When processing business report documents, set λtb > λt > λi > λf, for example, λtb = 0.5, λt = 0.3, λi = 0.15, λf = 0.05.

[0064] When processing promotional poster documents, set λi > λt > λtb > λf, for example, λi = 0.6, λt = 0.25, λtb = 0.1, λf = 0.05.

[0065] The S300 uses an artificial intelligence model to generate semantic tags for each content area.

[0066] In this embodiment of the invention, an artificial intelligence model is used to generate semantic tags for the content region. Preferably, the artificial intelligence model is a Large Language Model (LLM). The semantic tags include a first semantic tag divided according to function or layout and a second semantic tag divided according to content modality.

[0067] The first semantic tag is used to identify the functional role of the content area in the document structure, including but not limited to: headings, body text, section headings, headers, footers, notes, figure captions, and table captions. The second semantic tag is used to identify the main data type contained in the content area, including but not limited to: text, images, tables, formulas, and code.

[0068] In this embodiment of the invention, the artificial intelligence model is configured to generate the above two dimensions of tags in parallel for a single content region, thereby outputting a tag pair containing functional attributes and content modality for each content region.

[0069] Preferably, the generation method of the tag pairs includes:

[0070] The image and text context information of the content area, as well as its positional features in the document, are collectively input into the artificial intelligence model.

[0071] After comprehensively understanding the above information, the artificial intelligence model outputs the label pairs.

[0072] Using the above method, a content region can simultaneously obtain a semantic description in two dimensions. For example:

[0073] The first semantic tag is "body text" and the second semantic tag is "text";

[0074] The first semantic tag is "caption" and the second semantic tag is "text";

[0075] The first semantic tag is "body text" and the second semantic tag is "table";

[0076] The first semantic tag is "footer" and the second semantic tag is "text".

[0077] S400, based on the semantic tags, select a processing model to extract the content of each content area and obtain the raw data.

[0078] Furthermore, the S400 specifically includes:

[0079] S401, estimate the computing resources required to process each content region, and obtain the estimated resource consumption of each content region.

[0080] In this embodiment of the invention, based on the type of content region and historical resource consumption data, the computing resources required to process each content region are estimated, resulting in the estimated resource consumption R for each content region. The estimated resource consumption R is a composite indicator, determined jointly based on historical data of the GPU memory and system memory required to process that type of region, specifically satisfying the relationship: R = f(M est G est ), where G estTo estimate GPU memory consumption, M est To estimate system memory consumption.

[0081] In one illustrative embodiment, R = (1-θ) × (M) est / M max ) + θ × (G est / G max M max G represents the total system memory capacity. max θ represents the total GPU memory capacity, and θ is a configurable weighting coefficient between 0 and 1 used to adjust the relative importance of GPU memory in the overall resource evaluation; for example, it can be set to 0.6.

[0082] In another illustrative embodiment, R = sqrt(f1 × (G) est ) 2 +f2×(M est ) 2 f1 and f2 are weighting coefficients used to normalize G. est and M est Differences in different dimensions and orders of magnitude. sqrt() is the square root function.

[0083] S402 monitors the amount of available computing resources in real time.

[0084] In this embodiment of the invention, the available computing resources are a fusion index corresponding to the estimated resource consumption R, denoted as R. available It is a comprehensive evaluation value derived based on the amount of computing resources currently not occupied by the system and using the same resource dimensions and computing model as in S401.

[0085] The real-time monitoring and acquisition of available computing resources refers to periodically or event-triggered acquisition of the original available amount of the resource dimension by calling the interfaces provided by the operating system, hardware drivers and runtime libraries, and performing real-time calculations based on the computing model.

[0086] Furthermore, the available computing resources R available The core resource dimensions are consistent with S401, including at least:

[0087] Current available GPU memory capacity (G) available ): Refers to the total amount of unallocated video memory on the GPU device in the system;

[0088] Current available system memory capacity (M) available ): This refers to the total amount of memory in the system's main memory (RAM) that is not currently occupied by processes.

[0089] The available computing resources R availableThe method for determining it is as follows:

[0090] The raw available quantity of the core resource dimension, i.e., G, is obtained through a programming interface. available With M available ;

[0091] Substituting the original available quantity into the function f defined in S401, a unified integration index R is calculated. available R available =f(M available G available ).

[0092] As an illustrative example corresponding to S401:

[0093] When the estimated resource consumption R is calculated using R=(1-θ)×(M) est / M max )+θ×(G est / G max During calculation, the available computing resources R available Using R available =(1-θ)×(M available / M max )+θ×(G available / G max )calculate;

[0094] When the estimated resource consumption R is calculated using R=sqrt(f1×(Gest)²+f2×(Mest)²), the available computational resource quantity Ra vailable Using R available =sqrt(f1×(G available )²+f2×(M available )²) Calculate.

[0095] To smooth out the impact of instantaneous fluctuations, the original available quantity G available With M available It can be further filtered by the average or weighted average within a sliding time window before being used to calculate R. available The specific implementation methods of the filtering process include, but are not limited to, the following: simple moving average, exponentially weighted moving average, and weighted moving average. The simple moving average method calculates the arithmetic mean of all raw available data collected within the moving time window. The exponentially weighted moving average method assigns weights to historical data that decay exponentially over time, with newer data receiving higher weights, thus smoothing fluctuations and responding more quickly to trend changes. The weighted moving average method assigns different fixed weights to data at different times within the moving window.

[0096] This invention, through any of the above-mentioned filtering methods, can effectively filter out short-term spikes or spikes in resource availability caused by instantaneous task scheduling, garbage collection, etc., and obtain a smooth curve that better reflects the true trend of available resources in the system. This avoids the scheduler making frequent or erroneous start and stop decisions due to instantaneous fluctuations, and significantly improves the stability and resource utilization efficiency of the system.

[0097] This invention, through a resource monitoring mechanism consistent with the prediction model, ensures the basic consistency of scheduling decisions, thereby significantly improving the accuracy of resource scheduling decisions and the overall stability of the system.

[0098] S403 calculates a comprehensive scheduling evaluation value for all content areas to be processed.

[0099] In this embodiment of the invention, the comprehensive scheduling evaluation value satisfies the following condition: Score = (W × P) / R; where Score is the comprehensive scheduling evaluation value of the content area, P is the service priority of the content area, R is the estimated resource consumption of the content area, and W is the weight of the service priority.

[0100] Furthermore, the service priority P satisfies the following condition:

[0101] P = α×A + β×B + γ×C - δ×D;

[0102] Where A is the standardized weight value based on the positional order of the content region in the document, B is the weight value based on the importance of semantic tags, C is the urgency indicator weight, D is the processing complexity weight, and α, β, γ, δ are configurable weight coefficients that satisfy α+β+γ+δ=1 and δ>max(α, β, γ).

[0103] The value of B is determined based on the semantic label category assigned by the artificial intelligence model through a predefined mapping relationship:

[0104] If the semantic tag is "title" or "section title", then B takes the first preset value;

[0105] If the semantic tag is "body text", then B takes the second preset value;

[0106] If the semantic tag is "header" or "footer", then B takes the third preset value;

[0107] Among them, the first preset value > the second preset value > the third preset value.

[0108] Furthermore, for non-plain text regions processed by specific dedicated processing models, their B values ​​can be further weighted and adjusted based on the inherent importance of their content modality, building upon the above mapping.

[0109] In this embodiment of the invention, the first preset value can be [0.9, 1.0], for example, 1.0; the second preset value is [0.6, 0.8], for example, 0.7; and the third preset value is [0.1, 0.3], for example, 0.2.

[0110] Those skilled in the art will understand that the specific values ​​of the preset values ​​can be adjusted according to different document types and processing objectives. All those that adopt this classification and assignment principle fall within the protection scope of this invention.

[0111] In this embodiment of the invention, the urgency level identifier weight C is an external input variable, whose value is dynamically assigned by the upstream task management system or the user based on business logic. Typically, the assignment strategy for C is as follows:

[0112] For routine processing tasks, set C=0;

[0113] For high-priority tasks (such as real-time interactive requests or tasks in critical data processing pipelines), set C=C high C high For a constant greater than zero (e.g., C). high =0.5);

[0114] For low-priority or background batch tasks, C=C can be set. low C low For a constant less than zero (e.g., C), low =-0.2).

[0115] The specific assignment strategy for C can be configured according to the actual application scenario.

[0116] Furthermore, the processing complexity weight D is calculated using the following formula:

[0117] D=k1×(M est / M max ) + k2 × (G est / G max )+k3×(T est / T max ), where T est To estimate processing time, T max The maximum allowable processing time is given by k1, k2, and k3, which are weighting coefficients, and k1 + k2 + k3 = 1.

[0118] Furthermore, α, β, γ, and δ can be adjusted using an adaptive strategy based on the system load. The strategy is as follows:

[0119] When the system resource utilization rate is lower than the first threshold, increase the values ​​of α and β;

[0120] When the system resource utilization rate is higher than the second threshold, increase the values ​​of δ and γ;

[0121] The system resource utilization rate U satisfies the following conditions:

[0122] U=λ1×(M used / M max )+λ2×(G used / G max )+λ3×(C used / C max ),

[0123] Among them, M used For memory already used, G used For used video memory, C used λ1, λ2, and λ3 represent CPU utilization, and λ1 + λ2 + λ3 = 1.

[0124] In this embodiment of the invention, the first set threshold (low load threshold) and the second set threshold (high load threshold) are constants pre-configured based on system performance results.

[0125] The preferred range for the first set threshold is [0.3, 0.4], for example, 0.35. That is, when the system resource utilization rate U < 0.35, the system is determined to be in a low-load state. The preferred range for the second set threshold is [0.7, 0.8], for example, 0.75. That is, when the system resource utilization rate U > 0.75, the system is determined to be in a high-load state. When U is between the first and second set thresholds, the system is in a normal load state, and the weighting coefficients remain unchanged.

[0126] Furthermore, the weighting coefficients α, β, γ, δ are adjusted using the following adaptive strategy:

[0127] Initialization: Set a set of initial values, such as α=0.3, β=0.3, γ=0.2, δ=0.2, satisfying α+β+γ+δ=1 and δ>max(α,β,γ).

[0128] Low load adjustment: When the system resource utilization U is lower than the first set threshold, a priority service strategy is adopted. The values ​​of α and β are increased by a fixed step size Δ (e.g., Δ=0.05), and the values ​​of γ and δ are decreased accordingly. After adjustment, [α,β,γ,δ] are immediately normalized to ensure that their sum is always 1.

[0129] High load adjustment: When the system resource utilization U is higher than the second set threshold, a priority efficiency and stability strategy is adopted. The values ​​of δ and γ are increased by a fixed step size Δ (e.g., Δ=0.05), and the values ​​of α and β are decreased accordingly. After adjustment, normalization is also performed.

[0130] Normal load maintenance: When U is in the normal range, the weighting coefficients remain unchanged.

[0131] This adaptive adjustment process runs periodically, enabling the scheduling strategy to dynamically respond to changes in system load.

[0132] S404, Sort all content regions to be processed according to the comprehensive scheduling evaluation value.

[0133] S405, Based on the current available computing resources and the aforementioned sorting, the dynamic scheduling processing model executes the extraction task.

[0134] Furthermore, S405 specifically includes the following steps:

[0135] S4051, create a global queue containing all regions to be processed, and sort them in descending order according to the comprehensive scheduling evaluation value.

[0136] S4052, sequentially check each content region in the global queue. If the estimated resource consumption of the currently checked content region is less than or equal to the current available computing resources, immediately schedule the corresponding processing model to process the content region, and update the current available computing resources after processing is completed.

[0137] In this embodiment of the invention, the "corresponding processing model" refers to a dedicated algorithm or model selected from a predefined processing model library based on the semantic tags generated by the artificial intelligence model in step S300 for the content area, specifically designed to process this particular type of content.

[0138] There is a predefined mapping relationship between the processing model and the semantic tags, and the specific mapping methods include, but are not limited to:

[0139] If the semantic label of the content area is "table", then the corresponding processing model is a table recognition model (such as the table recognition module in TableNet or PaddleOCR based on deep learning), which is used to extract the structured data, row and column content and table header information of the table;

[0140] If the semantic tag of the content area is "mathematical formula" or "chemical equation", then the corresponding processing model is a formula recognition engine (such as LaTeX-OCR or a MathPix compatible recognition model), which is used to convert the formula image into an editable standard format (such as LaTeX code).

[0141] If the semantic label of the content area is "natural image", "illustration", or "chart", then the corresponding processing model is an image understanding model or object detection model (such as YOLO, Faster R-CNN or a dedicated chart recognition model), which is used to identify objects in the image, identify chart types and extract their key data labels;

[0142] If the semantic tag of the content area is "body text" or "title text", then the corresponding processing model can be a named entity recognition (NER) model, a keyword extraction algorithm or a text summarization model, used to extract more fine-grained semantic information from the text block;

[0143] If the semantic tags of the content area indicate that its content is data in a specific domain (such as "financial statements" or "legal terms"), then the corresponding processing model can be a customized information extraction model optimized for that domain.

[0144] The scheduling process is as follows: taking the preprocessed content area data as input, loading and executing the dedicated processing model determined by the semantic tags, and outputting the original data of the extracted and structured area after execution, and releasing the computing resources it occupies.

[0145] S4053 If the current available computing resources cannot meet the estimated resource consumption requirements of the next content area in the global queue, then the scheduling process is paused.

[0146] In this embodiment of the invention, "cannot be satisfied" means that the currently available computing resources are less than the estimated resource consumption of the next pending content region in the global queue. The pause scheduling process refers to the resource scheduler entering a low-power waiting state. In this state:

[0147] The scheduler stops sequentially checking new content regions from the global queue to avoid unnecessary computational loops and reduce system overhead.

[0148] The scheduler registers one or more system callback functions to listen for two types of events:

[0149] Resource release event: Triggered by a computing unit that has completed its processing task, it notifies the scheduler that new computing resources have been released;

[0150] New task arrival event: Triggered by the document preprocessing module, it notifies the scheduler that a new area of ​​content to be processed has been added to the global queue.

[0151] The global queue and its sorting state are preserved and maintained during this process until the scheduling process is triggered again.

[0152] S4054, when a new processing task completes and releases resources, or when a new area of ​​content to be processed is added to the global queue, the calculation, sorting, and scheduling process that started in S403 is re-triggered. In this embodiment of the invention, re-triggering means that the above-registered callback function initiates an interrupt or message to wake up the scheduler in a waiting state and start a completely new scheduling cycle. This process specifically includes:

[0153] Resource status reassessment: After the scheduler is woken up, it immediately executes S402 to re-monitor and obtain the latest amount of currently available computing resources.

[0154] Queue recalculation and reordering: Next, execute S403 and S404. This is because the system state has changed:

[0155] If a new task is added: the composition of the global queue has changed. The overall scheduling evaluation score must be recalculated for all pending areas (including newly added ones), and the global queue must be reordered according to the new score to maintain the optimal scheduling priority.

[0156] If only resources are released: Although the queue composition remains unchanged, the increase in available resources may allow high-priority tasks that were previously ranked lower due to insufficient resources to be executed, so recalculation and reordering are also required.

[0157] A new round of scheduling attempts: Finally, execute S405 to restart the sequential check and scheduling based on the latest global queue sorting and the latest available resource quantity.

[0158] Through the above mechanism, this invention implements an event-driven, high-efficiency resource scheduling strategy. The scheduler is activated only when a critical change occurs in the system state, ensuring both the timeliness and accuracy of scheduling decisions and avoiding the waste of computing resources caused by polling, thereby significantly improving the overall energy efficiency and responsiveness of the system.

[0159] S500, the original data is processed to generate structured entity data and extract the semantic relationships between entities.

[0160] In this embodiment of the invention, the processing of the raw data includes: using a pre-trained large language model to perform deep semantic understanding and structured transformation on the raw data extracted from different content regions, thereby generating standardized entity data represented by a unified pattern, specifically including the following processing stages:

[0161] First, contextual semantic validation is performed. By analyzing the context of the raw data within the document, potential anomalies are identified and labeled. Specifically, this includes: checking the reasonableness of the value range of numerical data (e.g., identifying temperature or pressure values ​​outside the reasonable range), discovering logical contradictions (e.g., inconsistent descriptions of the rated power of the same device in different parts), identifying missing references (e.g., undefined component numbers referenced in parameter tables), and labeling semantic ambiguities (e.g., the term "compressor" may refer to either the main unit or the auxiliary unit in different contexts). The large language model, with its powerful semantic understanding capabilities, performs preliminary classification and confidence scoring of these anomalies and generates anomaly-tagged metadata.

[0162] Then, semantic correction is performed, intelligently processing the identified abnormal data based on the overall contextual information of the document. This process includes: for identifiable errors, direct correction based on contextual evidence (e.g., correcting the obviously incorrect 25° to 25℃); for missing information, probabilistic completion through contextual reasoning (e.g., completing the abbreviated device name based on context); for ambiguous expressions, semantic disambiguation through analysis of the context to determine the most likely meaning. During this process, the large language model generates correction suggestions and their confidence assessments. Corrections with low confidence can trigger a manual review process.

[0163] Finally, data normalization is performed, converting data with different representations but identical semantics into a standard form that conforms to a predefined schema specification.

[0164] In this embodiment of the invention, the predefined schema specification is a standardized data model that defines entity types, attributes, data types, and constraints. It specifically specifies entity types, attributes, and constraints. Entity types specify what types of entities are allowed, such as: equipment, personnel, fault events, and operating procedures; attributes specify what attributes each type of entity can have and what data type each attribute is. For example, for a device entity, its schema specification might specify that it must have: device ID (device_id, text type, unique and required), device name (name, text type, required), installation date (install_date, date type), and current status (status, text type, optional values: ['Running', 'Out of service', 'Under maintenance']); constraints are rules that data must follow, including: Requirement: which attributes are mandatory (e.g., device ID) and which are optional (e.g., remarks); Uniqueness: whether the value of an attribute must be unique (e.g., device ID cannot be repeated); Value range: the range of values ​​an attribute can take (e.g., status can only be 'Running', 'Out of service', or 'Under maintenance').

[0165] This phase includes: unifying data representation formats (e.g., unifying various date formats such as October 1, 2023, 10 / 1 / 2023 into the ISO 8601 standard format of 2023-10-01), standardizing terminology (e.g., unifying synonyms such as motor, electric motor, and electric motor into electric motor), standardizing units of measurement (e.g., unifying various power expressions such as 5kW, 5000W, and 5 kilowatts into 5000 watts), and reorganizing structured data (converting unstructured descriptive text into structured key-value pair format).

[0166] Through the above processing flow, the final output is structured entity data defined by a unified pattern and with clearly defined types and standardized values. Each entity data includes processing traceability information, including the original value, correction records, confidence score, and data source location, ensuring the traceability and verifiability of the data, and providing a high-quality, standardized, and semantically consistent data foundation for subsequent knowledge base construction.

[0167] In this embodiment of the invention, an entity refers to a key information object that has been extracted, identified, and standardized from document content. Each entity typically has a unique ID (e.g., PER-12345) to accurately identify it in the knowledge base and avoid confusion. An entity must belong to a predefined category (e.g., Person, Device, Event). This type is determined by semantic tags and subsequent processing. Entities describe their characteristics through attribute-value pairs (e.g., name: Zhang San, model: RC-P-1000). Entities include their source information, such as which document and location they were extracted from.

[0168] In this embodiment of the invention, the extraction of semantic relationships includes: using the artificial intelligence model to comprehensively analyze and infer the layout structure information and deep semantic content of the document, identifying and extracting one or more semantic relationships between different entity data, specifically including the following process:

[0169] The large language model serves as the inference engine, establishing a multimodal collaborative analysis framework for layout structure and semantic content. The model first constructs a spatial relationship graph between entities based on the structured entity data and its spatial distribution characteristics in the original document. These spatial relationships include, but are not limited to: co-occurrence relationships based on the same or adjacent content areas, logical connections based on the document reading order, and hierarchical relationships based on visually separating elements.

[0170] Building on this foundation, the model further performs deep semantic reasoning, analyzing multi-dimensional semantic cues between entities to determine their inherent connections. Specifically, this includes: analyzing the co-occurrence frequency and co-occurrence patterns of entities in contextual descriptions; parsing the grammatical dependency structure at the sentence level to identify the semantic connections implied by grammatical relationships such as subject-verb-object; understanding the argumentative logic at the paragraph and chapter levels to infer possible causal, comparative, or illustrative relationships between entities.

[0171] Through the multimodal comprehensive analysis described above, the model can identify and extract one or more types of semantic relationships between entity pairs. These semantic relationships include, but are not limited to: attribute relationships, component relationships, spatiotemporal relationships, causal relationships, functional dependencies, and domain-specific professional relationships.

[0172] For the identified semantic relationships, the model further calculates their relationship confidence and generates a corresponding set of relationship attributes. The final output semantic relationships, together with the structured entity data, constitute the edges and nodes of the knowledge graph, forming a complete structured knowledge representation.

[0173] This invention significantly improves the accuracy and completeness of semantic relationship extraction by integrating multimodal analysis of document layout structure and deep semantic content, providing core support for building a high-quality domain knowledge base.

[0174] In a preferred embodiment, the semantic association strength is calculated using a quantitative evaluation function that satisfies the following condition:

[0175] Str=g1×Ss+g2×(1 / (1+DL))+g3×Cc;

[0176] Wherein, Str represents the association strength evaluation value of the semantic association, Ss represents the semantic space similarity based on entity vector representation, DL represents the standardized layout distance of the entity in the original document, Cc represents the association degree based on co-occurrence frequency and context dependency analysis, and g1, g2 and g3 are configurable association weight coefficients, and satisfy g1+g2+g3=1.

[0177] S600, Based on the entity data and semantic relationships, construct a domain knowledge base.

[0178] Furthermore, the process of constructing the domain knowledge base specifically includes:

[0179] Knowledge fusion and structured storage: Entity data and semantic relationships from different multimodal documents are integrated. Through entity alignment and disambiguation techniques, conflicts and redundancies in data from different sources are eliminated to form a unified global knowledge graph that is internally consistent and interconnected. The knowledge graph uses structured entity data as nodes and semantic relationships as edges, and attribute information is added to the nodes and edges.

[0180] Source information recording: During the knowledge fusion process, the unique identifier information of the source document and the position information in the original document are recorded for each entity node and relation edge, establishing a traceable link from knowledge base elements to the original source document;

[0181] Index building: Build multi-dimensional indexes for entities, relationships, attributes and source information in the knowledge graph to support efficient multi-condition joint queries and source retrieval.

[0182] Furthermore, entity data and semantic relationships from different multimodal documents are integrated. Entity alignment and disambiguation techniques are used to eliminate conflicts and redundancies in data from different sources, forming a unified global knowledge graph that is internally consistent and interconnected. Specifically, this includes:

[0183] S601, Preprocessing and Normalization:

[0184] The entity data and semantic relationships extracted from all input documents are standardized and preprocessed to ensure that they conform to a unified schema specification. This includes uniform conversion of formats such as time, date, and units of measurement, and normalization mapping of synonyms.

[0185] S602, Entity Alignment:

[0186] Identify and merge entities from different data sources that point to the same object in the real world. The entity alignment process calculates and matches similarity based on one or more of the following features:

[0187] Attribute feature similarity: Calculate the text similarity of core attributes of entities (such as name, unique identifier) ​​(e.g., based on Jaccard distance, edit distance or word vector model).

[0188] Similarity of relational features: Compare the neighboring nodes and relational types of entities (for example, if two entities with similar names, such as "engineer", are both associated with the same "work unit" entity, the probability that they are the same person is significantly increased).

[0189] Contextual feature similarity: Matching entities using the contextual semantic information in the original document.

[0190] When the overall similarity score exceeds the preset confidence threshold, the two entities are determined to refer to the same object, and a merging operation is triggered.

[0191] S603, Entity Disambiguation:

[0192] Distinguish entities with the same or similar names but referring to different real-world objects. The disambiguation process is based on the following strategy:

[0193] Contextual analysis: Analyzing the local context in which an entity appears. For example, an entity named "Li A" should be associated with the "athlete" type in the context of a tennis-related document, while it may be associated with the "singer" type in a music-related document, thus distinguishing it as two different entities.

[0194] Relationship identification: Identifying entities based on their different relationships. For example, "Zhang San" who appears in different companies and holds different positions should be classified as different "Person" entities.

[0195] Each disambiguated entity is assigned a globally unique identifier (UUID) to ensure its distinguishability in the knowledge base.

[0196] S604, Conflict Detection and Resolution:

[0197] Detect conflicting attribute values ​​provided by different data sources to the same entity and resolve them according to predefined strategies. These strategies include, but are not limited to:

[0198] Trusted Source Priority: Assign trust weights to different data sources, and prioritize values ​​provided by high-trust sources;

[0199] Latest timestamp priority: Use the attribute value provided in the latest source document;

[0200] Voting strategy: Use the value supported by the majority of sources;

[0201] Manual intervention interface: For high-confidence conflicts or critical data, the conflict information will be submitted to the manual review interface for processing.

[0202] S605, Global Graph Construction and Storage:

[0203] The entities and relationships, after alignment, disambiguation, and conflict resolution, are stored in a graph database in the form of a graph structure. In this graph structure, entities are used as nodes, semantic relationships are used as edges, and source information of all their source documents is attached to each node and edge, ultimately forming a unified and internally consistent global knowledge graph.

[0204] S606, Quality Assessment and Iterative Updates:

[0205] The completed global knowledge graph undergoes a quality assessment, with evaluation metrics including completeness, consistency, and accuracy. Based on the assessment results and subsequent newly ingested document data, the knowledge graph is iteratively updated and optimized to ensure its continuous evolution and maintain high quality.

[0206] Through the above steps, this invention enables the automatic construction of a high-quality, highly consistent, and semantically rich unified knowledge base from multi-source, heterogeneous, and potentially noisy document data, providing reliable structured knowledge support for upper-layer applications.

[0207] Furthermore, the application of the domain knowledge base includes:

[0208] The domain knowledge base is configured to be externally connected to various artificial intelligence models, serving as their external knowledge source and providing real-time and accurate knowledge enhancement support for the models;

[0209] When the artificial intelligence model is in use, it first receives the user's query and retrieves relevant entities and relational subgraphs from the domain knowledge base based on the query; then, it combines the retrieved structured knowledge with the user's query to form enhanced prompt information.

[0210] The artificial intelligence model performs reasoning and content generation based on the enhanced prompts, and finally outputs a response that is rich in structured knowledge and has traceable evidence.

[0211] Furthermore, the domain knowledge base supports querying and tracing by document source, specifically implemented as follows:

[0212] It provides a dedicated traceability query interface, allowing users or systems to query all or part of the source document information by specifying an entity or relationship;

[0213] Conversely, one or more source documents can be specified to query all entities and relationships extracted from the document, thereby achieving the verifiability of knowledge sources and the traceability of knowledge updates.

[0214] This invention, through the aforementioned knowledge base construction and application methods, realizes a value loop from multimodal documents to structured knowledge, and then to intelligent applications, significantly improving the cognitive ability and answer accuracy of artificial intelligence models in professional fields.

[0215] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0216] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. A process can be terminated when its operation is complete, but it may also have additional steps not included in the figures. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0217] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in this invention.

[0218] This invention also provides a computer-readable storage medium storing computer-executable instructions for performing the methods described in this invention.

[0219] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0220] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for intelligent document processing and knowledge base construction, characterized in that, The method includes the following steps: S100, receiving at least one input multimodal document; S200 performs layout analysis on each multimodal document and divides it into several content areas; S300 uses an artificial intelligence model to generate semantic tags for each content area; S400, based on the semantic tags, select a processing model to extract the content of each content area and obtain the raw data; S500, the original data is processed to generate structured entity data and extract the semantic relationships between entities; S600, Based on the entity data and semantic relationships, construct a domain knowledge base; The S400 includes: S401, Estimate the computing resources required to process each content region, and obtain the estimated resource consumption of each content region; S402, real-time monitoring of available computing resources; S403, calculate a comprehensive scheduling evaluation value for all content areas to be processed; the comprehensive scheduling evaluation value satisfies the following condition: Score = (W × P) / R; where Score is the comprehensive scheduling evaluation value of the content area, P is the service priority of the content area, R is the estimated resource consumption of the content area, and W is the weight of the service priority. The service priority P satisfies the following condition: P = α×A + β×B + γ×C - δ×D; Where A is the standardized weight value based on the positional order of the content region in the document, B is the weight value based on the importance of semantic tags, C is the urgency indicator weight, D is the processing complexity weight, and α, β, γ, δ are configurable weight coefficients that satisfy α+β+γ+δ=1 and δ>max(α, β, γ). S404, Sort all content regions to be processed according to the comprehensive scheduling evaluation value; S405, Based on the current available computing resources and the aforementioned sorting, the dynamic scheduling processing model executes the extraction task.

2. The method according to claim 1, characterized in that, S405 specifically includes the following steps: S4051, Create a global queue containing all regions to be processed, and sort them in descending order according to the comprehensive scheduling evaluation value; S4052, sequentially check each content region in the global queue. If the estimated resource consumption of the currently checked content region is less than or equal to the current available computing resources, immediately schedule the corresponding processing model to process the content region, and update the current available computing resources after processing is completed. S4053, If the current available computing resources cannot meet the estimated resource consumption requirements of the next content area in the global queue, then the scheduling process is paused. S4054 When a new processing task completes and releases resources, or when a new area of ​​content to be processed is added to the global queue, the calculation, sorting, and scheduling process that started from S403 is re-triggered.

3. The method according to claim 2, characterized in that, D=k1×(M est / M max ) + k2 × (G est / G max )+k3×(T est / T max ), where M est To estimate memory consumption, M max G represents the total system memory capacity. est To estimate GPU memory consumption, G max T represents the total video memory capacity of the GPU. est To estimate processing time, T max The maximum allowable processing time is given by k1, k2, and k3, which are weighting coefficients, and k1 + k2 + k3 = 1.

4. The method according to claim 1, characterized in that, In S200, the multimodal document is divided into several content regions using the following multimodal segmentation confidence function: H=λt×ft(L)+λi×fi(E)+λtb×ftb(G)+λf×ff(Sym); Where H is the multimodal segmentation confidence score, ft(L) is the text semantic coherence function, calculated based on the text line spacing L, fi(E) is the image edge density function, calculated based on the image edge feature E, ftb(G) is the table structure integrity function, calculated based on the table alignment feature G, ff(Sym) is the formula symbol clustering function, calculated based on the symbol distribution feature Sym, λt, λi, λtb and λf are all modal weight coefficients, and λt+λi+λtb+λf=1.

5. The method according to claim 1, characterized in that, The processing of the raw data includes: using a pre-trained artificial intelligence model to perform context-based semantic verification, content correction, and format normalization on the raw data, thereby generating standardized entity data represented by a unified pattern.

6. An electronic device, characterized in that, Including processor and memory; The processor executes the steps of the method as described in any one of claims 1 to 5 by invoking programs or instructions stored in the memory.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a program or instructions that cause a computer to perform the steps of the method as described in any one of claims 1 to 5.

Images