Multi-modal document structured processing and knowledge extraction method based on large language model

Abstract

The invention discloses a multi-modal document structured processing and knowledge extraction method based on a large language model. The method comprises the following steps: S1, receiving an input multi-modal document; s2, performing preprocessing on the text data in the multi-mode document; s3, performing feature extraction on non-text data in the multi-mode document; s4, performing multi-modal data fusion on the preprocessed text data and the non-text data after feature extraction; s5, performing deep semantic analysis on the fused multi-modal data through a pre-trained improved BERT model, wherein the deep semantic analysis comprises key entity recognition, relation extraction and event detection; s6, based on a deep semantic analysis result, automatically constructing a knowledge graph by using the extracted information; and S7, outputting the data of the knowledge graph in a format for analysis or application. According to the method and the system, structured processing and knowledge extraction of unstructured and multi-modal documents are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal documents, and in particular to a multimodal document structured processing and knowledge extraction method based on a large language model. Background Art

[0002] The processing of unstructured and multimodal documents in the existing technology mainly relies on single-modal data processing methods, such as processing only one type of data among text, images or charts. Although the processing methods of the existing technology can effectively realize information extraction and processing in some scenarios, when faced with complex documents containing multi-modal data, the existing technology seems to be unable to cope with it. The traditional single-modal processing method cannot effectively integrate and analyze multiple types of data, resulting in obvious limitations in information extraction and knowledge representation. Especially when a large number of unstructured and multimodal documents need to be processed automatically, traditional technologies often need to rely on a lot of manual intervention and cannot achieve efficient automatic processing.

[0003] In the existing technology, text processing technologies such as natural language processing have developed to a relatively mature level. In particular, applications based on the BERT large language model can achieve certain text semantic understanding, word segmentation, and part-of-speech tagging functions. However, most of these are only for pure text data and lack the ability to process non-text data. As for feature extraction technologies for images and charts, convolutional neural network visual models are mostly used, but the models cannot be deeply integrated with text data, resulting in insufficient accuracy and breadth of knowledge extracted from multimodal data.

[0004] To sum up, the existing technology has the following main shortcomings: first, the existing unimodal processing method is less applicable in multimodal document processing, and lacks effective multimodal data fusion methods; second, the existing text processing and visual feature extraction technologies are separated from each other and cannot achieve a joint understanding of multimodal data. Summary of the invention

[0005] One object of the present invention is to propose a method for structuring multimodal documents and extracting knowledge based on a large language model. The present invention realizes structuring processing and knowledge extraction of unstructured and multimodal documents.

[0006] A multimodal document structured processing and knowledge extraction method based on a large language model according to an embodiment of the present invention comprises the following steps:

[0007] S1. receiving an input multimodal document, wherein the multimodal document comprises at least one text data and at least one non-text data;

[0008] S2, preprocessing the text data in the multimodal document, wherein the preprocessing includes word segmentation, part-of-speech tagging, syntactic analysis and entity recognition;

[0009] S3, performing feature extraction on the non-text data in the multimodal document;

[0010] S4, performing multimodal data fusion on the preprocessed text data and the feature-extracted non-text data;

[0011] S5. performing deep semantic analysis on the fused multimodal data through a pre-trained improved BERT model;

[0012] S6. Based on the result of the deep semantic analysis, automatically construct a knowledge graph from the extracted information;

[0013] S7. Output the knowledge graph data into a format that can be analyzed or applied.

[0014] Optionally, the step S1 includes:

[0015] S11, receiving a multimodal document input signal provided by multiple data sources, wherein the data sources include a text data source and a non-text data source, wherein the text data source provides text data containing natural language content, and the non-text data source provides images, charts, or other types of non-text data:

[0016]

[0017] Among them, D m represents the received multimodal document input dataset, T di Represents the i-th text data unit, N dj represents the jth non-text data unit, N t and N n Represents the number of text data and non-text data respectively. Represents the parallel reception and combination operations of multimodal data;

[0018] S12, text data T d and non-text data N d Perform synchronous input operations so that text data and non-text data are simultaneously imported into the processing system;

[0019] S13, input text data T d and non-text data N d Perform preliminary data type identification based on metadata identifier M t Verify the text data type through the image feature matrix I f Validate non-text data types;

[0020] S14, according to the recognition result, the text data T d and non-text data N dClassification and storage in text data cache unit C t and non-text data cache unit C n middle.

[0021] Optionally, step S2 includes:

[0022] S21, using the vocabulary matching algorithm based on the contextual attention mechanism to receive the text data T d Perform word segmentation processing and execute word segmentation operation:

[0023]

[0024] Among them, T d Represents the input text data, T di Represents the i-th word in the text data, L m (T d ) is the result of word segmentation processing, α i and β i Respectively represent the global dictionary matching weight and context similarity weight, W(T di ) represents the matching score of the entries in the dictionary, C(T di ) represents word T di The context word set, Sim(T di , T j ) represents word T di With context word T j similarity;

[0025] S22. Use the part-of-speech tagging model based on conditional random fields to perform part-of-speech tagging on the text data after word segmentation. The part-of-speech tagging process is as follows:

[0026]

[0027] Among them, T di represents the i-th word, P di represents the corresponding part-of-speech tag, ψ(T di , P di ) represents word T di Its part of speech P di The compatibility function between di-1 , P di ) represents the probability of part-of-speech transfer, P s (T d ) represents the part-of-speech tag set of the text data after word segmentation;

[0028] S23, perform syntactic analysis on the annotated text data, and use the dependency tree syntactic analysis model to construct a grammatical structure tree of the text. The specific process is as follows:

[0029]

[0030] Among them, T di and T dj Represents two words in the text, δ(T di , T dj ) represents the distance between words, τ(P di , P dj ) represents the dependency weight between parts of speech, Dep(T di , T dj ) represents the strength of the dependency relationship between words, S t (T d ) represents the syntax structure tree of text data;

[0031] S24. Use an entity recognition model based on bidirectional long short-term memory and conditional random field to perform entity recognition on the text data after syntactic analysis, and perform entity recognition:

[0032]

[0033] Among them, T di represents the i-th word in the text, E di represents the corresponding entity label, λ(T di , E di ) represents the compatibility score between the word and the entity tag, μ(E di-1 , E di ) represents the transfer score between entity labels, γ represents the fusion coefficient between multimodal data, Att(T di , F j ) represents the text word T di and non-text data features F j The attention weight, E r (T d ) represents the entity recognition result in text data.

[0034] Optionally, step S3 includes:

[0035] S31, image data I in multimodal document d Perform visual feature extraction and use convolutional neural network to analyze the spatial features in the image data to extract the image feature vector V i :

[0036]

[0037] Among them, I d Represents the input image data, I lk represents the lth layer and the kth feature point in the image, w lk represents the weight parameter, f(I lk) represents the nonlinear activation function of the image feature points, b lk Indicates the bias value, V i A visual feature vector representing an image;

[0038] S32, use the model based on the region proposal network to detect the target object in the image data and generate the candidate region box R i , and perform target classification on each region box:

[0039]

[0040] Among them, R pi represents the i-th candidate region, O i represents the true target area, P represents the number of candidate areas, φ(R pi ) is the confidence score of the region proposal, IoU(R pi , O i ) represents the intersection-over-union ratio between the candidate region and the true target region;

[0041] S33, for the chart data G in the multimodal document d Perform structural recognition and extract key structural information S in the graph g , including the relationship between axes, data points, labels and lines:

[0042]

[0043] Among them, G d Represents the input chart data, G dj represents the jth data element in the graph, L j Represents the label associated with the data element, γ j Represents the weight parameter, Rel(G dj , L j ) represents the strength of the association between the chart data element and its label, S g Represents the structure recognition result of the graph.

[0044] Optionally, step S4 includes:

[0045] S41, the pre-processed text data T d The eigenvector P s (T d ) and the non-text data N after feature extraction v Perform feature vector fusion and construct a multimodal feature matrix M f :

[0046]

[0047] Among them, P s (Td ) represents the part-of-speech tagging feature vector of text data, V i Represents the visual feature vector of image data, S g represents the structural features of the graph data, λ1 and λ2 represent the fusion weights of text and non-text data respectively, and M f Represents the fused multimodal feature matrix;

[0048] S42. Calculate the correlation score A between text data and non-text data based on the attention mechanism s (T d , N d ), where the word segmentation result of the text data is L m (T d ) and the candidate region box R of non-text data i Used to calculate the similarity score:

[0049]

[0050] Among them, L m (T di ) represents the word segmentation result in the text data, R j represents the candidate region in non-text data, α ij represents the attention weight between text data and non-text data, Sim(L m (T di ), R j ) represents the similarity score between the text data word and the non-text data candidate region, A s (T d , N d ) represents the correlation score between text data and non-text data;

[0051] S43. Based on the correlation score A s (T d , N d ) Information fusion is performed on multimodal data to generate the final multimodal feature matrix M r :

[0052]

[0053] Among them, β ij represents the weight coefficient after fusion, represents the fusion operation of text data words with image and chart features, M r Represents the final generated multimodal representation.

[0054] Optionally, step S5 includes:

[0055] S51, the fused multimodal feature matrix M rThe input is fed into the pre-trained improved BERT model to perform deep semantic analysis. The improved BERT model jointly encodes the embedded representations of text and non-text data to generate a multimodal semantic vector S v ;

[0056] S52, through the attention mechanism in the multimodal semantic vector M r The improved BERT model calculates the attention weights of each word and non-text features to identify key entities E in multimodal data. k :

[0057]

[0058] Among them, S vi represents the i-th word or feature in the multimodal semantic vector, T di represents the corresponding text words, α i represents the attention weight;

[0059] S53, deep semantic relationship extraction module based on the improved BERT model, calculates the relationship R between entities in the multimodal semantic vector s (E k ), and identify the event Ev in the multimodal semantic vector through the event detection module s .

[0060] Optionally, step S6 includes:

[0061] S61. Based on the results of deep semantic analysis, the multimodal semantic vector S v Extract text entity node E t and non-text entity node E n , construct the initial node set N of the knowledge graph g :

[0062] Text entity node E t Represents people, places, and events extracted from text data;

[0063] Non-text entity node E n Represent objects and data points extracted from images and charts as non-text data;

[0064] S62: Generate edges L between nodes based on the semantic relationship between text data and non-text data g , the edges include edges representing the relationship between the text description and the object in the image and edges representing the relationship between the chart data and the text description;

[0065] S63. Weights ω of nodes and edges based on graph structure learning model n and ω lOptimize and generate the optimized knowledge graph K g .

[0066] Optionally, the step S63 includes:

[0067] S631, node weights ω in the initial knowledge graph based on graph structure learning model n Initialize with edge weight ω1, node weight ω n Initialized to the degree of the node, the edge weight ω1 is initialized to the strength of the association between the nodes;

[0068] S632: Optimize node weights and edge weights based on the gradient descent algorithm, and iteratively update the weight values ​​of nodes and edges and

[0069]

[0070]

[0071] Among them, η2 is the learning rate, and Respectively represent the partial derivatives of the loss function L with respect to the node weight and edge weight;

[0072] S633, the loss function L is optimized by minimizing the error between nodes and edges. The purpose of the optimization is to improve the accuracy and relevance of the knowledge graph by minimizing L:

[0073]

[0074] Among them, Sim(E i , E j ) represents node E i With node E j The similarity between them, L represents the total loss between nodes and edges;

[0075] S634, in the optimization iteration process, according to the optimal node weight and edge weights Generate the final optimized knowledge graph K g :

[0076]

[0077] Among them, K g Represents the final optimized knowledge graph, including the optimized node set N g , edge set L g And the node weight and edge weights

[0078] The beneficial effects of the present invention are:

[0079] (1) The present invention fuses text data with non-text data and proposes a multimodal fusion method based on an improved BERT model. The feature extraction model is used to extract features from text and non-text data respectively, and then the features are fused through an adaptive weight mechanism to generate a multimodal feature matrix. This not only improves the accuracy of identifying the correlation between text and image and chart data, but also enhances the system's ability to understand complex documents, enabling the system to extract more accurate and comprehensive semantic information from multimodal data, thus overcoming the limitations of single-modal processing in the prior art.

[0080] (2) In the technical solution of the present invention, the node weights and edge weights of the knowledge graph are adaptively optimized based on the graph structure learning model. By introducing the gradient descent algorithm and optimizing the calculation of the loss function, the weight distribution of the nodes and edges in the knowledge graph can be dynamically adjusted. The optimized knowledge graph can more accurately express the semantic relationship between text and non-text data. Compared with the existing knowledge graph construction technology, the present invention can not only process pure text data, but also effectively combine image and chart data, so that the constructed knowledge graph has higher expression ability and accuracy, ensuring that the key knowledge in complex documents can be fully represented.

[0081] (3) The present invention significantly improves the efficiency of automated processing of complex documents by introducing an improved BERT model and a self-supervised learning mechanism, combined with multimodal fusion technology. Unlike traditional technologies that require a lot of manual intervention, the present invention can achieve automated knowledge extraction and structured processing through deep semantic analysis, entity recognition, and relationship extraction steps under unsupervised or weak supervision, reducing the complexity of manual operations and greatly reducing the time cost of processing multimodal documents. It improves the overall processing efficiency of the system and solves the problem of low efficiency of existing technologies in large-scale, multimodal document processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0083] Figure 1 This is a flowchart of a multimodal document structuring processing and knowledge extraction method based on a large language model proposed by the present invention. DETAILED DESCRIPTION

[0084] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0085] refer to Figure 1 , a multimodal document structured processing and knowledge extraction method based on a large language model, comprising the following steps:

[0086] S1. Receive an input multimodal document, where the multimodal document includes at least one text data and at least one non-text data;

[0087] S2, preprocessing the text data in the multimodal document, including word segmentation, part-of-speech tagging, syntactic analysis and entity recognition;

[0088] S3, extracting features from non-text data in multimodal documents;

[0089] S4, performing multimodal data fusion on the preprocessed text data and the feature-extracted non-text data;

[0090] S5. Perform deep semantic analysis on the fused multimodal data through the pre-trained improved BERT model;

[0091] S6. Based on the results of deep semantic analysis, automatically construct a knowledge graph from the extracted information;

[0092] S7. Output the knowledge graph data into a format that can be analyzed or applied.

[0093] In this implementation, step S1 includes:

[0094] S11. Receive a multimodal document input signal provided by multiple data sources, where the data sources include a text data source and a non-text data source, where the text data source provides text data containing natural language content, and the non-text data source provides images, charts, or other types of non-text data:

[0095]

[0096] Among them, D m represents the received multimodal document input dataset, T di Represents the i-th text data unit, N dj represents the jth non-text data unit, N t and N n Represents the number of text data and non-text data respectively. Represents the parallel reception and combination operations of multimodal data;

[0097] S12, text data T d and non-text data N d Perform synchronous input operations so that text data and non-text data are simultaneously imported into the processing system;

[0098] S13, input text data T dand non-text data N d Perform preliminary data type identification based on metadata identifier M t Verify the text data type through the image feature matrix I f Validate non-text data types;

[0099] S14, according to the recognition result, the text data T d and non-text data N d Classification and storage in text data cache unit C t and non-text data cache unit C n middle.

[0100] In this implementation, step S2 includes:

[0101] S21, using the vocabulary matching algorithm based on the contextual attention mechanism to receive the text data T d Perform word segmentation processing and execute word segmentation operation:

[0102]

[0103] Among them, T d Represents the input text data, T di Represents the i-th word in the text data, L m (T d ) is the result of word segmentation processing, α i and β i Respectively represent the global dictionary matching weight and context similarity weight, W(T di ) represents the matching score of the entries in the dictionary, C(T di ) represents word T di The context word set, Sim(T dj , T j ) represents word T di With context word T j similarity;

[0104] S22. Use the part-of-speech tagging model based on conditional random fields to perform part-of-speech tagging on the text data after word segmentation. The part-of-speech tagging process is as follows:

[0105]

[0106] Among them, T di represents the i-th word, P di represents the corresponding part-of-speech tag, ψ(T di , P di ) represents word T di Its part of speech P di The compatibility function between di-1 , P di) represents the probability of part-of-speech transfer, P s (T d ) represents the part-of-speech tag set of the text data after word segmentation;

[0107] S23, perform syntactic analysis on the annotated text data, and use the dependency tree syntactic analysis model to construct a grammatical structure tree of the text. The specific process is as follows:

[0108]

[0109] Among them, T di and T dj Represents two words in the text, δ(T di , T dj ) represents the distance between words, τ(P di , P dj ) represents the dependency weight between parts of speech, Dep(T di , T dj ) represents the strength of the dependency relationship between words, S t (T d ) represents the syntax structure tree of text data;

[0110] S24. Use an entity recognition model based on bidirectional long short-term memory and conditional random field to perform entity recognition on the text data after syntactic analysis, and perform entity recognition:

[0111]

[0112] Among them, T di represents the i-th word in the text, E di represents the corresponding entity label, λ(T di , E di ) represents the compatibility score between the word and the entity tag, μ(E di-1 , E di ) represents the transfer score between entity labels, γ represents the fusion coefficient between multimodal data, Att(T di , F j ) represents the text word T di and non-text data features F j The attention weight, E r (T d ) represents the entity recognition result in text data.

[0113] In this implementation, step S3 includes:

[0114] S31, image data I in multimodal document d Perform visual feature extraction and use convolutional neural network to analyze the spatial features in the image data to extract the image feature vector V i :

[0115]

[0116] Among them, I d Represents the input image data, I lk represents the lth layer and the kth feature point in the image, w lk represents the weight parameter, f(I lk ) represents the nonlinear activation function of the image feature points, b lk Indicates the bias value, V i A visual feature vector representing an image;

[0117] S32, use the model based on the region proposal network to detect the target object in the image data and generate the candidate region box R i , and perform target classification on each region box:

[0118]

[0119] Among them, R pi represents the i-th candidate region, O i represents the true target area, P represents the number of candidate areas, φ(R pi ) is the confidence score of the region proposal, IoU(R pi , O i ) represents the intersection-over-union ratio between the candidate region and the true target region;

[0120] S33, for the chart data G in the multimodal document d Perform structural recognition and extract key structural information S in the graph g , including the relationship between axes, data points, labels and lines:

[0121]

[0122] Among them, G d Represents the input chart data, G dj represents the jth data element in the graph, L j Represents the label associated with the data element, γ j Represents the weight parameter, Rel(G dj , L j ) represents the strength of the association between the chart data element and its label, S g Represents the structure recognition result of the graph.

[0123] In this implementation, step S4 includes:

[0124] S41, the pre-processed text data T d The eigenvector P s (Td ) and the non-text data N after feature extraction v Perform feature vector fusion and construct a multimodal feature matrix M f :

[0125]

[0126] Among them, P s (T d ) represents the part-of-speech tagging feature vector of text data, V i Represents the visual feature vector of image data, S g represents the structural features of the graph data, λ1 and λ2 represent the fusion weights of text and non-text data respectively, and M f Represents the fused multimodal feature matrix;

[0127] S42. Calculate the correlation score A between text data and non-text data based on the attention mechanism s (T d , N d ), where the word segmentation result of the text data is L m (T d ) and the candidate region box R of non-text data i Used to calculate the similarity score:

[0128]

[0129] Among them, L m (T di ) represents the word segmentation result in the text data, R j represents the candidate region in non-text data, α ij represents the attention weight between text data and non-text data, Sim(L m (T di ), R j ) represents the similarity score between the text data word and the non-text data candidate region, A s (T d , N d ) represents the correlation score between text data and non-text data;

[0130] S43. Based on the correlation score A s (T d , N d ) Information fusion is performed on multimodal data to generate the final multimodal feature matrix M r :

[0131]

[0132] Among them, β ij represents the weight coefficient after fusion, represents the fusion operation of text data words with image and chart features, M r Represents the final generated multimodal representation.

[0133] In this implementation, step S5 includes:

[0134] S51, the fused multimodal feature matrix M r The input is fed into the pre-trained improved BERT model to perform deep semantic analysis. The improved BERT model jointly encodes the embedded representations of text and non-text data to generate a multimodal semantic vector S v ;

[0135] S52, through the attention mechanism in the multimodal semantic vector M r The improved BERT model calculates the attention weights of each word and non-text features to identify key entities E in multimodal data. k :

[0136]

[0137] Among them, S vi represents the i-th word or feature in the multimodal semantic vector, T di represents the corresponding text words, α i represents the attention weight;

[0138] S53, deep semantic relationship extraction module based on the improved BERT model, calculates the relationship R between entities in the multimodal semantic vector s (E k ), and identify the event Ev in the multimodal semantic vector through the event detection module s .

[0139] In this implementation, step S6 includes:

[0140] S61. Based on the results of deep semantic analysis, the multimodal semantic vector S v Extract text entity node E t and non-text entity node E n , construct the initial node set N of the knowledge graph g :

[0141] Text entity node E t Represents people, places, and events extracted from text data;

[0142] Non-text entity node E n Represent objects and data points extracted from images and charts as non-text data;

[0143] S62: Generate edges L between nodes based on the semantic relationship between text data and non-text data g ,The edges include the edges representing the relationship between the text description and the object in the image and the edges representing the relationship between the graph data and the text description;

[0144] S63. Weights ω of nodes and edges based on graph structure learning model n and ω l Optimize and generate the optimized knowledge graph K g .

[0145] In this implementation, step S63 includes:

[0146] S631, node weights ω in the initial knowledge graph based on graph structure learning model n and edge weight ω l Initialize, node weight ω n Initialized to the node degree and edge weight ω l Initialized to the association strength between nodes;

[0147] S632: Optimize node weights and edge weights based on the gradient descent algorithm, and iteratively update the weight values ​​of nodes and edges and

[0148]

[0149]

[0150] Among them, η2 is the learning rate, and Respectively represent the partial derivatives of the loss function L with respect to the node weight and edge weight;

[0151] S633, the loss function L is optimized by minimizing the error between nodes and edges. The purpose of the optimization is to improve the accuracy and relevance of the knowledge graph by minimizing L:

[0152]

[0153] Among them, Sim(E i , E j ) represents node E i With node E j The similarity between them, L represents the total loss between nodes and edges;

[0154] S634, in the optimization iteration process, according to the optimal node weight and edge weights Generate the final optimized knowledge graph K g :

[0155]

[0156] Among them, K g Represents the final optimized knowledge graph, including the optimized node set N g , edge set L g And the node weight and edge weights

[0157] Embodiment 1:

[0158] In the daily equipment maintenance of a smart manufacturing factory, the technical documents used by the factory contain data in multiple modes, such as text descriptions, equipment photos, parameter tables, and operation diagrams. The maintenance manual plays an important role in equipment maintenance and operation training, but the document data is huge and the format is complex. The traditional manual processing method is inefficient and easy to miss key information, affecting the efficient operation and maintenance of the equipment.

[0159] In August 2023, when the factory was performing annual maintenance on a large precision CNC machine tool, the maintenance team needed to refer to the operating manual of the equipment to check various parameters. The equipment manual contained 25 pages of text descriptions, 15 equipment pictures, 10 operating diagrams, and 20 equipment parameter tables. The factory maintenance personnel processed this document using traditional methods, which took about 5 days. During this period, they found that multiple charts and text information were incorrectly associated, resulting in a decrease in maintenance efficiency. To solve this problem, the factory decided to apply the method of the present invention to improve efficiency and accuracy.

[0160] In early September 2023, the factory input the maintenance manual of the equipment into the system of the present invention. The system first decomposes the multimodal data in the document and processes the text, image and chart data separately. In the text processing stage, the system performs word segmentation, part-of-speech tagging and syntactic analysis on the equipment operating instructions in the manual, and identifies the key equipment names, operating procedures and maintenance requirements. In the image processing stage, the system uses convolutional neural networks to extract visual features of 15 equipment images, identifies the key components of the machine tool, and automatically pairs the components in the image with the operating steps in the text through association analysis with the text description. In the chart processing stage, the system performs structured recognition on 20 equipment parameter tables, extracts key parameters (temperature, pressure and operating speed) and associates them with the operating procedures in the text.

[0161] To verify the performance of the system, we compared the method of the present invention with the traditional manual processing method. The following are the specific comparison data:

[0162] Table 1 Comparative data of traditional method and method of the present invention

[0163]

[0164]

[0165] As can be seen from Table 1 above, the traditional method takes about 40 hours to process the equipment manual, including manual page-by-page analysis of the association between text, images and charts, while the system of the present invention can automatically complete the processing in only 3 hours, increasing the processing speed by 13 times. Through deep semantic analysis and multimodal data fusion, the system of the present invention can accurately extract key knowledge in the maintenance manual. The traditional method is prone to errors in the association annotation of images and texts, with an error rate of 8%, while the automatic annotation of the system has an error rate of only 1%. In the traditional method, factory technicians need to manually match the equipment images with the operating steps, which is prone to mismatching. The system of the present invention can automatically complete the matching through visual feature extraction and text association analysis, and the accuracy rate reaches 99%. The 20 parameter tables contained in the equipment manual need to be manually input in the traditional method, while the system of the present invention can automatically extract parameter data and match it with the text description through structured recognition, with a recognition accuracy rate of 98.5%. In the traditional method, knowledge extraction is limited to pure text information and cannot effectively process data in images and charts. The system of the present invention can integrate text, image and chart information to generate a complete knowledge graph, ensuring the integrity of multimodal information with a completeness of 99%. In traditional methods, manual intervention requires a lot of time, especially in the association annotation of images and texts and the input of parameter tables, which takes 15 hours of manual time. The system of the present invention has a high degree of automation, and only 0.5 hours of manual review is required in the whole process, which greatly reduces the manpower input.

[0166] In an actual operation in September 2023, maintenance personnel used the system of the present invention to process the maintenance manual of a precision CNC machine tool. The system first identified and extracted the key information of "equipment model: NC-5000" and "key components: spindle, tool holder" in the text. Subsequently, the system used the image processing module to identify the equipment photos related to these components in the maintenance manual, and automatically matched them with the operating steps. The system associated and labeled "spindle maintenance" with the corresponding equipment image, and further analyzed the "spindle speed range" and "operating temperature" in the parameter table, and finally formed a global information display of the equipment in the knowledge graph.

[0167] During this process, the system also automatically generates a maintenance report, which includes the key components of the equipment, operating steps, relevant parameters, and the associated results of images and chart information. The report is then submitted to the maintenance team as a reference document for equipment maintenance operations.

[0168] Through this application, the factory's maintenance efficiency has been significantly improved. The information structure provided by the maintenance personnel feedback system is clear and the operation steps are clear, which reduces the previous workload of manual review and association of images and parameter tables.

[0169] The present invention fuses text data with non-text data, proposes a multimodal fusion method based on an improved BERT model, utilizes a feature extraction model to extract features from text and non-text data respectively, and then fuses the features through an adaptive weight mechanism to generate a multimodal feature matrix. This not only improves the accuracy of identifying the correlation between text and image and chart data, but also enhances the system's ability to understand complex documents, enabling the system to extract more accurate and comprehensive semantic information from multimodal data, overcoming the limitations of single-modal processing in the prior art.

[0170] In the technical solution of the present invention, the node weights and edge weights of the knowledge graph are adaptively optimized based on the graph structure learning model. By introducing the gradient descent algorithm and optimizing the calculation of the loss function, the weight distribution of the nodes and edges in the knowledge graph can be dynamically adjusted. The optimized knowledge graph can more accurately express the semantic relationship between text and non-text data. Compared with the existing knowledge graph construction technology, the present invention can not only process pure text data, but also effectively combine image and chart data, so that the constructed knowledge graph has higher expressive power and accuracy, ensuring that the key knowledge in complex documents can be fully represented.

[0171] The present invention significantly improves the efficiency of automatic processing of complex documents by introducing an improved BERT model and a self-supervised learning mechanism, combined with multimodal fusion technology. Unlike traditional technologies that require a lot of manual intervention, the present invention can achieve automatic knowledge extraction and structured processing through deep semantic analysis, entity recognition, and relationship extraction steps under unsupervised or weak supervision, reducing the complexity of manual operations and greatly reducing the time cost of processing multimodal documents. It improves the overall processing efficiency of the system and solves the problem of low efficiency of existing technologies in large-scale, multimodal document processing.

[0172] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multimodal document structured processing and knowledge extraction method based on a large language model, characterized in that: The steps include: S1. receiving an input multimodal document, wherein the multimodal document comprises at least one text data and at least one non-text data; S2, preprocessing the text data in the multimodal document, wherein the preprocessing includes word segmentation, part-of-speech tagging, syntactic analysis and entity recognition; S3, performing feature extraction on the non-text data in the multimodal document; S4, performing multimodal data fusion on the preprocessed text data and the feature-extracted non-text data; S5. performing deep semantic analysis on the fused multimodal data through a pre-trained improved BERT model; S6. Based on the result of the deep semantic analysis, automatically construct a knowledge graph from the extracted information; S7. Output the knowledge graph data into a format that can be analyzed or applied.

2. According to claim 1, a multimodal document structured processing and knowledge extraction method based on a large language model is characterized in that: The step S1 comprises: S11, receiving a multimodal document input signal provided by multiple data sources, wherein the data sources include a text data source and a non-text data source, wherein the text data source provides text data containing natural language content, and the non-text data source provides images, charts, or other types of non-text data: Among them, D m represents the received multimodal document input dataset, T di Represents the i-th text data unit, N dj represents the jth non-text data unit, N t and N n Represents the number of text data and non-text data respectively, Represents the parallel reception and combination operations of multimodal data; S12, text data T d and non-text data N d Perform synchronous input operations so that text data and non-text data are simultaneously imported into the processing system; S13, input text data T d and non-text data N d Perform preliminary data type identification based on metadata identifier M t Verify the text data type through the image feature matrix I f Validate non-text data types; S14, according to the recognition result, the text data T d and non-text data N d Classification and storage in text data cache unit C t and non-text data cache unit C n middle.

3. The multimodal document structured processing and knowledge extraction method based on a large language model according to claim 1, characterized in that: The step S2 comprises: S21, using the vocabulary matching algorithm based on the contextual attention mechanism to receive the text data T d Perform word segmentation processing and execute word segmentation operation: Among them, T d Represents the input text data, T di Represents the i-th word in the text data, L m (T d ) is the result of word segmentation processing, α i and β i Respectively represent the global dictionary matching weight and context similarity weight, W(T di ) represents the matching score of the entries in the dictionary, C(T di ) represents word T di The context word set, Sim(T di ,T j ) represents word T di With context word T j similarity; S22. Use the part-of-speech tagging model based on conditional random fields to perform part-of-speech tagging on the text data after word segmentation. The part-of-speech tagging process is as follows: Among them, T di represents the i-th word, P di represents the corresponding part-of-speech tag, ψ(T di ,P di ) represents word T di Its part of speech P di The compatibility function between di-1 ,P di ) represents the probability of part-of-speech transfer, P s (T d ) represents the part-of-speech tag set of the text data after word segmentation; S23, perform syntactic analysis on the annotated text data, and use the dependency tree syntactic analysis model to construct a grammatical structure tree of the text. The specific process is as follows: Among them, T di and T dj Represents two words in the text, δ(T di ,T dj ) represents the distance between words, τ(P di ,P dj ) represents the dependency weight between parts of speech, Dep(T di ,T dj ) represents the strength of the dependency relationship between words, S t (T d ) represents the syntax structure tree of text data; S24. Use an entity recognition model based on bidirectional long short-term memory and conditional random field to perform entity recognition on the text data after syntactic analysis, and perform entity recognition: Among them, T di represents the i-th word in the text, E di represents the corresponding entity label, λ(T di ,E di ) represents the compatibility score between the word and the entity tag, μ(E di-1 ,E di ) represents the transfer score between entity labels, γ represents the fusion coefficient between multimodal data, Att(T di ,F j ) represents the text word T di and non-text data features F j The attention weight, E r (T d ) represents the entity recognition results in text data.

4. The method for multimodal document structure processing and knowledge extraction based on a large language model according to claim 1, characterized in that: The step S3 comprises: S31, image data I in multimodal document d Perform visual feature extraction and use convolutional neural network to analyze the spatial features in the image data to extract the image feature vector V i : Among them, I d Represents the input image data, I lk represents the lth layer and the kth feature point in the image, w lk represents the weight parameter, f(I lk ) represents the nonlinear activation function of the image feature points, b lk Indicates the bias value, V i A visual feature vector representing an image; S32, use the model based on the region proposal network to detect the target object in the image data and generate the candidate region box R i , and perform target classification on each region box: Among them, R pi represents the i-th candidate region, O i represents the true target area, P represents the number of candidate areas, φ(R pi ) is the confidence score of the region proposal, IoU(R pi ,O i ) represents the intersection-over-union ratio between the candidate region and the true target region; S33, for the chart data G in the multimodal document d Perform structural recognition and extract key structural information S in the graph g , including the relationship between axes, data points, labels and lines: Among them, G d Represents the input chart data, G dj represents the jth data element in the graph, L j Represents the label associated with the data element, γ j Represents the weight parameter, Rel(G dj ,L j ) represents the strength of the association between the chart data element and its label, S g Represents the structure recognition result of the graph.

5. The multimodal document structured processing and knowledge extraction method based on a large language model according to claim 1, characterized in that: The step S4 comprises: S41, the pre-processed text data T d The eigenvector P s (T d ) and the non-text data N after feature extraction v Perform feature vector fusion and construct a multimodal feature matrix M f : Among them, P s (T d ) represents the part-of-speech tagging feature vector of text data, V i Represents the visual feature vector of image data, S g represents the structural features of the graph data, λ1 and λ2 represent the fusion weights of text and non-text data respectively, and M f Represents the fused multimodal feature matrix; S42. Calculate the correlation score A between text data and non-text data based on the attention mechanism s (T d ,N d ), where the word segmentation result of the text data is L m (T d ) and the candidate region box R of non-text data i Used to calculate the similarity score: Among them, L m (T di ) represents the word segmentation result in the text data, R j represents the candidate region in non-text data, α ij represents the attention weight between text data and non-text data, Sim(L m (T di ),R j ) represents the similarity score between the text data word and the non-text data candidate region, A s (T d ,N d ) represents the correlation score between text data and non-text data; S43. Based on the correlation score A s (T d ,N d ) Information fusion is performed on multimodal data to generate the final multimodal feature matrix M r : Among them, β ij represents the weight coefficient after fusion, represents the fusion operation of text data words and image and chart features, M r Represents the final generated multimodal representation.

6. The multimodal document structured processing and knowledge extraction method based on a large language model according to claim 1, characterized in that: The step S5 comprises: S51, the fused multimodal feature matrix M r The input is fed into the pre-trained improved BERT model to perform deep semantic analysis. The improved BERT model jointly encodes the embedded representations of text and non-text data to generate a multimodal semantic vector S v ; S52, through the attention mechanism in the multimodal semantic vector M r The improved BERT model calculates the attention weights of each word and non-text features to identify key entities E in multimodal data. k : Among them, S vi represents the i-th word or feature in the multimodal semantic vector, T di represents the corresponding text words, α i represents the attention weight; S53, deep semantic relationship extraction module based on the improved BERT model, calculates the relationship R between entities in the multimodal semantic vector s (E k ), and identify the event Ev in the multimodal semantic vector through the event detection module s .

7. The multimodal document structured processing and knowledge extraction method based on a large language model according to claim 1, characterized in that: The step S6 comprises: S61. Based on the results of deep semantic analysis, the multimodal semantic vector S v Extract text entity node E t and non-text entity node E n , construct the initial node set N of the knowledge graph g : Text entity node E t Represents people, places, and events extracted from text data; Non-text entity node E n Represent objects and data points extracted from images and charts as non-text data; S62: Generate edges L between nodes based on the semantic relationship between text data and non-text data g , the edges include edges representing the relationship between the text description and the object in the image and edges representing the relationship between the chart data and the text description; S63. Weights ω of nodes and edges based on graph structure learning model n and ω l Optimize and generate the optimized knowledge graph K g .

8. The multimodal document structured processing and knowledge extraction method based on a large language model according to claim 1, characterized in that: The step S63 comprises: S631, node weights ω in the initial knowledge graph based on graph structure learning model n and edge weight ω l Initialize, node weight ω n Initialized to the node degree and edge weight ω l Initialized to the association strength between nodes; S632: Optimize node weights and edge weights based on the gradient descent algorithm, and iteratively update the weight values ​​of nodes and edges and Among them, η2 is the learning rate, and Respectively represent the partial derivatives of the loss function L with respect to the node weight and edge weight; S633, the loss function L is optimized by minimizing the error between nodes and edges. The purpose of the optimization is to improve the accuracy and relevance of the knowledge graph by minimizing L: Among them, Sim(E i ,E j ) represents node E i With node E j The similarity between them, L represents the total loss between nodes and edges; S634, in the optimization iteration process, according to the optimal node weight and edge weights Generate the final optimized knowledge graph K g : Among them, K g Represents the final optimized knowledge graph, including the optimized node set N g , edge set L g And the node weight and edge weights

Images