A multi-modal semantic enhancement-based unstructured data processing method and system

Abstract

The application discloses a kind of based on multimodal semantic enhancement's unstructured data processing method and system, the method includes the following steps: S1, respectively to the visual feature extraction of input unstructured data, text recognition and semantic processing, symbol or structure information standardization and mapping, generate unified standardized multimodal feature representation;S2, based on improved language model, semantic coding and similarity optimization are carried out to multimodal feature, and semantic representation and graph structure representation are cooperatively optimized by joint loss function;S3, the interactive alignment between semantic feature and structure feature is realized using cross-modal attention mechanism, and similarity calculation and matching analysis are carried out to embedding representation from multiple feature dimensions;S4, the recognition result and similarity analysis result are visualized and output, and data retrieval, anomaly detection, consistency check and semantic level comparison function are expanded to realize, then introduce user feedback information to update and optimize model parameter.

Description

Technical Field

[0001] This invention belongs to the field of computer data processing system technology, specifically relating to a method and system for processing unstructured data based on multimodal semantic enhancement. Background Technology

[0002] With the continuous improvement of informatization and digitalization in fields such as electronic information, mechanical manufacturing and construction engineering, various types of design and engineering data are widely used in system design, verification and production processes. This type of data usually exists in an unstructured form containing symbols, text and structural relationships, carrying rich semantic information and complex logical connections, and is an important data foundation for engineering analysis and intelligent decision-making.

[0003] Existing automatic parsing methods for this type of unstructured data mostly focus on local feature calculation or rule matching based on perceptual data. For example, they achieve element recognition through visual data feature representation generation or template-based matching. While the implementation path is relatively clear, they have significant shortcomings in semantic association modeling, structural relationship expression, and cross-modal collaborative analysis. Specifically, on the one hand, existing methods have limited ability to model the association between textual semantic information and symbolic or structural data, making it difficult to achieve a unified understanding of the data content from an overall semantic level. On the other hand, unstructured data often contains complex structural relationships and logical constraints, and traditional methods lack effective means to model the topological relationships and dependencies between data units, resulting in parsing results remaining at the surface recognition level. Furthermore, as the scale of data expands and application scenarios diversify, single modality or single computational models gradually become limited in terms of generalization ability and adaptability, making it difficult to meet the high-precision and high-reliability processing requirements in complex scenarios.

[0004] Therefore, there is an urgent need for a data processing method for unstructured data that integrates multimodal feature parsing, semantic modeling, and structured representation learning to achieve deep semantic understanding and structural correlation analysis of data content. This would improve the accuracy, robustness, and versatility of unstructured data processing, and provide reliable data processing support for engineering design automation, intelligent manufacturing, and information systems. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes an unstructured data processing system based on multimodal semantic enhancement. This system integrates visual data features, textual semantics, and graph structure information to achieve multimodal collaborative recognition and semantic-level understanding of complex unstructured data. It solves the problems of insufficient semantic understanding, limited structural modeling capabilities, and inadequate multimodal information fusion in existing technologies, and has promising prospects for widespread application.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for processing unstructured data based on multimodal semantic enhancement includes the following steps:

[0008] S1. Visual feature extraction, text recognition and semantic processing, symbol or structural information standardization and mapping are performed on the input unstructured data respectively, and the association and fusion of different modal features are realized through a multimodal alignment mechanism to generate a unified standardized multimodal feature representation;

[0009] S2. Based on the improved language model, semantic encoding and similarity optimization of multimodal features are performed, and a graph structure representation reflecting the inherent relationship of data is constructed. The graph neural network is used to aggregate and update the node features in the graph, and the semantic representation and graph structure representation are jointly optimized through a joint loss function.

[0010] S3. A cross-modal attention mechanism is adopted to achieve interactive alignment between semantic features and structural features. A unified embedding representation is generated through a weighted fusion strategy, and similarity calculation and matching analysis are performed on the embedding representation from multiple feature dimensions.

[0011] S4. Visualize the recognition results and similarity analysis results, and extend the functionality to include data retrieval, anomaly detection, consistency verification, and semantic-level comparison. Then, incorporate user feedback to update and optimize the model parameters to support cross-scenario and cross-domain unstructured data processing applications.

[0012] Preferably, the specific process of step S1 is as follows:

[0013] S11. Visual Data Feature Extraction: A convolutional neural network is used to obtain a symbolic feature map. The calculation formula is as follows: ,in, Symbolic feature map; For convolutional neural network operations; For the input visual data, H, W, and C are the height, width, and number of channels of the visual data, respectively, and h, w, and d are the height, width, and channel dimension of the feature map, respectively.

[0014] S12. Text Recognition and Processing: Text information is extracted using OCR technology, and after word segmentation, noise reduction, and normalization, the text is embedded using Word2Vec model, FastText model, or large language model embedding function.

[0015] S13. Symbol Standardization and Mapping: Construct a standard symbol library for common components, and calculate the cosine similarity between the features of the symbol to be identified and the standard symbols. The formula for calculating the cosine similarity is: ,in, The cosine similarity between the features of the symbol to be identified and the standard symbol; Features of the symbol to be identified; Standard symbols; It is the vector norm;

[0016] If the cosine similarity is greater than the threshold θ, the symbol to be identified is determined to belong to the standard element category represented by the standard symbol, and is bidirectionally bound to the text label;

[0017] S14. Multimodal Alignment and Feature Fusion: A cross-modal alignment method is adopted, using a linear projection function to map visual data features and text features to the same latent space, thereby achieving a unified representation of visual data features and text features. The calculation formula is as follows: , ,in, Visual data features mapped to the same latent space; is the linear projection matrix of the visual data modal features; The original visual data feature matrix output by the visual data feature extraction network; Text features mapped to the same latent space; is the linear projection matrix of the text modal features; The text feature matrix extracted by the text semantic encoding model;

[0018] A contrastive learning loss function is used to optimize cross-modal similarity, minimizing the distance between visual data and text of the same symbol in the embedding space, and maintaining the distinguishability between representations of different symbols. The formula for calculating the contrastive learning loss function is as follows: ,in, To learn the loss function by comparison; The cosine similarity function; For temperature parameters; Index representing a negative sample; Let represent the text feature vector of the k-th negative sample.

[0019] Preferably, in step S13, the threshold θ = 0.85; in step S14, the temperature parameter... =0.07.

[0020] Preferably, the specific process of step S2 is as follows:

[0021] S21. Semantic Modeling: An improved large language model is used to model the preprocessed text input. Encoding is performed; among them, the improved large language model uses the BERT model, DeBERTaV3 model, or RoBERTa model. The input consists of the first n preprocessed texts; the encoding calculation formula is: ,in, For the contextual representation of symbolic text; Embedded in text; This means that the text embedding E(T) of the input text T is input into the Transformer model, and the output is a feature representation containing contextual semantic information, which is used for subsequent semantic modeling and similarity calculation; n is the number of text inputs; For semantic vector dimensions;

[0022] By constraining similar symbols to text in the embedding space using the binary cross-entropy loss function, different categories of symbols are kept distinguishable. The formula for calculating the binary cross-entropy loss function is as follows: ,in, is the binary cross-entropy loss function; i is the index of the training sample pair; The index of the other node paired with sample i, where the node is a symbol or text; This represents the total number of all symbolic sample pairs during training. Indicate whether the symbol pair is of the same type; For the Sigmoid function; Let be the embedded feature vector of the i-th node; The embedded feature vector of the node that forms a sample pair with node i;

[0023] S22. Graph Structure Modeling: Each symbol or text element is treated as a node, and the spatial or semantic relationships between nodes are used as edges to construct a graph structure. Edge weights are calculated using spatial distance or semantic similarity, with the following formula: ,in, The edge weights; For balance parameters; The spatial distance between nodes; For node semantic similarity;

[0024] S23. Node Feature Aggregation: Node feature aggregation is performed using a graph convolutional neural network. The calculation formula is as follows: ,in, Features of the (l+1)th layer nodes; It is a degree matrix; For adjacency matrices with self-loops, This is the original adjacency matrix; Features of the l-th layer nodes; Here is the weight matrix for the l-th layer;

[0025] S24. Joint Optimization: Joint training is performed by weighting the semantic loss and the graph structure embedding loss to maintain semantic consistency and structural rationality. The calculation formula for joint training is as follows: ,in, The objective function is used to jointly optimize the constraints on semantic consistency and graph structure rationality. For graph structure embedding loss; and These are the weighting coefficients.

[0026] Preferably, the specific process of step S3 is as follows:

[0027] S31. Cross-modal fusion: A cross-modal attention mechanism is used to achieve interactive alignment between semantic features and graph structure features. The calculation formula for the cross-modal attention mechanism is as follows: , , , ,in, For cross-modal attention output; For query matrix; The key matrix; It is a value matrix; This is the normalization function; The dimension of the key vector; for Transpose of; Semantic features; The trainable weight matrix for the query vector; Graph structural features; is the trainable weight matrix for the key vectors; The trainable weight matrix is ​​a value vector;

[0028] S32. Unified Representation Learning: A unified embedding representation is obtained through a weighted fusion strategy. The calculation formula for the weighted fusion strategy is as follows: ,in, β is a unified embedding representation of unstructured data symbols; β is a fusion weighting factor. For cross-modal fusion features; For feature splicing operations;

[0029] S33. Multi-scale similarity calculation: Cosine similarity is used to measure the semantic and structural matching degree of individual symbols. The formula for calculating symbol-level similarity is: ,in, Symbol-level similarity; Let i be the fused feature vector of symbol i; Let be the fusion feature vector of symbol j;

[0030] The subgraph matching method maps a subset of symbols to a subgraph, and calculates local structural similarity using graph edit distance. The formula for calculating local structural similarity is as follows: ,in, For local structural level similarity; Edit distance for the image; It is a local subgraph consisting of the location of symbol i and its neighboring symbols; This is the corresponding local subgraph consisting of symbol j and its neighboring symbols;

[0031] Combining symbol-level similarity and local structural-level similarity, a weighted approach is used to obtain the global unstructured data-level similarity. The formula for calculating the global unstructured data-level similarity is as follows: ,in, Similarity at the global unstructured data level; These are weighting coefficients used to balance symbol precision and structural integrity; It is a function of average value;

[0032] S34. Loss Function Optimization: Combine binary cross-entropy loss and contrastive loss for model training to maintain the discriminative power of different class embeddings. The calculation formula is as follows: , ,in, To optimize the overall objective function; and These are the weighting coefficients; For binary cross-entropy loss; Compare the losses; This is the boundary threshold.

[0033] Preferably, the specific process of step S4 is as follows:

[0034] S41. Results Visualization: The recognition results are presented intuitively through symbol annotation, relationship visualization, and similarity heatmap.

[0035] S42. Functional Expansion: Implement data retrieval based on global unstructured data level similarity, error detection based on graph structure comparison, standardized proofreading by comparison with industry standard symbol libraries, and semantic level comparison across languages ​​and standards;

[0036] S43. Feedback Optimization: Update model parameters based on user annotations. The formula for calculating the updated model parameters is: ,in, For the first Model parameters after the next iteration; For the first The model parameters for this iteration; η is the learning rate; For parameters Below, the model uses the original training dataset. loss function The calculated gradient; These are the original training samples; The original training labels or supervision information; μ is the feedback weight; Gradient updates are provided to correct user behavior.

[0037] S44. Cross-domain applications: Provide unstructured data recognition services for electronic design automation, building information modeling, and industrial quality inspection.

[0038] A system for processing unstructured data based on multimodal semantic enhancement, used to implement the aforementioned method for processing unstructured data based on multimodal semantic enhancement, includes:

[0039] The multimodal feature extraction and fusion module is used to extract visual features, recognize and process text, standardize and map symbolic or structural information from the input unstructured data, and realize the association and fusion of different modal features through a multimodal alignment mechanism to generate a unified standardized multimodal feature representation.

[0040] The semantic encoding and structural modeling module is used to perform semantic encoding and similarity optimization on the multimodal features based on the improved language model, and to construct a graph structure representation that reflects the inherent relationship between the data. The graph neural network is used to aggregate and update the node features in the graph, and the semantic representation and graph structure representation are jointly optimized through a joint loss function.

[0041] The cross-modal alignment and embedding generation module is used to achieve interactive alignment between semantic features and structural features using a cross-modal attention mechanism, generate a unified embedding representation through a weighted fusion strategy, and perform similarity calculation and matching analysis on the embedding representation from multiple feature dimensions.

[0042] The results output and adaptive optimization module is used to visualize the recognition results and similarity analysis results, and to extend the functions of data retrieval, anomaly detection, consistency verification and semantic comparison. At the same time, user feedback information is introduced to update and optimize model parameters to support cross-scenario and cross-domain unstructured data processing applications.

[0043] By adopting the above technical solution, the present invention has the following beneficial effects:

[0044] 1. This invention accurately captures the correspondence between text and symbols through OCR text extraction, pre-trained language model encoding, and token-level semantic enhancement, thereby improving the accuracy of semantic matching in professional contexts and enhancing the semantic understanding of unstructured data as well as the accuracy of data parsing and recognition.

[0045] 2. This invention effectively captures the topological relationships and logical dependencies between symbols through graph structure abstraction and graph neural network modeling, achieving a deep understanding of unstructured data and accurate structural modeling.

[0046] 3. This invention employs a cross-modal attention mechanism and a weighted fusion strategy to achieve efficient collaboration of visual data, text, and graph structural features, thereby enhancing the adaptability of complex unstructured data, fully integrating multimodal data, and improving the accuracy of unstructured data parsing.

[0047] 4. This invention supports multiple functions such as data retrieval, error detection, and standardized verification, and can be applied across multiple fields such as electronics, construction, and machinery, significantly improving the intelligence level of related industries and having a wide range of application scenarios. Attached Figure Description

[0048] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0050] like Figure 1 As shown, an unstructured data processing method based on multimodal semantic enhancement includes the following steps:

[0051] S1. Visual feature extraction, text recognition and semantic processing, symbol or structural information standardization and mapping are performed on the input unstructured data respectively, and the association and fusion of different modal features are realized through a multimodal alignment mechanism to generate a unified standardized multimodal feature representation;

[0052] The specific process of step S1 is as follows:

[0053] S11. Visual Data Feature Extraction: A convolutional neural network is used to obtain a symbolic feature map. The calculation formula is as follows: ,in, Symbolic feature map; For convolutional neural network operations; For the input visual data, H, W, and C are the height, width, and number of channels of the visual data, respectively, and h, w, and d are the height, width, and channel dimension of the feature map, respectively.

[0054] S12. Text Recognition and Processing: Text information is extracted using OCR technology, and after word segmentation, noise reduction, and normalization, the text is embedded using Word2Vec model, FastText model, or large language model embedding function.

[0055] S13. Symbol Standardization and Mapping: Construct a standard symbol library for common components, and calculate the cosine similarity between the features of the symbol to be identified and the standard symbols. The formula for calculating the cosine similarity is: ,in, The cosine similarity between the features of the symbol to be identified and the standard symbol; Features of the symbol to be identified; Standard symbols; It is the vector norm;

[0056] If the cosine similarity is greater than the threshold θ, the symbol to be identified is determined to belong to the standard element category represented by the standard symbol, and is bidirectionally bound to the text label; in step S13, the threshold θ = 0.85;

[0057] S14. Multimodal Alignment and Feature Fusion: A cross-modal alignment method is adopted, using a linear projection function to map visual data features and text features to the same latent space, thereby achieving a unified representation of visual data features and text features. The calculation formula is as follows: , ,in, Visual data features mapped to the same latent space; is the linear projection matrix of the visual data modal features; The original visual data feature matrix output by the visual data feature extraction network; Text features mapped to the same latent space; is the linear projection matrix of the text modal features; The text feature matrix extracted by the text semantic encoding model;

[0058] A contrastive learning loss function is used to optimize cross-modal similarity, minimizing the distance between visual data and text of the same symbol in the embedding space, and maintaining the distinguishability between representations of different symbols. The formula for calculating the contrastive learning loss function is as follows: ,in, To learn the loss function by comparison; The cosine similarity function; For temperature parameters; Index representing a negative sample; Let this represent the text feature vector of the k-th negative sample;

[0059] In step S14, the temperature parameter =0.07;

[0060] S2. Based on the improved language model, semantic encoding and similarity optimization of multimodal features are performed, and a graph structure representation reflecting the inherent relationship of data is constructed. The graph neural network is used to aggregate and update the node features in the graph, and the semantic representation and graph structure representation are jointly optimized through a joint loss function.

[0061] The specific process of step S2 is as follows:

[0062] S21. Semantic Modeling: An improved large language model is used to model the preprocessed text input. Encoding is performed; among them, the improved large language model uses the BERT model, DeBERTaV3 model, or RoBERTa model. The input consists of the first n preprocessed texts; the encoding calculation formula is: ,in, For the contextual representation of symbolic text; Embedded in text; This means that the text embedding E(T) of the input text T is input into the Transformer model, and the output is a feature representation containing contextual semantic information, which is used for subsequent semantic modeling and similarity calculation; n is the number of text inputs; For semantic vector dimensions;

[0063] By constraining similar symbols to text in the embedding space using the binary cross-entropy loss function, different categories of symbols are kept distinguishable. The formula for calculating the binary cross-entropy loss function is as follows: ,in, is the binary cross-entropy loss function; i is the index of the training sample pair; The index of the other node paired with sample i, where the node is a symbol or text; This represents the total number of all symbolic sample pairs during training. Indicate whether the symbol pair is of the same type; For the Sigmoid function; Let be the embedded feature vector of the i-th node; The embedded feature vector of the node that forms a sample pair with node i;

[0064] S22. Graph Structure Modeling: Each symbol or text element is treated as a node, and the spatial or semantic relationships between nodes are used as edges to construct a graph structure. Edge weights are calculated using spatial distance or semantic similarity, with the following formula: ,in, The edge weights; For balance parameters; The spatial distance between nodes; For node semantic similarity;

[0065] S23. Node Feature Aggregation: Node feature aggregation is performed using a graph convolutional neural network. The calculation formula is as follows: ,in, Features of the (l+1)th layer nodes; It is a degree matrix; For adjacency matrices with self-loops, This is the original adjacency matrix; Features of the l-th layer nodes; Here is the weight matrix for the l-th layer;

[0066] S24. Joint Optimization: Joint training is performed by weighting the semantic loss and the graph structure embedding loss to maintain semantic consistency and structural rationality. The calculation formula for joint training is as follows: ,in, The objective function is used to jointly optimize the constraints on semantic consistency and graph structure rationality. For graph structure embedding loss; and These are the weighting coefficients;

[0067] S3. A cross-modal attention mechanism is adopted to achieve interactive alignment between semantic features and structural features. A unified embedding representation is generated through a weighted fusion strategy, and similarity calculation and matching analysis are performed on the embedding representation from multiple feature dimensions.

[0068] The specific process of step S3 is as follows:

[0069] S31. Cross-modal fusion: A cross-modal attention mechanism is used to achieve interactive alignment between semantic features and graph structure features. The calculation formula for the cross-modal attention mechanism is as follows: , , , ,in, For cross-modal attention output; For query matrix; The key matrix; It is a value matrix; This is the normalization function; The dimension of the key vector; for Transpose of; Semantic features; The trainable weight matrix for the query vector; Graph structural features; is the trainable weight matrix for the key vectors; The trainable weight matrix is ​​a value vector;

[0070] S32. Unified Representation Learning: A unified embedding representation is obtained through a weighted fusion strategy. The calculation formula for the weighted fusion strategy is as follows: ,in, β is a unified embedding representation of unstructured data symbols; β is a fusion weighting factor. For cross-modal fusion features; For feature splicing operations;

[0071] S33. Multi-scale similarity calculation: Cosine similarity is used to measure the semantic and structural matching degree of individual symbols. The formula for calculating symbol-level similarity is: ,in, Symbol-level similarity; Let i be the fused feature vector of symbol i; Let be the fusion feature vector of symbol j;

[0072] The subgraph matching method maps a subset of symbols to a subgraph, and calculates local structural similarity using graph edit distance. The formula for calculating local structural similarity is as follows: ,in, For local structural level similarity; Edit distance for the image; It is a local subgraph consisting of the location of symbol i and its neighboring symbols; This is the corresponding local subgraph consisting of symbol j and its neighboring symbols;

[0073] Combining symbol-level similarity and local structural-level similarity, a weighted approach is used to obtain the global unstructured data-level similarity. The formula for calculating the global unstructured data-level similarity is as follows: ,in, Similarity at the global unstructured data level; These are weighting coefficients used to balance symbol precision and structural integrity; It is a function of average value;

[0074] S34. Loss Function Optimization: Combine binary cross-entropy loss and contrastive loss for model training to maintain the discriminative power of different class embeddings. The calculation formula is as follows: , ,in, To optimize the overall objective function; and These are the weighting coefficients; For binary cross-entropy loss; Compare the losses; For boundary thresholds;

[0075] S4. Visualize the recognition results and similarity analysis results, and extend the functions to include data retrieval, anomaly detection, consistency verification and semantic comparison. Then, introduce user feedback to update and optimize the model parameters to support cross-scenario and cross-domain unstructured data processing applications.

[0076] The specific process of step S4 is as follows:

[0077] S41. Results Visualization: The recognition results are presented intuitively through symbol annotation, relationship visualization, and similarity heatmap.

[0078] S42. Functional Expansion: Implement data retrieval based on global unstructured data level similarity, error detection based on graph structure comparison, standardized proofreading by comparison with industry standard symbol libraries, and semantic level comparison across languages ​​and standards;

[0079] S43. Feedback Optimization: Update model parameters based on user annotations. The formula for calculating the updated model parameters is: ,in, For the first Model parameters after the next iteration; For the first The model parameters for this iteration; η is the learning rate; For parameters Below, the model uses the original training dataset. loss function The calculated gradient; These are the original training samples; The original training labels or supervision information; μ is the feedback weight; Gradient updates are provided to correct user behavior.

[0080] S44. Cross-domain applications: Provide unstructured data recognition services for electronic design automation, building information modeling, and industrial quality inspection.

[0081] A system for processing unstructured data based on multimodal semantic enhancement, used to implement the aforementioned method for processing unstructured data based on multimodal semantic enhancement, includes:

[0082] The multimodal feature extraction and fusion module is used to extract visual features, recognize and process text, standardize and map symbolic or structural information from the input unstructured data, and realize the association and fusion of different modal features through a multimodal alignment mechanism to generate a unified standardized multimodal feature representation.

[0083] The semantic encoding and structural modeling module is used to perform semantic encoding and similarity optimization on the multimodal features based on the improved language model, and to construct a graph structure representation that reflects the inherent relationship between the data. The graph neural network is used to aggregate and update the node features in the graph, and the semantic representation and graph structure representation are jointly optimized through a joint loss function.

[0084] The cross-modal alignment and embedding generation module is used to achieve interactive alignment between semantic features and structural features using a cross-modal attention mechanism, generate a unified embedding representation through a weighted fusion strategy, and perform similarity calculation and matching analysis on the embedding representation from multiple feature dimensions.

[0085] The results output and adaptive optimization module is used to visualize the recognition results and similarity analysis results, and to extend the functions of data retrieval, anomaly detection, consistency verification and semantic comparison. At the same time, user feedback information is introduced to update and optimize model parameters to support cross-scenario and cross-domain unstructured data processing applications.

[0086] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for processing unstructured data based on multimodal semantic enhancement, characterized in that, Includes the following steps: S1. Visual feature extraction, text recognition and semantic processing, symbol or structural information standardization and mapping are performed on the input unstructured data respectively, and the association and fusion of different modal features are realized through a multimodal alignment mechanism to generate a unified standardized multimodal feature representation; The specific process of step S1 is as follows: S11. Visual Data Feature Extraction: A convolutional neural network is used to obtain a symbolic feature map. The calculation formula is as follows: ,in, Symbolic feature map; For convolutional neural network operations; For the input visual data, H, W, and C are the height, width, and number of channels of the visual data, respectively, and h, w, and d are the height, width, and channel dimension of the feature map, respectively. S12. Text Recognition and Processing: Text information is extracted using OCR technology, and after word segmentation, noise reduction, and normalization, the text is embedded using Word2Vec model, FastText model, or large language model embedding function. S13. Symbol Standardization and Mapping: Construct a standard symbol library for common components, and calculate the cosine similarity between the features of the symbol to be identified and the standard symbols. The formula for calculating the cosine similarity is: ,in, The cosine similarity between the features of the symbol to be identified and the standard symbol; Features of the symbol to be identified; Standard symbols; It is the vector norm; If the cosine similarity is greater than the threshold θ, the symbol to be identified is determined to belong to the standard element category represented by the standard symbol, and is bidirectionally bound to the text label; S14. Multimodal Alignment and Feature Fusion: A cross-modal alignment method is adopted, using a linear projection function to map visual data features and text features to the same latent space, thereby achieving a unified representation of visual data features and text features. The calculation formula is as follows: , ,in, Visual data features mapped to the same latent space; is the linear projection matrix of the visual data modal features; The original visual data feature matrix output by the visual data feature extraction network; Text features mapped to the same latent space; is the linear projection matrix of the text modal features; The text feature matrix extracted by the text semantic encoding model; A contrastive learning loss function is used to optimize cross-modal similarity, minimizing the distance between visual data and text of the same symbol in the embedding space, and maintaining the distinguishability between representations of different symbols. The formula for calculating the contrastive learning loss function is as follows: ,in, To learn the loss function by comparison; The cosine similarity function; For temperature parameters; Index representing a negative sample; Let this represent the text feature vector of the k-th negative sample; S2. Based on the improved language model, semantic encoding and similarity optimization of multimodal features are performed, and a graph structure representation reflecting the inherent relationship of data is constructed. The graph neural network is used to aggregate and update the node features in the graph, and the semantic representation and graph structure representation are jointly optimized through a joint loss function. S3. A cross-modal attention mechanism is adopted to achieve interactive alignment between semantic features and structural features. A unified embedding representation is generated through a weighted fusion strategy, and similarity calculation and matching analysis are performed on the embedding representation from multiple feature dimensions. S4. Visualize the recognition results and similarity analysis results, and extend the functionality to include data retrieval, anomaly detection, consistency verification, and semantic-level comparison. Then, incorporate user feedback to update and optimize the model parameters to support cross-scenario and cross-domain unstructured data processing applications.

2. The unstructured data processing method based on multimodal semantic enhancement as described in claim 1, characterized in that: In step S13, the threshold θ = 0.85; in step S14, the temperature parameter... =0.

07.

3. The unstructured data processing method based on multimodal semantic enhancement as described in claim 1, characterized in that, The specific process of step S2 is as follows: S21. Semantic Modeling: An improved large language model is used to model the preprocessed text input. Encoding is performed; among them, the improved large language model uses the BERT model, DeBERTaV3 model, or RoBERTa model. The input consists of the first n preprocessed texts; the encoding calculation formula is: ,in, For the contextual representation of symbolic text; Embedded in text; This means that the text embedding E(T) of the input text T is input into the Transformer model, and the output is a feature representation containing contextual semantic information, which is used for subsequent semantic modeling and similarity calculation; n is the number of text inputs; For semantic vector dimensions; By constraining similar symbols to text in the embedding space using the binary cross-entropy loss function, different categories of symbols are kept distinguishable. The formula for calculating the binary cross-entropy loss function is as follows: ,in, is the binary cross-entropy loss function; i is the index of the training sample pair; The index of the other node paired with sample i, where the node is a symbol or text; This represents the total number of all symbolic sample pairs during training. Indicate whether the symbol pair is of the same type; For the Sigmoid function; Let be the embedded feature vector of the i-th node; For the first sample pair that forms with node i Embedded feature vectors of each node; S22. Graph Structure Modeling: Each symbol or text element is treated as a node, and the spatial or semantic relationships between nodes are used as edges to construct a graph structure. Edge weights are calculated using spatial distance or semantic similarity, with the following formula: ,in, The edge weights; For balance parameters; For the i-th node and the i-th node Spatial distance between nodes; For node semantic similarity; S23. Node Feature Aggregation: Node feature aggregation is performed using a graph convolutional neural network. The calculation formula is as follows: ,in, Features of the (l+1)th layer nodes; It is a degree matrix; For adjacency matrices with self-loops, This is the original adjacency matrix; Features of the l-th layer nodes; Here is the weight matrix for the l-th layer; S24. Joint Optimization: Joint training is performed by weighting the semantic loss and the graph structure embedding loss to maintain semantic consistency and structural rationality. The calculation formula for joint training is as follows: ,in, The objective function is used to jointly optimize the constraints on semantic consistency and graph structure rationality. For graph structure embedding loss; and These are the weighting coefficients.

4. The unstructured data processing method based on multimodal semantic enhancement as described in claim 3, characterized in that, The specific process of step S3 is as follows: S31. Cross-modal fusion: A cross-modal attention mechanism is used to achieve interactive alignment between semantic features and graph structure features. The calculation formula for the cross-modal attention mechanism is as follows: , , , ,in, For cross-modal attention output; For query matrix; The key matrix; It is a value matrix; This is the normalization function; The dimension of the key vector; for transpose; Semantic features; The trainable weight matrix for the query vector; Graph structural features; is the trainable weight matrix for the key vectors; The trainable weight matrix is ​​a value vector; S32. Unified Representation Learning: A unified embedding representation is obtained through a weighted fusion strategy. The calculation formula for the weighted fusion strategy is as follows: ,in, β is a unified embedding representation of unstructured data symbols; β is a fusion weighting factor. For cross-modal fusion features; For feature splicing operations; S33. Multi-scale similarity calculation: Cosine similarity is used to measure the semantic and structural matching degree of individual symbols. The formula for calculating symbol-level similarity is: ,in, Symbol-level similarity; Let be the fusion feature vector of symbol 'a'; Let be the fusion feature vector of symbol b; The subgraph matching method maps a subset of symbols to a subgraph, and calculates local structural similarity using graph edit distance. The formula for calculating local structural similarity is as follows: ,in, For local structural level similarity; Edit distance for the image; It is a local subgraph consisting of the location of symbol 'a' and its neighboring symbols; This is the corresponding local subgraph composed of symbol b and its neighboring symbols; Combining symbol-level similarity and local structural-level similarity, a weighted average is used to obtain the global unstructured data-level similarity. The formula for calculating the global unstructured data-level similarity is as follows: ,in, Similarity at the global unstructured data level; These are weighting coefficients used to balance symbol precision and structural integrity; It is a function of average value; S34. Loss Function Optimization: Combine binary cross-entropy loss and contrastive loss for model training to maintain the discriminative power of different class embeddings. The calculation formula is as follows: , ,in, To optimize the overall objective function; and These are the weighting coefficients; For binary cross-entropy loss; Compare the losses; This is the boundary threshold.

5. The unstructured data processing method based on multimodal semantic enhancement as described in claim 4, characterized in that, The specific process of step S4 is as follows: S41. Results Visualization: The recognition results are presented intuitively through symbol annotation, relationship visualization, and similarity heatmap. S42. Functional Expansion: Implement data retrieval based on global unstructured data level similarity, error detection based on graph structure comparison, standardized proofreading by comparison with industry standard symbol libraries, and semantic level comparison across languages ​​and standards; S43. Feedback Optimization: Update model parameters based on user annotations. The formula for calculating the updated model parameters is: ,in, For the first Model parameters after the next iteration; For the first The model parameters for this iteration; η is the learning rate; For parameters Below, the model uses the original training dataset. loss function The calculated gradient; These are the original training samples; The original training labels or supervision information; μ is the feedback weight; Gradient updates are provided to correct user behavior. S44. Cross-domain applications: Provide unstructured data recognition services for electronic design automation, building information modeling, and industrial quality inspection.

6. A system for processing unstructured data based on multimodal semantic enhancement, used to implement the method for processing unstructured data based on multimodal semantic enhancement as described in any one of claims 1-5, characterized in that, include: The multimodal feature extraction and fusion module is used to extract visual features, recognize and process text, standardize and map symbolic or structural information from the input unstructured data, and realize the association and fusion of different modal features through a multimodal alignment mechanism to generate a unified standardized multimodal feature representation. The semantic encoding and structural modeling module is used to perform semantic encoding and similarity optimization on the multimodal features based on the improved language model, and to construct a graph structure representation that reflects the inherent relationship between the data. The graph neural network is used to aggregate and update the node features in the graph, and the semantic representation and graph structure representation are jointly optimized through a joint loss function. The cross-modal alignment and embedding generation module is used to achieve interactive alignment between semantic features and structural features using a cross-modal attention mechanism, generate a unified embedding representation through a weighted fusion strategy, and perform similarity calculation and matching analysis on the embedding representation from multiple feature dimensions. The results output and adaptive optimization module is used to visualize the recognition results and similarity analysis results, and to extend the functions of data retrieval, anomaly detection, consistency verification and semantic comparison. At the same time, user feedback information is introduced to update and optimize model parameters to support cross-scenario and cross-domain unstructured data processing applications.

Images