Multi-modal data drawing logical relationship analysis method, electronic equipment and medium

Abstract

The invention discloses a multi-modal data drawing logical relationship analysis method, electronic equipment and a medium, and the method comprises the steps: generating a node set based on drawing image data and text data; generating a cross-modal hyperedge set based on the spatial proximity relationship, the visual feature similarity and the semantic correlation between the node sets; generating a hypergraph embedding input representation based on the node set and the cross-modal hyperedge set; the hypergraph is embedded into the input representation input improved hypergraph self-attention network model, and a hyperedge logic relation type and a corresponding hyperedge confidence coefficient are generated; generating a graph structure result based on the hyperedge logic relationship type and the node set, wherein the graph structure result meets the structure legality requirement; and performing hyper-parameter automatic adjustment and convergence control on the atlas structure result based on hyper-edge confidence, and generating an optimal atlas analysis model and a structured output result. According to the method, the reliability and the quality of analysis of component nodes, logic edge relationships and semantic structures in the drawing are improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for parsing logical relationships in multimodal data drawings, electronic devices, and media. Background Technology

[0002] In modern engineering design and construction management, drawings, as a core form of data representation, are widely used in various fields such as architecture, machinery, and power. Traditional drawings contain a large number of structural components, symbols, text annotations, and dimensions, expressing information in a hybrid of image and text modalities, exhibiting significant unstructured, multimodal, and high semantic density characteristics. With the increasing demand for information management, how to structurally analyze the content of complex drawings, extract the logical relationships between components, and achieve automated modeling and graphical representation has become one of the key issues in the digital application of drawings.

[0003] In related technologies, engineering drawing information parsing methods can achieve partial information extraction under specific conditions, but most of them suffer from problems such as limited structural analysis capabilities, weak intermodal connections, and low semantic abstraction levels, making it difficult to meet the needs of in-depth analysis of the logical relationships in drawings under complex scenarios. Summary of the Invention

[0004] The purpose of this invention is to construct a high-order hypergraph structure by integrating a multimodal perception mechanism of text and images, thereby enabling a multimodal data drawing logical relationship parsing method, electronic device, and medium that improves the reliability and quality of parsing component nodes, logical edge relationships, and semantic structure in drawings.

[0005] To achieve the above objectives, this invention proposes a method for parsing logical relationships in multimodal data drawings, comprising: generating a node set based on drawing image data and text data; generating a cross-modal hyperedge set based on spatial proximity, visual feature similarity, and semantic relevance among the node sets; generating a hypergraph embedding input representation based on the node set and the cross-modal hyperedge set; inputting the hypergraph embedding input representation into an improved hypergraph self-attention network model to generate hyperedge logical relationship types and corresponding hyperedge confidence scores; generating a graph structure result based on the hyperedge logical relationship types and the node set, wherein the graph structure result satisfies the structural validity requirement; and automatically adjusting and converging the hyperparameters of the graph structure result based on the hyperedge confidence scores to generate an optimal graph parsing model and a structured output result.

[0006] In one optional implementation, a node set is generated based on drawing image data and text data, specifically including: performing grayscale conversion, perspective distortion correction, and structural enhancement operations on the drawing image data to obtain a structurally clear image input; performing region segmentation based on the image input to extract graphic component regions and auxiliary annotation regions, and constructing graphic node candidate regions and auxiliary node candidate regions respectively; performing vectorization analysis on the graphic node candidate regions to extract contour shape, proportional features, and spatial coordinate information to generate a graphic node set; performing template matching and image text recognition on the auxiliary node candidate regions to extract auxiliary explanatory content and its spatial position to generate an auxiliary node set; performing character extraction, text cleaning, and spatial positioning on the text data corresponding to the drawing, and embedding it with a pre-trained language model to generate a text node set; uniformly identifying and encoding the graphic node set, the text node set, and the auxiliary node set to generate the node set, wherein the node set includes node number, node type, spatial position, and corresponding attribute information.

[0007] In one optional implementation, a cross-modal hyperedge set is generated based on the spatial proximity, visual feature similarity, and semantic relevance among node sets. Specifically, this includes: calculating the spatial proximity between the text node set, the graphic node set, and the auxiliary node set; constructing an initial spatial adjacency matrix based on the Euclidean distance and layout density between nodes; extracting local geometric features of graphic nodes and texture features of auxiliary nodes, and fusing them to generate a visual similarity scoring matrix between node pairs; encoding text nodes using a pre-trained language model, and constructing a semantic relevance scoring matrix between node pairs by combining the cosine similarity between context word vectors; weighting and fusing the initial spatial adjacency matrix, the visual similarity scoring matrix, and the semantic relevance scoring matrix to construct a cross-modal candidate hyperedge set, where each candidate hyperedge connects multiple node pairs with different modalities; performing attribute scoring on each hyperedge in the cross-modal candidate hyperedge set, with scoring factors including node modality combination weights, local spatial density constraints, and semantic mutual information strength, and generating an initial confidence score list; and inputting the initial confidence score list into a benchmarking optimization algorithm to generate the cross-modal hyperedge set.

[0008] In one optional implementation, the initial confidence score list is input into a benchmark management optimization algorithm to generate the cross-modal hyperedge set. Specifically, this includes: dynamically selecting the top N% of hyperedges based on the initial confidence score list as a benchmark set, and marking their attribute distribution as a locally optimal reference indicator; performing score offset measurement and confidence redistribution on non-benchmark hyperedges according to the feature differences between them and the benchmark set, thereby increasing the score weight of hyperedges close to the benchmarks; constructing a hyperedge coverage matrix, detecting whether each hyperedge has redundant coverage and blind spot omissions in different modal node spaces, and generating detection results; dynamically adjusting the score threshold based on the detection results to generate global structural coverage quality; and jointly performing score updates and confidence iterative convergence based on the locally optimal reference indicator and the global structural coverage quality, iteratively updating the hyperedge score weights and confidence thresholds to generate the cross-modal hyperedge set.

[0009] In one optional implementation, the hypergraph embedding input representation is input into an improved hypergraph self-attention network model to generate hyperedge logical relationship types and corresponding hyperedge confidence scores. Specifically, this includes: performing a trainable hyperedge filtering operation on the cross-modal hyperedge set in the hypergraph embedding input representation based on the improved hypergraph self-attention network model to generate a filterable hyperedge set; based on a modality-aware multi-head attention mechanism, constructing multiple sets of modal attention weights for the text node set, the graphic node set, and the auxiliary node set under the filterable hyperedge set, modeling the feature coupling relationship between different modalities through parallel attention heads, and completing modal information fusion to generate a fusion model; based on a structure-order-preserving hyperedge attention mechanism, introducing component arrangement order constraints into the fusion model, modeling order-preserving dependent attention weights for hyperedge order relationships, and generating a structural logical sequence model; based on a cross-modal auxiliary edge suppression mechanism, detecting and identifying low-confidence or abnormal cross-modal auxiliary edges in the structural logical sequence model during the model output stage, weakening the attention weights by constructing a dynamic decay mask matrix, and generating hyperedge logical relationship types and corresponding hyperedge confidence scores.

[0010] In one optional implementation, generating a hypergraph embedding input representation based on the node set and the cross-modal hyperedge set specifically includes: uniformly numbering the text node set, the graphic node set, and the auxiliary node set, assigning node type labels to each, extracting the modal label, spatial coordinates, and preliminary attribute feature information of each node, and constructing a node attribute dictionary; performing an association node analysis operation on each hyperedge in the cross-modal hyperedge set, obtaining the type combination and spatial coordinates of each associated node based on the node attribute dictionary, and identifying the cross-modal connection structure; generating a structured position encoding vector for each node based on the spatial coordinates in the node attribute dictionary using a rotation-consistent position encoding mechanism; fusing the modal labels, attribute features, and structured position encoding vectors in the node attribute dictionary to construct a node representation vector set; and jointly constructing the cross-modal hypergraph structure with the node representation vector set and the cross-modal connection structure to generate the hypergraph embedding input representation.

[0011] In one optional implementation, a graph structure result is generated based on the hyperedge logical relationship type and the node set. The graph structure result satisfies the structural legality requirement. Specifically, this includes: constructing a preliminary structured drawing logical relationship graph based on the hyperedge logical relationship type, the text node set, the graphic node set, and the auxiliary node set. Each edge in the preliminary structured drawing logical relationship graph is associated with a corresponding logical relationship type label and a confidence value. The preliminary structured drawing logical relationship graph is input into a differentiable integer programming constraint model. Based on the drawing design specifications and component assembly constraints, a node connection rule set is defined, and a structural legality verification objective function in integer programming form is constructed. In the differentiable integer programming constraint model, a structural consistency solution operation is performed on each graph edge relationship to identify all edge sets that do not satisfy the constraint rules. The edge sets that do not satisfy the constraint rules are removed, and the drawing logical relationship graph result after structural consistency verification and edge removal processing is output as the graph structure result.

[0012] In one optional implementation, the hyperparameters of the graph structure result are automatically adjusted and converged based on the hyperedge confidence to generate the optimal graph parsing model and structured output result. Specifically, this includes: setting a joint optimization objective function, using the accuracy and consistency indices of the graph structure result as optimization targets, and constructing a fitness function based on the performance evaluation results of the improved hypergraph self-attention network model during training; initializing the population structure of the yin-yang pair optimization algorithm, generating candidate individuals, wherein the candidate individuals include hyperedge threshold, logical relationship confidence threshold, node embedding dimension, network learning rate, and multiple... The task loss function weights are defined; based on the candidate individuals, two sets of opposing subpopulations are used to perform population splitting operations, and multi-directional perturbation is applied to each dimension based on a perturbation strategy to generate new solution sub-individuals; the fitness of the new solution sub-individuals is evaluated to obtain fitness evaluation results; guided update operations are performed on the negative subpopulations according to the fitness evaluation results, retaining sub-individuals with high distribution diversity and good fitness, and an archiving mechanism is performed in the positive subpopulations to gradually retain the global optimal solution through alternating replacement; after the convergence condition is met or the preset number of iterations is reached, the optimal graph analysis model and structured output results are generated.

[0013] On the other hand, the present invention also proposes an electronic device, comprising: at least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute any of the multimodal data drawing logical relationship parsing methods described above.

[0014] On the other hand, the present invention also proposes a medium storing a computer program, which, when executed by a processor, implements the multimodal data drawing logical relationship parsing method described in any one of the claims.

[0015] The beneficial effects of this invention are as follows: through precise construction of multimodal nodes, multi-dimensional generation of cross-modal hyperedges, improved hypergraph self-attention network modeling, compliance graph verification, and automatic optimization of hyperparameters, high precision, high reliability, and high efficiency in parsing the logical relationship of drawings are achieved. It can not only fully capture the multimodal information association of drawings, but also ensure the structural legality of the output graph and the optimal performance of the model, providing effective support for the automated and accurate parsing of the logical relationship of drawings. Attached Figure Description

[0016] Figure 1 A flowchart of a method for parsing the logical relationship of multimodal data drawings provided in one embodiment of the present invention;

[0017] Figure 2 A flowchart of a method for parsing the logical relationship of multimodal data drawings provided in another embodiment of the present invention. Detailed Implementation

[0018] Methods for analyzing engineering drawing information mainly focus on two directions: one is component extraction methods based on image processing and graphic recognition, which analyze drawing images using edge detection, template matching, morphological operations, etc., to identify standard graphics, symbols, and connections; the other is natural language processing technology to identify and understand text annotations in drawings, extract annotation content, and attempt to associate it with graphic information. While these methods can achieve partial information extraction under specific conditions, most suffer from limited structural analysis capabilities, weak intermodal connections, and low semantic abstraction levels, making it difficult to meet the needs for deep analysis of logical relationships in complex scenarios.

[0019] Traditional image recognition methods primarily focus on extracting geometric structures at the graphic level, making it difficult to understand the roles and relationships of graphic components in engineering semantics. Even with the introduction of image segmentation and object detection techniques in some methods, there is still a lack of ability to model the order, hierarchy, and dependencies between components. Secondly, in terms of multimodal data fusion, existing technologies mostly employ simple splicing or rule matching methods, which struggle to effectively combine the spatial associations and semantic orientations between text and graphics, resulting in difficulties in integrating cross-modal information into a unified structural expression. Furthermore, most current drawing parsing methods do not fully incorporate graph structure modeling techniques and lack support for hypergraph modeling, causing the models to be unable to handle complex semantic relationships such as one-to-many and many-to-many relationships, thus limiting the expressive power of logical relationship graphs.

[0020] Graph neural network technology has found some applications in multimodal structural modeling, with some studies attempting to simulate the relationship between drawing components and annotations by constructing structures of nodes and edges. However, these methods mostly use conventional graph structures, making it difficult to simultaneously represent high-order relationships between multiple nodes, and they fail to effectively model the rotational invariance, spatial clustering features, and annotation relationships unique to drawings. The application of attention mechanisms also suffers from insufficient specificity, failing to distinguish modal differences or adjust structural hierarchical relationships, easily introducing mismatches or information redundancy. More importantly, issues such as component order, engineering semantic consistency, and spatial standardization in drawing parsing tasks have not been systematically modeled and verified, leading to unstable structural outputs and frequent logical relationship conflicts, severely limiting the practicality of the parsing results.

[0021] In terms of model training and parameter tuning, current methods mostly rely on manual setting or adjustment based on a single metric, failing to construct an effective multi-objective optimization mechanism and making it difficult to achieve a good balance between accuracy, structural consistency, and parsing speed. Some studies have introduced evolutionary algorithms for optimization, but these are mostly general animal heuristic algorithms that do not take into account the specific constraints of the map structure, resulting in slow convergence speed and poor generalization ability, which is not conducive to the rapid deployment and dynamic updating of map parsing models.

[0022] Existing multimodal drawing parsing methods still have significant limitations in terms of information fusion depth, structural modeling capabilities, logical relationship expression, and optimization adjustment mechanisms. In particular, they lack the ability to systematically model high-order logical relationships within drawings, making it difficult to meet the high accuracy, consistency, and interpretability requirements of structured drawing parsing results in intelligent engineering design and management scenarios. Therefore, there is an urgent need to propose a multimodal perception mechanism that integrates text and images, constructs a high-order hypergraph structure, employs a graph neural network model with structure-preserving and modal difference recognition capabilities, and introduces an interpretable and tunable optimization mechanism. This will enable high-quality intelligent parsing of component nodes, logical edge relationships, and semantic structures in drawings, thereby supporting the practical needs of downstream design automation, engineering drawing modeling, and structural review.

[0023] Therefore, this invention primarily addresses the challenges of unified parsing of multimodal information and accurate modeling and verification of logical relationships in complex engineering drawings. Existing methods often rely solely on image recognition or text processing, resulting in information silos, weak modal connections, and shallow semantic levels, making it difficult to fully understand the logical relationships between components in the drawings. Furthermore, traditional methods lack the ability to model hypergraphs of high-order associations among multiple nodes, cannot handle complex semantic relationships such as one-to-many and many-to-many relationships, and have insufficient logical consistency checks, leading to frequent mismatches and conflicts in the output results.

[0024] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] like Figure 1 As shown, according to an embodiment of the present invention, in one aspect, a method for parsing the logical relationship of multimodal data drawings is provided, comprising the following steps:

[0026] Step S101: Generate a node set based on the drawing image data and text data.

[0027] Step S103: Generate a cross-modal hyperedge set based on the spatial proximity relationship, visual feature similarity and semantic relevance between node sets.

[0028] Step S105: Generate a hypergraph embedding input representation based on the node set and the cross-modal hyperedge set.

[0029] Step S107: Embed the hypergraph into the input representation of the improved hypergraph self-attention network model, and generate the hyperedge logical relationship type and the corresponding hyperedge confidence.

[0030] Step S109: Generate a graph structure result based on the hyperedge logical relationship type and node set. The graph structure result meets the structural legality requirements.

[0031] Step S1011: Based on the hyperedge confidence, perform automatic adjustment and convergence control of hyperparameters on the graph structure results to generate the optimal graph analytical model and structured output results.

[0032] In this embodiment, drawing image data and text data are collected, preprocessed, and a node set is generated. The cross-modal hyperedge set is a refined cross-modal hyperedge set. A cross-modal hypergraph structure is constructed based on the node set and the refined cross-modal hyperedge set, generating node representation vectors containing rotationally consistent position codes to form a hypergraph embedding input representation. The hypergraph embedding input representation is input into an improved hypergraph self-attention network model. A structure-preserving hyperedge attention mechanism is used to model the component arrangement order constraints, and a cross-modal auxiliary edge suppression mechanism is introduced to suppress the propagation effect of mismatched hyperedges. The logical relationship type and corresponding confidence level of each refined hyperedge are output. A structured drawing logical relationship graph is constructed based on the logical relationship type and the node set. A differentiable integer programming constraint model is input for logical consistency verification, and a graph structure result that meets the structural legality requirements is output. Based on the graph structure result and the parsing process of the improved hypergraph self-attention network model, a yin-yang optimization algorithm is used to automatically adjust and control the hyperparameters, generating an optimized graph parsing model and structured output results.

[0033] By generating node sets through the fusion of multimodal data from drawing images and text, and producing refined cross-modal hyperedge sets based on spatial, visual, and semantic dimensions, the limitations of single-modal data can be effectively reduced, lowering the hyperedge mismatch rate. The introduction of rotationally consistent position encoding optimizes the hypergraph embedding input representation. Combined with the "structure-preserving hyperedge attention mechanism" and "cross-modal auxiliary edge suppression mechanism" of the improved hypergraph self-attention network, it can accurately capture component arrangement order constraints and suppress the propagation impact of mismatched hyperedges, significantly improving the accuracy of hyperedge logical relationship type and confidence judgment. A differentiable integer programming constraint model is used to perform logical consistency verification on the preliminary graph, ensuring that the output graph meets structural legality requirements. Simultaneously, a yin-yang pair optimization algorithm is used to achieve automatic hyperparameter adjustment and convergence control, eliminating the need for repeated manual debugging and efficiently generating the optimal graph analysis model and structured output results, significantly improving analysis efficiency and practicality.

[0034] Further, step S101, based on the drawing image data and text data, generates a node set, specifically including the following steps:

[0035] Step S1011: Perform grayscale conversion, perspective distortion correction, and structural enhancement operations on the drawing image data to obtain a clearly structured image input.

[0036] Step S1013: Perform region segmentation based on image input, extract graphic component regions and auxiliary annotation regions, and construct candidate regions for graphic nodes and candidate regions for auxiliary nodes respectively.

[0037] Step S1015: Perform vectorization analysis on the candidate regions of graphic nodes, extract contour shape, proportion features and spatial coordinate information, and generate a set of graphic nodes.

[0038] Step S1017: Perform template matching and image text recognition on the candidate regions of auxiliary nodes to extract the auxiliary description content and its spatial location, and generate a set of auxiliary nodes.

[0039] Step S1019: Extract characters, clean and spatially locate text data corresponding to the drawing, and generate a set of text nodes by embedding a pre-trained language model.

[0040] Step S10201: Unify the identification and encoding of the graphic node set, text node set and auxiliary node set to generate a node set. The node set includes node number, node type, spatial location and corresponding attribute information.

[0041] In this embodiment, the candidate regions of graphic nodes are vectorized to extract contour shape, proportion features and spatial coordinate information, and a set of graphic nodes and their preliminary attribute information are generated; the candidate regions of auxiliary nodes are combined with template matching and image text recognition to extract auxiliary explanatory content, and their spatial positions are recorded to generate a set of auxiliary nodes and their text content attributes; the text data corresponding to the drawing is subjected to character extraction, text cleaning and spatial positioning, and a set of text nodes and their semantic attributes are generated by combining BERT word vector embedding.

[0042] By simplifying image dimensions through grayscale conversion, eliminating shooting / scanning deviations through perspective distortion correction, and highlighting details through structural enhancement, interference information in the drawing image is effectively removed, resulting in a clear and structurally sound image input. This lays a solid foundation for the accuracy of subsequent region segmentation and avoids misjudgment of regions due to image quality issues.

[0043] Precise segmentation of graphic components and auxiliary annotation areas, and construction of corresponding candidate areas, eliminates the mixing of different types of information, allows subsequent node generation to be processed in a targeted manner, avoids cross-interference between graphic and annotation information, and improves the accuracy of node classification.

[0044] Geometric features are extracted from graphical nodes, descriptive and location information is extracted from auxiliary nodes, and semantic attributes are extracted from text nodes using BERT word vectors. Dedicated feature extraction methods are matched for different types of nodes, which not only ensures the accuracy of feature extraction but also enriches the dimensions of node attributes, providing sufficient basis for node differentiation and association.

[0045] By unifying the identification and coding of the three types of nodes, clarifying the node number, type, location and attributes, the format barriers between image and text multimodal data are broken down, the problem of information fragmentation is eliminated, and a structured node set is formed. This provides standardized data support for subsequent cross-modal hyperedge construction and logical relationship parsing, and reduces the processing complexity of subsequent processes.

[0046] Further, step S103, generating a cross-modal hyperedge set based on spatial proximity, visual feature similarity, and semantic relevance among node sets, specifically includes the following steps:

[0047] Step S1031: Calculate the spatial proximity between the set of text nodes, the set of graphic nodes, and the set of auxiliary nodes, and construct an initial spatial adjacency matrix based on the Euclidean distance and layout density between nodes.

[0048] Step S1033: Extract the local geometric features of the graphic nodes and the texture features of the auxiliary nodes, and fuse them to generate a visual similarity scoring matrix between node pairs.

[0049] Step S1035: Encode the text nodes using a pre-trained language model, and construct a semantic relevance score matrix between node pairs by combining the cosine similarity between context word vectors.

[0050] Step S1037: The initial spatial adjacency matrix, visual similarity score matrix and semantic relevance score matrix are weighted and fused to construct a cross-modal candidate hyperedge set. Each candidate hyperedge connects multiple node pairs with different modalities.

[0051] Step S1039: Attribute scoring is performed on each hyperedge in the cross-modal candidate hyperedge set. The scoring factors include node modality combination weight, local spatial density constraint and semantic mutual information strength, and an initial confidence score list is generated.

[0052] Step S1041: Input the initial confidence score list into the benchmark management optimization algorithm to generate a cross-modal hyperedge set.

[0053] The association matrix is ​​constructed from three core dimensions: spatial proximity (Euclidean distance, layout density), visual feature similarity (geometric + texture features), and semantic relevance (cosine similarity). This matrix covers the spatial, visual, and semantic information of multimodal drawing data, avoiding the limitations of single-dimensional association and allowing cross-modal hyperedges to more comprehensively reflect the real associations between nodes.

[0054] Candidate hyperedges are generated by weighted fusion of multi-dimensional matrices, and attribute scoring is performed by combining "modal combination weights + spatial density constraints + semantic mutual information". This allows for the selection of highly relevant candidate hyperedges from the source, significantly reducing the proportion of irrelevant or mismatched hyperedges and improving the quality of the hyperedge set.

[0055] By using this algorithm to further optimize the candidate hyperedges of the initial confidence score, the optimal hyperedges can be selected based on benchmark standards, and low-confidence hyperedges can be eliminated. The final cross-modal hyperedge set is more in line with the logical association rules of the drawing, providing high-quality input for subsequent hypergraph embedding and logical relationship analysis.

[0056] Furthermore, in step S1041, the initial confidence score list is input into the benchmark management optimization algorithm to generate a cross-modal hyperedge set, which specifically includes the following steps:

[0057] Step S10411: Based on the initial confidence score list, dynamically select the top N% of hyperedges as the benchmark set and mark their attribute distribution as the local optimal reference index.

[0058] Step S10413: For non-benchmark hyperedges, perform a score offset measurement and confidence redistribution based on the feature difference between them and the benchmark set, and increase the score weight of hyperedges that are close to the benchmark.

[0059] Step S10415: Construct a hyperedge coverage matrix, detect whether there is redundant coverage and blind spot omission for each hyperedge in different modal node spaces, and generate detection results.

[0060] Step S10417: Dynamically adjust the scoring threshold based on the detection results to generate global structural coverage quality.

[0061] Step S10419: Based on the local optimal reference index and the global structure coverage quality, jointly perform score update and confidence iterative convergence, iteratively update the hyperedge score weight and confidence threshold, and generate a cross-modal hyperedge set.

[0062] In this embodiment, the scoring threshold is dynamically adjusted to balance the global structural coverage quality. By dynamically adjusting the scoring threshold to balance the global coverage quality, and combining iterative convergence of confidence with local optimal indicators, it is possible to avoid introducing a large number of low-quality hyperedges in pursuit of high coverage, and also to prevent incomplete coverage due to the pursuit of quality. Ultimately, a balanced set of cross-modal hyperedges with "high quality + full coverage" is generated, providing a more reliable structural foundation for subsequent hypergraph modeling.

[0063] By dynamically selecting hyperedges with high confidence scores as the benchmark set, a local optimal reference is provided for non-benchmark hyperedges. Then, by measuring the reconfiguration confidence through feature difference, candidate hyperedges with similar features to high-quality hyperedges can be targeted to strengthen, ensuring the basic quality of the hyperedge set at the local level and reducing the retention of low-quality hyperedges.

[0064] Constructing a hyperedge coverage matrix to detect redundant coverage and blind spot omissions can accurately identify the distribution defects of hyperedges in the multimodal node space, avoid global structural imbalance caused by local optimization (such as some nodes having no associated hyperedges or some regions having excessive hyperedge overlap), and ensure the global integrity of the hyperedge set.

[0065] Further, step S105, generating a hypergraph embedding input representation based on the node set and the cross-modal hyperedge set, specifically includes the following steps:

[0066] Step S1051: Number the text node set, graphic node set and auxiliary node set uniformly, assign node type labels to each, extract the modal label, spatial coordinates and preliminary attribute feature information of each node, and construct a node attribute dictionary.

[0067] Step S1053: Perform an associated node analysis operation on each hyperedge in the cross-modal hyperedge set, obtain the type combination and spatial coordinates of each associated node based on the node attribute dictionary, and identify the cross-modal connection structure.

[0068] Step S1055: Based on the spatial coordinates in the node attribute dictionary, generate the structured position encoding vector for each node using a rotation-consistent position encoding mechanism.

[0069] Step S1057: Fuse the modal labels, attribute features and structured location encoding vectors in the node attribute dictionary to construct a set of node representation vectors.

[0070] Step S1059: Construct a cross-modal hypergraph structure by combining the set of node representation vectors with the cross-modal connection structure, and generate a hypergraph embedding input representation.

[0071] By unifying numbering, assigning type labels, and constructing a node attribute dictionary, the modal, spatial, and attribute information of multimodal nodes is systematically integrated, avoiding information fragmentation. Using rotation-consistent position encoding based on node spatial coordinates effectively eliminates positional information deviations caused by drawing rotation, ensuring the consistency of node positional features at different angles and improving the reliability of spatial relationship representation in hypergraph embedding. Integrating node modal labels, attribute features, and structured position encoding to generate node representation vectors allows node vectors to simultaneously contain modal, semantic, and spatial dimensional information, providing a more comprehensive characterization of node features and richer input information for modeling hyperedge logical relationships.

[0072] By combining the set of node representation vectors with the cross-modal connection structure to construct a hypergraph, the cross-modal associations between nodes are explicitly presented in the form of a hypergraph structure. This allows the hypergraph embedded input representation to reflect both node features and the relationships between nodes, laying a structured foundation for the subsequent analysis of logical relationships in the network model.

[0073] Construct a structured drawing logical relationship graph based on logical relationship types and node sets. Input a differentiable integer programming constraint model for logical consistency verification and output a graph structure result that meets the requirements of structural legality.

[0074] Further, in step S107, the hypergraph is embedded into the input representation of the improved hypergraph self-attention network model, and the hyperedge logical relationship types and corresponding hyperedge confidence scores are generated. This specifically includes the following steps:

[0075] Step S1071: Based on the improved hypergraph self-attention network model, perform a trainable hyperedge filtering operation on the cross-modal hyperedge set in the hypergraph embedding input representation to generate a filterable hyperedge set.

[0076] Step S1073: Based on the modality-aware multi-head attention mechanism, under the filterable hyperedge set, construct multiple sets of modality attention weights for the text node set, the graphic node set, and the auxiliary node set respectively. Model the feature coupling relationship between different modalities through parallel attention heads and complete the modality information fusion to generate a fusion model.

[0077] Step S1075: Based on the structure-preserving hyperedge attention mechanism, introduce component arrangement order constraints into the fusion model, model the hyperedge order relationship with order-preserving dependency attention weights, and generate a structural logic sequence model;

[0078] Step S1077: Based on the cross-modal auxiliary edge suppression mechanism, the structural logic sequence model is detected and identified in the model output stage. Low-confidence or abnormal cross-modal auxiliary edges are identified. Attention weights are weakened by constructing a dynamic decay mask matrix, and the hyperedge logic relationship type and corresponding hyperedge confidence are generated.

[0079] Differentiable superedge filtering structure is used to perform trainable superedge filtering operations on a refined cross-modal superedge set in the supergraph embedding input representation. It generates a superedge retention mask through joint optimization to retain high-importance superedge connections and filter low-confidence superedges, forming a filterable superedge set.

[0080] By generating a superedge retention mask through a differentiable superedge filtering structure, highly important superedges are retained while low-confidence superedges are filtered out. This reduces noise interference from the input layer, reduces redundancy for subsequent model calculations, and improves the network's efficiency in modeling effective superedge logical relationships.

[0081] By constructing multiple sets of attention weights for different modal nodes and modeling the feature coupling relationship between modalities in parallel, we can more accurately capture the cross-modal associations of text, graphics, and auxiliary nodes, avoid the limitations of a single attention head, and improve the comprehensiveness and accuracy of modal information fusion.

[0082] By introducing component arrangement order constraints and modeling order-preserving dependencies with attention to weights, the model can fit the actual arrangement logic of components in the drawings (such as process sequence and positional order), avoid disordered order of hyper-edge logical relationships, and ensure that the generated structural logic sequence conforms to the objective laws of the drawings.

[0083] In the output stage, attention weights of low-confidence / abnormal auxiliary edges are detected and weakened. The influence of mismatched hyperedges is suppressed by dynamically decaying the mask matrix, reducing the output of erroneous logical relationships. At the same time, the accuracy of hyperedge logical relationship type and confidence judgment is improved, ensuring the quality of model output.

[0084] Further, in step S109, a graph structure result is generated based on the hyperedge logical relationship type and the node set. The graph structure result satisfies the structural validity requirement, specifically including the following steps:

[0085] Step S1091: Construct a preliminary structured drawing logical relationship graph based on the super-edge logical relationship type, the set of text nodes, the set of graphic nodes, and the set of auxiliary nodes. Each edge in the preliminary structured drawing logical relationship graph is associated with a corresponding logical relationship type label and confidence value.

[0086] Step S1093: Input the logical relationship diagram of the preliminary structured drawings into the differentiable integer programming constraint model, and construct the structural legality verification objective function in the form of integer programming based on the node connection rule set defined by the drawing design specifications and component assembly constraints.

[0087] Step S1095: In the differentiable integer programming constraint model, perform a structural consistency solution operation on each graph edge relationship to identify all edge sets that do not satisfy the constraint rules.

[0088] Step S1097: Perform a removal operation on the set of edges that do not meet the constraint rules, and output the drawing logical relationship graph result after structural consistency verification and edge removal processing as the graph structure result.

[0089] A preliminary graph is constructed based on the logical relationship types of hyperedges and the set of multimodal nodes. The relationship type and confidence level of each edge are labeled, so that the logical relationship of the drawing is presented explicitly in a structured form, providing a clear initial analysis object for subsequent legality verification.

[0090] By combining the design specifications of drawings with the node connection rule set defined by the component assembly constraints, the objective compliance requirements are transformed into a verification objective function in the form of integer programming, providing a precise and quantifiable standard for judging the legality of the drawing and avoiding subjective judgment bias.

[0091] By using a differentiable integer programming model to solve each graph edge consistently, it can efficiently identify all non-compliant edges that do not meet the constraint rules. Compared with manual inspection, it is more comprehensive and accurate, and avoids missing potential structural and logical errors.

[0092] Identified non-compliant edges are removed, and the final output is a graph that conforms to structural constraints. This ensures that the graph strictly follows the design and assembly logic of the drawings, guarantees the reliability of subsequent analysis and application from a structural perspective, and avoids decision-making errors caused by non-compliant logic.

[0093] Further, step S1011 involves automatically adjusting and controlling the hyperparameters of the graph structure results based on the hyperedge confidence level to generate the optimal graph analytical model and structured output results. This specifically includes the following steps:

[0094] Step S10111: Set a joint optimization objective function, with the accuracy and consistency indicators of the graph structure results as the optimization objectives, and construct a fitness function by combining the performance evaluation results of the improved hypergraph self-attention network model during the training process;

[0095] Step S10113: Initialize the population structure of the Yin-Yang pair optimization algorithm and generate candidate individuals. The candidate individuals include the super-edge threshold, logical relationship confidence threshold, node embedding dimension, network learning rate and multi-task loss function weights.

[0096] Step S10115: Based on the candidate individuals, two sets of opposing subpopulations are used to perform population splitting operations respectively. Based on the perturbation strategy, multi-directional perturbation is performed on each dimension to generate new solution sub-individuals.

[0097] Step S10117: Evaluate the fitness of the new solution individual to obtain the fitness evaluation results;

[0098] Step S10119: Perform guided update operation on the negative subpopulation based on the fitness assessment results, retain sub-individuals with high distribution diversity and good fitness, and perform an archiving mechanism in the positive subpopulation to gradually retain the global optimal solution through alternating replacement.

[0099] Step S101201: After the convergence condition is met or the preset number of iterations is reached, the optimal graph analysis model and structured output results are generated.

[0100] For edge sets that do not meet the constraint rules, a removal operation is performed to delete them from the structured graph, ensuring that the overall structure of the graph meets the design rules and logical consistency requirements.

[0101] A fitness function is constructed based on graph accuracy, consistency metrics, and network training performance to avoid model imbalance caused by optimizing a single metric (such as focusing only on accuracy while ignoring logical consistency), ensuring that the final model achieves a balance between parsing performance and structural compliance.

[0102] By splitting the antiparameter population and generating new solutions through multi-directional perturbation, and combining the mechanism of maintaining diversity in the negative subpopulation and preserving the global optimum in the positive subpopulation, this algorithm can traverse the hyperparameter space more efficiently than traditional optimization algorithms, reduce the cost of manual parameter tuning, and quickly locate the optimal hyperparameter combination.

[0103] Clearly define convergence conditions and iteration thresholds to avoid infinite loops or premature convergence in the optimization process, ensuring the stability of the final optimal graph analytical model and that the output structured results possess both high accuracy and logical compliance. Edges that do not meet the constraints are removed to eliminate logical contradictions at the structural level, ensuring that the final structured graph strictly conforms to the drawing design rules. This provides a reliable structural foundation for subsequent graph-based applications (such as drawing review and component assembly analysis).

[0104] This invention significantly improves the accuracy of identifying logical relationships between complex components in drawings and enhances structural modeling capabilities by constructing a multimodal data drawing logical relationship parsing method based on a combination of text and images. First, in the preprocessing stage, this invention performs structural enhancement and semantic cleaning operations on the drawing image and text data respectively, and constructs a complete set of text nodes, graphic nodes, and auxiliary nodes, enabling the original unstructured drawing data to obtain a high-quality structured expression, providing an accurate foundation for subsequent relationship modeling. Second, by introducing a cross-modal candidate hyperedge generation and benchmark refinement mechanism, this invention integrates multiple heterogeneous feature dimensions such as spatial proximity, visual feature similarity, and semantic relevance, achieving high-order semantic edge construction between modalities. This avoids the problems of information silos and severe mismatches that traditional graph modeling methods suffer from when dealing with multimodal coupling relationships.

[0105] Furthermore, this invention innovatively designs an improved hypergraph self-attention network model, introducing a structure-preserving hyperedge attention mechanism and a cross-modal auxiliary edge suppression mechanism to ensure the structural consistency of the component arrangement order and logical connectivity in the drawing, effectively suppressing the propagation of misleading edges caused by modal noise. The introduction of a differentiable hyperedge filtering structure further enhances the network's ability to discriminate candidate relation edges and its training adjustability, strengthening the robustness of the logical classification task in complex drawing environments. In the structural output stage, a differentiable integer programming constraint model is used to perform consistency verification and structural legality repair on the generated logical relation graph, ensuring that the final output graph structure not only conforms to semantic reasoning logic but also possesses practical engineering usability.

[0106] Finally, this invention introduces a specifically designed yin-yang pair optimization algorithm to jointly adjust and control key hyperparameters such as hyperedge selection threshold, node embedding dimension, network learning rate, and loss function weights. This enables the entire analytical model to possess stronger adaptability and generalization performance when facing different drawing types and complex structures. In summary, this invention not only breaks through the performance bottlenecks of existing methods in modal fusion and logical parsing, but also achieves systematic innovation in model structure, edge relationship constraints, and parameter optimization, resulting in beneficial technical effects such as high accuracy, strong structural consistency, good interpretability, and high parsing efficiency.

[0107] Application example:

[0108] To verify the application effect and technical feasibility of this invention in a real-world engineering scenario, this embodiment applies the proposed multimodal data drawing logic relationship parsing method based on the combination of text and images to the drawing review task of a residential building project in a new district of a certain city by an architectural design institute. The project's design phase generated a large number of two-dimensional CAD drawings, involving multiple disciplines such as floor slabs, beams and columns, doors and windows, electrical systems, and water supply and drainage. The design drawings were not uniformly formatted; some were scanned images, and some were PDF files with mixed text and images, totaling over 320 drawings. Traditional manual drawing review suffers from significant problems such as a large workload, a high risk of missing logical errors, and ambiguity in understanding drawing information. Therefore, an intelligent method is urgently needed to analyze drawing relationships and verify structural logic.

[0109] In this scenario, the personnel organizing the drawings first input the aforementioned drawing data into the drawing parsing system designed in this invention. The system first performs joint preprocessing on the drawing image and text data. For the image portion, it performs grayscale conversion, perspective correction, and region structure enhancement operations. For the text portion, it uses the BERT language model for semantic cleaning and embedding. Simultaneously, it combines image OCR to extract text annotations from the image, constructing a set of graphic nodes, text nodes, and auxiliary annotation nodes. In actual processing, the system achieves a recognition accuracy of 93.6% for scanned drawings, far exceeding the 81.2% of traditional OCR methods, and performs particularly well with older drawings containing hand-drawn annotations.

[0110] Next, the system utilizes three types of features—spatial proximity, visual similarity, and semantic relevance—to establish a cross-modal candidate hyperedge set. The system then employs the benchmark management optimization algorithm proposed in this invention to iteratively update the weights of these hyperedges, generating a refined cross-modal hyperedge set with higher confidence. In this engineering drawing set, the system automatically generated 12,748 initial candidate hyperedges, and through optimization and screening, retained 6,721 high-confidence hyperedges, improving the average confidence level by 23.8%.

[0111] Subsequently, the system constructs a cross-modal hypergraph containing structured positional encoding and inputs it into the improved hypergraph self-attention network model proposed in this invention for logical relationship parsing. This model introduces a differentiable hyperedge filtering structure to suppress the influence of mismatched interference edges, a modality-aware multi-head attention mechanism to enhance cross-modal semantic alignment capabilities, a structure-preserving hyperedge attention mechanism to model component arrangement relationships, and a cross-modal auxiliary edge suppression mechanism to eliminate unreasonable logical pairings in the drawings. The system outputs the logical type label and confidence value corresponding to each logical edge in the drawings, achieving an accuracy rate of 96.4% in key areas such as structural beam-column connections, annotation associations, and electrical switch wiring.

[0112] The logical analysis results are further input into the differentiable integer programming constraint model designed in this invention for graph consistency verification and unreasonable relationship elimination, ultimately outputting a drawing relationship graph that satisfies the structural logic rules. In this scenario, the system discovered and eliminated 247 relationship edges that violated component design specifications, including issues such as beams not connected to columns, incorrect labeling positions, and abnormal circuit switch associations. The system automatically labeled the location information of each conflicting edge, significantly improving the review quality.

[0113] Finally, the system uses the Yin-Yang pair optimization algorithm to automatically adjust the hyperparameter combination and control training convergence. The optimal hyperparameter configuration is obtained in 15 iterations, which enables the analytical model to control the average processing time of each drawing to 12.6 seconds when processing complex building drawings, which is 31.5% shorter than the initial model. At the same time, the map integrity index is improved by 8.2%.

[0114] Table 1 is a summary of comparative data on the application effects of this invention in the task of analyzing residential building drawings in a certain area.

[0115] Table 1

[0116]

[0117] As can be seen from the table data, the method of this invention significantly outperforms traditional methods in terms of accuracy and consistency in drawing structure information extraction, logical relationship modeling, and anomaly detection. In scenarios involving mixed text and image parsing of drawings, its OCR recognition accuracy is improved by 12.4%, the graph logic recognition accuracy is improved by 11.7%, and the logical consistency score is improved by 19.2 points. Furthermore, it maintains stability and rapid response even in complex drawing processing scenarios. Particularly in the removal of illegal logical relationships, this invention achieves structural-level graph relationship verification for the first time and actually removes nearly 250 erroneous graph edges, significantly reducing the cost of manual review and verifying the invention's good practicality, scalability, and engineering promotion value.

[0118] On the other hand, the present invention also proposes an electronic device, comprising: at least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute any one of the multimodal data drawing logical relationship parsing methods.

[0119] On the other hand, the present invention also proposes a computer storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any one of the following multimodal data drawing logical relationship parsing methods.

[0120] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Dual Data SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus Direct RAM (RDRAM), Direct Memory Bus Dynamic RAM (DRDRAM), and Memory Bus Dynamic RAM (RDRAM). The various embodiments described in this specification are presented in a progressive manner, and similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, for embodiments of apparatus, devices, and non-volatile computer storage media, since they are substantially similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to the description of the method embodiments.

[0121] The above embodiments are merely illustrative examples and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multi-modal data graph logical relationship parsing method, characterized in that, The method comprises the following steps: Based on the image data and text data of the drawing, a node set is generated; Based on the spatial proximity, visual feature similarity and semantic correlation between the node sets, a cross-modal hyperedge set is generated; Based on the node set and the cross-modal hyperedge set, a hypergraph embedding input representation is generated; The hypergraph embedding input representation is input into an improved hypergraph self-attention network model to generate hyperedge logical relationship types and corresponding hyperedge confidence scores; Based on the hyperedge logical relationship types and the node set, a graph structure result is generated, which meets the structural legality requirement; Based on the hyperedge confidence scores, the graph structure result is automatically adjusted and converged to generate an optimal graph analysis model and a structured output result.

2. The multi-modal data graph logical relationship resolution method according to claim 1, characterized in that, Based on the image data and text data of the drawing, a node set is generated, specifically including: Performing grayscale, perspective distortion correction and structure enhancement operations on the drawing image data to obtain a clear structure image input; Based on the image input, region segmentation is performed to extract the graphic component region and the auxiliary annotation region, and the graphic node candidate region and the auxiliary node candidate region are constructed respectively; Performing vectorization analysis on the graphic node candidate region to extract contour shape, scale feature and spatial coordinate information, and generating a graphic node set; Performing template matching and image text recognition on the auxiliary node candidate region to extract auxiliary explanation content and its spatial position, and generating an auxiliary node set; Extracting characters, cleaning text and positioning space from the text data corresponding to the drawing, and combining the pre-trained language model embedding to generate a text node set; The graphic node set, the text node set and the auxiliary node set are uniformly identified and encoded to generate the node set, which contains node number, node type, spatial position and corresponding attribute information.

3. The multi-modal data graph logical relationship resolution method according to claim 2, characterized in that, Based on the spatial proximity, visual feature similarity and semantic correlation between the node sets, a cross-modal hyperedge set is generated, specifically including: Calculate the spatial proximity between the text node set, the graphic node set and the auxiliary node set, and construct an initial spatial adjacency matrix based on the Euclidean distance and layout density between nodes; Extract the local geometric features of the graphic nodes and the texture features of the auxiliary nodes, and fuse to generate the visual similarity score matrix between node pairs; Pre-training language model coding is performed on the text nodes, and the semantic correlation score matrix between node pairs is constructed based on the cosine similarity between context word vectors; The initial spatial adjacency matrix, the visual similarity score matrix and the semantic correlation score matrix are weighted and fused to construct a cross-modal candidate hyperedge set, and each candidate hyperedge connects multiple node pairs of different modal types; Attribute scores are calculated for each hyperedge in the cross-modal candidate hyperedge set, and the scoring factors include node modal combination weight, local spatial density constraint and semantic mutual information strength, and an initial confidence score list is generated; The initial confidence score list is input into a benchmark management optimization algorithm to generate the cross-modal hyperedge set.

4. The multi-modal data graph logical relationship resolution method according to claim 3, characterized in that, The initial confidence score list is input into a benchmark management optimization algorithm to generate the cross-modal hyperedge set, specifically including: According to the initial confidence score list, the top N% hyperedges are dynamically selected as the benchmark set, and their attribute distribution is marked as the local optimal reference index; For non-benchmark hyperedges, the feature gap between them and the benchmark set is measured to perform score offset measurement and confidence reassignment, and the score weight of hyperedges similar to the benchmark is improved; A hyperedge coverage matrix is constructed to detect whether each hyperedge has redundant coverage and blind area omission in different modal node spaces, and a detection result is generated; According to the detection result, the score threshold is dynamically adjusted to generate a global structure coverage quality; Based on the local optimal reference index and the global structure coverage quality, score updating and confidence iterative convergence are jointly performed to iteratively update the hyperedge score weight and the confidence threshold, and the cross-modal hyperedge set is generated.

5. The multi-modal data graph logical relationship resolution method of claim 2, wherein, The hypergraph embedding input representation is input into an improved hypergraph self-attention network model to generate hyperedge logical relationship types and corresponding hyperedge confidence, specifically including: Based on the improved hypergraph self-attention network model, a trainable hyperedge screening operation is performed on the cross-modal hyperedge set in the hypergraph embedding input representation to generate a screenable hyperedge set; Based on the modal perception multi-head attention mechanism, a plurality of sets of modal attention weights are constructed for the text node set, the graph node set and the auxiliary node set under the screenable hyperedge set, the feature coupling relationship between different modalities is modeled through parallel attention heads, and modal information fusion is completed to generate a fusion model; Based on the structure-preserving hyperedge attention mechanism, a component arrangement order constraint is introduced to the fusion model to model the order-dependent attention weight of the hyperedge order relationship and generate a structure logical sequence model; Based on the cross-modal auxiliary edge suppression mechanism, the structure logical sequence model is detected and low-confidence or abnormal cross-modal auxiliary edges are identified in the model output stage, and a dynamic decay mask matrix is constructed to weaken the attention weight, thereby generating hyperedge logical relationship types and corresponding hyperedge confidence.

6. The multi-modal data graph logical relationship resolution method according to any one of claims 2 to 5, characterized in that, Based on the node set and the cross-modal hyperedge set, a hypergraph embedding input representation is generated, specifically including: The text node set, the graph node set and the auxiliary node set are uniformly numbered, and node type labels are assigned to each node, the modal label, spatial coordinates and preliminary attribute feature information of each node are extracted, and a node attribute dictionary is constructed; Each hyperedge in the cross-modal hyperedge set is subjected to an associated node analysis operation, and the type combination and spatial coordinates of each associated node are obtained based on the node attribute dictionary to identify the cross-modal connection structure; Based on the spatial coordinates in the node attribute dictionary, a rotation-consistent position encoding mechanism is used to generate a structured position encoding vector for each node; The modal label, attribute feature and structured position encoding vector in the node attribute dictionary are fused to construct a node representation vector set; The node representation vector set and the cross-modal connection structure jointly construct the cross-modal hypergraph structure to generate the hypergraph embedding input representation.

7. The multi-modal data graph logical relationship resolution method according to any one of claims 2 to 5, characterized in that, Based on the hyperedge logical relationship type and the node set, a graph structure result is generated, and the graph structure result meets the structure legality requirement, specifically including: According to the hyperedge logical relationship type and the set of text nodes, the set of graphical nodes and the set of auxiliary nodes, a preliminary structured drawing logical relationship graph is constructed, and each edge in the preliminary structured drawing logical relationship graph is associated with a corresponding logical relationship type label and a confidence value; The preliminary structured drawing logical relationship graph is input into a differentiable integer programming constraint model, a node connection rule set is defined based on drawing design specifications and component assembly constraints, and a structured legality verification objective function in the form of integer programming is constructed; In the differentiable integer programming constraint model, a structure consistency solving operation is performed on each graph edge relationship, and all edge sets that do not satisfy the constraint rules are identified; The edge set that does not satisfy the constraint rules is removed, and a drawing logical relationship graph result after structure consistency verification and edge removal processing is output as the graph structure result.

8. The multi-modal data graph logical relationship resolution method according to any one of claims 2 to 5, characterized in that, Based on the hyperedge confidence, the graph structure result is automatically adjusted and converged to generate an optimal graph analysis model and a structured output result, which specifically includes: A joint optimization objective function is set, the accuracy index and the consistency index of the graph structure result are used as optimization objectives, and an adaptability function is constructed by combining the performance evaluation result of the improved hypergraph self-attention network model during the training process; The population structure of the yin-yang optimization algorithm is initialized to generate candidate individuals, and the candidate individuals include hyperedge thresholds, logical relationship confidence thresholds, node embedding dimensions, network learning rates and multi-task loss function weights; Based on the candidate individuals, two groups of opposite sub-populations are used to perform population splitting operations, multi-directional perturbations are performed on each dimension based on a perturbation strategy, and new solution sub-individuals are generated; The fitness of the new solution sub-individuals is evaluated to obtain a fitness evaluation result; According to the fitness evaluation result, a guided update operation of the negative sub-population is performed, sub-individuals with high distribution diversity and good fitness are retained, and an archiving mechanism is performed in the positive sub-population to gradually retain the global optimal solution through an alternating replacement method; After the convergence condition is met or the preset iteration number is reached, the optimal graph analysis model and the structured output result are generated.

9. An electronic device, comprising: It includes: At least one processor; A memory connected in communication with the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-modal data drawing logical relationship analysis method of any one of claims 1-8.

10. A medium characterized by, A computer program is stored, and the computer program is executed by a processor to implement the multi-modal data drawing logical relationship analysis method of any one of claims 1-8.

Images