A method for analyzing and judging the rationality of architecture diagrams based on multimodal feature fusion

Abstract

The present invention relates to the field of rationality judgment of intelligent architecture diagram design, and discloses an architecture diagram rationality analysis and judgment method based on multimodal feature fusion, which includes obtaining an architecture diagram to be analyzed; identifying and extracting an architecture diagram text part in the architecture diagram, and recording it; identifying and obtaining a feature map set of regions of interest of image elements through an R-CNN model, and recording it; establishing a corresponding relationship between the position and logical information of OCR module text information and R-CNN model image elements, and generating an extended architecture diagram text part; performing word segmentation and semantic coding on the extended architecture diagram text, and obtaining a word granularity feature vector set; performing semantic coding, and obtaining a word granularity feature vector set; generating a comprehensive feature vector of the same dimension; and calculating its semantic matching coefficient to judge whether the architecture diagram design to be analyzed meets the rationality requirements of the overall plan. The method can effectively evaluate whether the design and function of the architecture diagram meet the requirements of the overall plan, and improve the accuracy and efficiency of analysis and review.

Description

Technical Field

[0001] The present invention relates to the technical field of rationality judgment of intelligent architecture diagram design, and more specifically to a method for analyzing and judging the rationality of architecture diagrams based on multimodal feature fusion. Background Art

[0002] As a key visual tool, architecture diagrams are widely used in many fields such as engineering, architecture, information technology, project management and business. These diagrams describe in detail the components of a system, organization, process or project and their relationships through a combination of graphics and text, aiming to provide a high-level overview to help engineers, designers and other relevant personnel understand and communicate the design and implementation of complex systems more intuitively and deeply.

[0003] In traditional methods, the judgment of the rationality of architectural diagram design mainly relies on manual analysis and understanding, which requires professionals to carefully read the various components, lines and arrows in the architectural diagram and infer the relationship and function between them. However, this method has significant limitations. First, manual judgment is easily affected by personal subjective consciousness and experience, which may lead to inconsistency and deviation in judgment results. Second, for large-scale and complex architectural diagrams, manual judgment requires a lot of time and effort and is inefficient. This limitation is particularly obvious when dealing with architectural diagrams containing a large amount of text information and graphic elements.

[0004] With the development of information technology, it has become a trend to use computer technology to intelligently judge the rationality of architecture diagram design. Traditional automatic judgment methods usually rely on rules and templates, which are difficult to cope with complex and changeable architecture diagrams. In recent years, with the rapid development of artificial intelligence and deep learning technology, multimodal fusion technology based on large models has provided a new solution for judging the rationality of architecture diagram design. Multimodal fusion technology can combine the features of text and images for comprehensive analysis and processing, thereby significantly improving the accuracy and efficiency of judgment.

[0005] The architecture diagram design rationality judgment technology based on multimodal fusion can convert text and image features into the same feature space by using a pre-trained large model (such as CLIP) to capture the complex semantic relationship between the two. This technology can not only overcome the subjectivity and inefficiency of traditional methods, but also handle complex and changeable architecture diagrams to provide more reliable judgment results.

[0006] Therefore, there is an urgent need for an optimized architecture diagram design rationality judgment scheme based on multimodal fusion, which can overcome the limitations of traditional methods and achieve efficient and accurate judgment and review of architecture diagrams. Summary of the invention

[0007] The purpose of the present invention is to solve the problem that traditional automatic judgment methods usually rely on rules and templates and are difficult to deal with complex and changeable architecture diagrams.

[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0009] A method for analyzing and judging the rationality of an architecture diagram based on multimodal feature fusion includes the following steps:

[0010] S1. Obtain the architecture diagram to be analyzed, and obtain image data of the architecture diagram to be analyzed as input for subsequent processing;

[0011] S2. Use the OCR module to perform text recognition on the architecture diagram to be analyzed, obtain the text part of the architecture diagram, and record the position of each text information;

[0012] S3. Process the architecture diagram to be analyzed using the R-CNN model to obtain a set of feature maps of regions of interest of image elements in the architecture diagram, and record the position of each image element;

[0013] S4, establish the position and logical information correspondence between the OCR module text information and the R-CNN model image elements, and record the association between them;

[0014] S5, converting the positions and logical relationships of the image elements into text descriptions, and merging them with the text part of the architecture diagram to generate an expanded text part of the architecture diagram;

[0015] S6. Performing semantic encoding based on word granularity on the expanded text portion of the architecture diagram to obtain a set of word granularity feature vectors describing the text of the architecture diagram;

[0016] S7, performing semantic encoding based on word granularity on the architecture design rationality judgment text to obtain a set of architecture design rationality judgment text feature vectors;

[0017] S8. Use CLIP model to perform multimodal feature fusion and generate comprehensive feature vector;

[0018] S9, matching the comprehensive feature vector with the architecture design rationality judgment text feature vector, and calculating the semantic matching coefficient;

[0019] S10. Based on the comparison result of the semantic matching coefficient of the text feature vector and the preset threshold value of the comprehensive feature vector and the rationality of the architecture design, determine whether the architecture diagram to be analyzed meets the overall planning requirements.

[0020] Furthermore, establishing the position and logical information correspondence between the OCR module text information and the R-CNN model image elements includes the following steps:

[0021] For each text box recognized by the OCR module, check whether it overlaps with any image element bounding box recognized by the R-CNN model, using the intersection-over-union ratio as the metric;

[0022] If the IoU value between the text box and the image element bounding box exceeds the preset threshold, the association between them is recorded;

[0023] The correspondence between the large bounding box and the small bounding box is detected through the intersection relationship, and the hierarchical relationship is recorded to ensure that the hierarchical structure and module relationship in the architecture diagram are captured.

[0024] Furthermore, the OCR module and R-CNN model are used to extract the location information of text and image elements, which specifically includes the following steps:

[0025] Use the OCR module to perform text recognition on the architecture diagram, extract all the text parts in the architecture diagram, and record the position of each text information in the architecture diagram;

[0026] Use the R-CNN model to process the architecture diagram, identify and obtain the feature map set of the region of interest of the image element, and record the position of each image element;

[0027] According to the position and logical information correspondence established between the OCR module and the R-CNN model, the position and logical information of the image elements are converted into text descriptions and merged with the text part of the architecture diagram to generate the expanded text part of the architecture diagram.

[0028] Further, extracting the local semantic feature vector of the region of interest of the image element in the architecture diagram from the feature map of the region of interest of the image element in the architecture diagram specifically includes the following steps:

[0029] Upsampling the feature map of the region of interest of the image element in the architecture diagram to obtain the feature map of the region of interest of the image element in the upsampled architecture diagram;

[0030] Performing point convolution coding on the feature map of the region of interest of the image element in the upsampling architecture diagram to obtain the feature map of the region of interest of the image element in the channel modulation upsampling architecture diagram;

[0031] Two-dimensional convolution coding is performed on the feature map of the image element region of interest in the channel modulation upsampling architecture diagram to obtain a local semantic feature vector of the image element region of interest in the architecture diagram.

[0032] Furthermore, semantic encoding based on word granularity is performed on the text portion of the architecture diagram to obtain a set of word granularity feature vectors describing the text of the architecture diagram, which specifically includes the following steps:

[0033] After word segmentation processing is performed on the text portion of the architecture diagram, a set of granularity feature vectors of the text description words of the architecture diagram is obtained through a semantic encoder of the word embedding layer included in the large model;

[0034] The text of the rationality judgment of the architectural design is semantically encoded based on word granularity to obtain a set of feature vectors describing the word granularity of the text of the rationality judgment of the architectural design, including: after word segmentation processing is performed on the text of the rationality judgment of the architectural design, the text is passed through a semantic encoder of the word embedding layer included in the large model to obtain a set of feature vectors describing the word granularity of the text of the rationality judgment of the architectural design.

[0035] Furthermore, after word segmentation processing is performed on the text portion of the architecture diagram, a set of word granularity feature vectors describing the text of the architecture diagram is obtained through a semantic encoder of the word embedding layer included in the large model, which specifically includes the following steps:

[0036] Performing word segmentation processing on the architecture diagram and the architecture design rationality judgment text to convert the architecture diagram text portion into a word sequence consisting of multiple words;

[0037] The embedding layer of the semantic encoder including the word embedding layer maps each word in the word sequence to a word embedding vector to obtain a sequence of word embedding vectors;

[0038] The converter including the semantic encoder of the word embedding layer performs global context semantic encoding on the sequence of the word embedding vectors based on the converter idea to obtain a set of granular feature vectors of the text description words of the architecture diagram.

[0039] Further, based on the comparison between the semantic matching coefficient between the feature sequences and the preset threshold, determining whether the architecture diagram to be analyzed meets the overall planning requirements includes: in response to the semantic matching coefficient between the feature sequences being greater than the preset threshold, determining that the architecture diagram to be analyzed meets the overall planning requirements.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention provides a method for judging the rationality of architecture diagram design based on multimodal fusion, which utilizes the OCR module (optical character recognition) capability in a large model to identify text information in the architecture diagram, including but not limited to component names, labels, annotations, etc. In addition, computer vision technology is used to identify graphic elements in the architecture diagram, such as block diagrams, lines, arrows, etc., and understand their positions, logical relationships, and directional relationships. The present invention generates a comprehensive feature vector by multimodally fusing the feature vector of the annotated text with the feature vector of the image element, and matches it with the feature vector for judging the rationality of the architecture diagram design, calculates the semantic matching coefficient, and judges whether the architecture diagram to be analyzed meets the overall planning requirements based on the comparison result of the matching coefficient with the preset threshold. This method can realize the automation and comprehensive analysis of the architecture diagram, overcoming the subjectivity and inefficiency of traditional manual analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a method for judging the rationality of architecture diagram design based on feature fusion according to an embodiment of the present application;

[0043] Figure 2 A system architecture diagram for judging the rationality of architecture diagram design based on feature fusion according to an embodiment of the present application;

[0044] Figure 3 A block diagram of a system for judging the rationality of architecture diagram design based on feature fusion according to an embodiment of the present application;

[0045] Figure 4 This is the department information construction architecture diagram of a certain region’s production supervision department.

[0046] Attached figure numbers: 310, image acquisition module; 320, text recognition module; 330, region of interest extraction module; 335, position and logic information processing module; 337, position description generation module; 350, semantic encoding module; 360, comprehensive feature vector generation module; 370, semantic matching coefficient calculation module; 380, judgment result generation module. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] like Figure 1 , 2As shown, a method for judging the rationality of an architecture diagram based on multimodal fusion includes the following steps:

[0049] S1: Get the architecture diagram to be analyzed;

[0050] Get the image data of the architecture diagram to be analyzed as input for subsequent processing, such as Figure 4 Diagram of the departmental information construction architecture of a certain region’s production supervision department.

[0051] S2: Use the OCR module to perform text recognition on the architecture diagram to be analyzed, obtain the text part of the architecture diagram, and record the location of the text information;

[0052] The text recognition of the architecture diagram can be automated through the OCR module, avoiding the tedious process of manually recognizing the text information in the architecture diagram one by one, which can save time and labor costs; the OCR module is usually trained with a large amount of data and has high accuracy and recognition ability. Compared with manual recognition, the OCR module can recognize the text content in the architecture diagram more quickly and accurately. Specifically, the text part of the architecture diagram often contains important component names, labels, annotations and other key information, which is helpful for subsequent semantic understanding and comprehensive analysis and judgment of the architecture diagram. Record the position of each text information in the architecture diagram, including the coordinates of the upper left and lower right corners of the text box (such as x1, y1, x2, y2).

[0053] S3: Use the R-CNN model to identify and obtain a set of feature maps of the region of interest of the image element, and record the position of the image element;

[0054] In the architecture diagram, there are various graphic elements, such as boxes, lines, arrows, etc., which have interrelated positions, logical relationships and directional relationships. In order to capture the hidden correlation relationship between these graphic elements, the R-CNN model is used to process the architecture diagram to be analyzed, and a set of feature maps of the regions of interest of the image elements in the architecture diagram is obtained. The R-CNN model is a deep learning model for target detection, which can identify targets of interest in different regions of the image and extract their features. In the architecture diagram parsing, the R-CNN model can help identify various image elements in the architecture diagram, such as boxes, lines, arrows, etc. Therefore, by using the R-CNN model, the local regions of interest and implicit feature information of each element in the architecture diagram can be automatically learned and captured, which helps to understand the layout and connection mode of each element in the architecture diagram more accurately in the future, so as to understand the content and structure of the architecture diagram more comprehensively and meticulously, and provide important information for subsequent analysis and judgment on whether the architecture diagram meets the overall planning requirements. Record the position of each image element, including the bounding box coordinates of the element (such as x1, y1, x2, y2).

[0055] S4: Establish the position and logical information correspondence between the text information of the OCR module and the image elements of the R-CNN model;

[0056] For each text box identified by the OCR module, check whether it overlaps with any image element bounding box identified by the R-CNN model. Use the intersection over union (IoU) as a metric. The higher the IoU value, the greater the overlap between the two boxes. If the IoU value of the text box and the image element bounding box exceeds the preset threshold (for example, 0.5), it is considered that the text information has a corresponding relationship with the image element, and the association between them is recorded. If it is completely within the range of the large bounding box, the small bounding box is considered to be contained by the large bounding box. Next, record the hierarchical relationship and the relationship between the large bounding box and the small bounding boxes it contains. For example, if a large box contains four small boxes, it is considered that the content corresponding to the large box contains the content corresponding to the four small boxes. In this way, the hierarchical structure and module relationships in the architecture diagram can be captured.

[0057] S5: converting the positions and logical relationships of image elements into text descriptions;

[0058] According to the position and logical information correspondence established between the OCR module and the R-CNN model, the position and logical information of the image elements are converted into text descriptions. For example, the "Business Middle Platform" contains three modules, namely "Process and Rule Management", "User and Permission Management", and "System and Operation Management". These position and logical information descriptions are merged into the corresponding text part of the architecture diagram to generate the expanded text part of the architecture diagram. In this way, the spatial relationship and position and logical information of image elements can be converted into text descriptions that are easier to understand and process, providing richer information support for subsequent semantic analysis and comprehensive judgment.

[0059] S6: Perform word segmentation and semantic encoding on the architecture diagram text (annotation text + text description of location and logical relationship) to obtain a set of granular feature vectors of the architecture diagram text description words.

[0060] In order to capture the text semantic information in the architecture diagram, including important semantics such as component names, labels, and annotations, it is necessary to further perform word segmentation on the text portion of the architecture diagram, and then use the semantic encoder of the word embedding layer contained in the large model to obtain a set of word granularity feature vectors for the text description of the architecture diagram. By performing word segmentation on the expanded text portion of the architecture diagram, the text information can be converted into a more structured and processable form. Word segmentation helps identify key words and phrases in the text, thereby better understanding the meaning and context of the text. The specific method is as follows:

[0061] Performing word segmentation processing on the text portion of the architecture diagram to convert the text portion of the architecture diagram into a word sequence consisting of multiple words;

[0062] Use the embedding layer of the semantic encoder including the word embedding layer to map each word in the word sequence into a word embedding vector, and obtain a sequence of word embedding vectors;

[0063] A converter containing a semantic encoder of a word embedding layer is used to perform global context semantic encoding based on the converter idea on the sequence of word embedding vectors to obtain a set of word granularity feature vectors of the text description of the architecture diagram.

[0064] S7: Perform semantic encoding on the architecture design rationality judgment text based on word granularity to obtain a set of feature vectors of the architecture design rationality judgment text;

[0065] In order to capture the semantic information in the rationality judgment of the architecture diagram design, including important contents such as functional requirements, design standards, and evaluation indicators, it is necessary to further segment the rationality judgment of the architecture diagram design, and use the semantic encoder of the word embedding layer contained in the large model to obtain a set of word granularity feature vectors of the rationality judgment of the architecture diagram design. The specific method is as follows:

[0066] Performing word segmentation processing on the architecture design rationality judgment text to convert the architecture design rationality judgment text into a word sequence consisting of multiple words;

[0067] Use the embedding layer of the semantic encoder including the word embedding layer to map each word in the word sequence into a word embedding vector to obtain a sequence of word embedding vectors;

[0068] A converter containing a semantic encoder of a word embedding layer is used to perform global context semantic encoding on the sequence of word embedding vectors based on the converter idea, generating a set of word-granularity feature vectors for the text of architectural design rationality judgment.

[0069] The text information of the rationality judgment of the architecture diagram design consists of specific judgment rules. For example, in a certain project, in order to ensure the rationality of the architecture diagram design and the consistency of the overall plan, the architecture design rationality judgment rules include the following aspects:

[0070] 1. Consistency between the architecture diagram and text description: Ensure that each module, component, and element in the architecture diagram is consistent with its corresponding text description.

[0071] 2. Reasonableness of the framework diagram and consistency of tasks: The framework diagram should clearly reflect the planning task positioning and be consistent with the main tasks.

[0072] 3. Comprehensiveness and compliance of the docking relationship: Describe the docking relationship between the system and other related systems or platforms to ensure that all necessary connections are made and that relevant standards and regulatory requirements are met.

[0073] 4. Integrity of the overall framework: Review the integrity of the overall framework of the architecture diagram to ensure that common application support platforms, data aggregation and sharing, unified portal channels, etc. are clearly presented.

[0074] S8: Use CLIP model to perform multimodal feature fusion and generate comprehensive feature vector;

[0075] Use the CLIP pre-trained model to convert image elements into feature vectors, fuse the architectural diagram annotation text feature vector with the image element feature vector, and generate a comprehensive feature vector. The specific method is as follows:

[0076] Feature extraction: The text encoder and image encoder of the CLIP model are used to extract the annotated text feature vector and image element feature vector respectively.

[0077] Feature space alignment: The CLIP model maps text and image feature vectors into the same multimodal feature space, so that text and image features with the same semantics are close in this space. In this way, the CLIP model is able to capture the complex semantic relationship between text and images.

[0078] Feature fusion: The text feature vector and the image feature vector are weightedly fused to generate a comprehensive feature vector. The self-attention mechanism is used to weight the fusion of the annotated text feature vector and the image element feature vector, and the weights are dynamically assigned according to the content of the feature vector to capture the complex relationship between the text and the image. Specifically, the annotated text feature vector and the image element feature vector are input into the self-attention mechanism model respectively, and the weighted fused feature vector is obtained by calculating the attention score between each feature vector and other feature vectors. Finally, the fused feature vector is further processed by the multi-layer perceptron (MLP) to improve the richness and discrimination ability of the feature representation. The multi-layer perceptron consists of several fully connected layers, and the feature vector is nonlinearly transformed through nonlinear activation functions to enhance the expressive power of the features.

[0079] S9: Match the comprehensive feature vector with the architecture design rationality judgment text feature vector and calculate their semantic matching coefficient;

[0080] In the step of matching the comprehensive feature vector with the architecture diagram design rationality judgment feature vector to calculate their semantic matching coefficient, it is first necessary to ensure that the comprehensive feature vector and the architecture diagram design rationality judgment feature vector are in the same vector space. The specific method is as follows:

[0081] The two feature vector sets are standardized to make them comparable on the same scale. The standardization process can use the Z-score standardization method to scale the feature vectors to the same scale, thereby eliminating the magnitude differences between different features and ensuring the accuracy of the matching results.

[0082] The cosine similarity between the comprehensive feature vector and the architecture diagram design rationality judgment feature vector is calculated to evaluate their semantic matching degree. Cosine similarity is a commonly used indicator to measure the similarity between two vectors. It is calculated by calculating the ratio of the inner product of the two vectors to the product of their moduli. Specifically, for each comprehensive feature vector and architecture diagram design rationality judgment feature vector, their cosine similarity values ​​are calculated. The value ranges from -1 to 1. The closer the value is to 1, the higher the semantic similarity of the two vectors.

[0083] The calculated semantic matching coefficient is compared with the preset threshold to determine the matching degree between the comprehensive feature vector of the architecture diagram and the feature vector for judging the rationality of the architecture diagram design. If the semantic matching coefficient is higher than the preset threshold, it is considered that the comprehensive feature vector and the feature vector for judging the rationality of the architecture diagram design are semantically matched, indicating that the design and function of the architecture diagram meet the overall planning requirements; if the semantic matching coefficient is lower than the preset threshold, it indicates that there is a mismatch in some aspects, which requires further analysis and adjustment.

[0084] S10: Based on the comparison result of the semantic matching coefficient of the feature vector of the comprehensive feature vector and the feature vector of the architecture design rationality judgment text (that is, the review rule text) and the preset threshold, determine whether the architecture diagram to be analyzed meets the overall planning requirements;

[0085] In the step of determining whether the architecture diagram to be analyzed meets the overall planning requirements based on the comparison result between the semantic matching coefficient of the comprehensive feature vector and the preset threshold for judging the rationality of the architecture diagram design, it is first necessary to count and analyze all the semantic matching coefficients. The specific steps are as follows:

[0086] For each pair of comprehensive feature vectors and architecture diagrams, the semantic matching coefficient of the feature vector is compared with the preset threshold, and the number and proportion of feature pairs with matching coefficients higher than the threshold are counted. The preset threshold is determined according to the actual application scenario and requirements. Usually, a critical value that can effectively distinguish between matches and mismatches is selected to ensure the accuracy and reliability of the evaluation results.

[0087] Based on the statistical results of the matching coefficient, a comprehensive evaluation is performed. For matching pairs that are higher than the preset threshold, they are considered to be highly consistent in semantics, indicating that the corresponding parts in the architecture diagram meet the requirements of the review rules; while for matching pairs that are lower than the preset threshold, they are considered to be different in semantics, indicating that there may be parts in the architecture diagram that do not meet or deviate from the review rules. Through a comprehensive analysis of the matching results, it can be determined whether the architecture diagram as a whole meets the planning requirements, and which specific parts need further optimization and adjustment.

[0088] Finally, the evaluation results are summarized and a detailed review report is formed. The review report should include the following: statistical data on the overall matching degree, parts that meet the planning requirements, parts that do not meet the planning requirements and their specific reasons, suggestions for improvement, etc. In this way, not only can the overall planning compliance of the architecture diagram be clarified, but also targeted improvement measures can be provided to help designers make effective adjustments and optimizations in subsequent work. Through a scientific and systematic review process, it is ensured that the design and implementation of the architecture diagram can better meet the expected plan and provide strong support for the successful implementation of the project.

[0089] The present invention has the following beneficial effects:

[0090] Automated processing: The OCR module automatically recognizes text information in the architecture diagram, avoiding the tedious process of manual recognition one by one, saving time and labor costs.

[0091] Accuracy and efficiency: Leverage the high accuracy and recognition capabilities of large models trained with large amounts of data to quickly and accurately identify text and image element information in architectural diagrams.

[0092] Multimodal fusion: Through multimodal fusion technology, the feature vector of the annotated text of the architecture diagram and the feature vector of the image element are effectively fused to generate a feature vector containing comprehensive semantic information.

[0093] Comprehensive evaluation: Use technologies such as self-attention mechanism and multi-layer perceptron to enhance the richness and discrimination ability of feature representation, and ensure that the design and functions of the architecture diagram meet the overall planning requirements.

[0094] Semantic matching: The system evaluates the design rationality of the architecture diagram by calculating the semantic matching coefficient between the comprehensive feature vector and the feature vector of the architecture diagram design rationality judgment, and comparing it with the preset threshold.

[0095] Comprehensive analysis: Capture the semantic features of elements in the architecture diagram from multiple scales and levels, conduct systematic evaluation and judgment, and provide a detailed review report, including the parts that meet and do not meet the planning requirements, their specific reasons, and improvement suggestions.

[0096] Scientific support: Provide in-depth support for system design and planning, ensure that the design and functions of the architecture diagram can better meet the expected planning, and provide strong guarantee for the successful implementation of the project.

[0097] like Figure 3 As shown, the present invention also proposes a system for judging the rationality of architecture diagram design based on feature fusion, including:

[0098] Image acquisition module 310: used to acquire image data of the architecture diagram to be analyzed as input for subsequent processing.

[0099] The text recognition module 320 is used to perform text recognition on the architecture diagram to be analyzed using the OCR module, extract all the text parts of the architecture diagram, including component names, labels, annotations, etc., and record the location of each text information.

[0100] The region of interest extraction module 330 is used to process the architecture diagram to be analyzed using the R-CNN model, identify and obtain a set of region of interest feature maps of image elements, such as boxes, lines, arrows, etc., and record the position of each image element.

[0101] Position and logical information processing module 335: used to establish the position and logical information correspondence between the text information of the OCR module and the image elements of the R-CNN model. For each text box recognized by the OCR module, check whether it overlaps with any image element bounding box recognized by the R-CNN model. Use the intersection over union (IoU) as a metric. If the IoU value of the text box and the image element bounding box exceeds the preset threshold, it is considered that the text information has a corresponding relationship with the image element, and the association between them is recorded.

[0102] Position description generation module 337: According to the position and logic information correspondence established between the OCR module and the R-CNN model, the position and logic information of the image elements are converted into text descriptions. For example, the "Business Middle Station" contains 10 modules, and the "Data Middle Station" is located in the middle left position of the figure. These position and logic information descriptions are merged into the corresponding architecture diagram text part to generate the expanded architecture diagram text part.

[0103] Semantic coding module 350: used to perform word segmentation and semantic coding on the expanded architecture diagram text part and the architecture diagram design rationality judgment part, and obtain a word granularity feature vector set of the architecture diagram text description and a word granularity feature vector set of the architecture diagram design rationality judgment description.

[0104] Comprehensive feature vector generation module 360: used to fuse the feature vector set of architectural diagram annotation text with the feature vector set of image element interest region to generate a comprehensive feature vector. Specifically, it includes three steps: feature extraction, feature space alignment and feature fusion. The fused feature vector is processed by self-attention mechanism and multi-layer perceptron (MLP) to improve the richness and discrimination ability of feature representation.

[0105] Semantic matching coefficient calculation module 370: used to match the comprehensive feature vector with the architecture diagram design rationality judgment feature vector and calculate their semantic matching coefficient. Ensure that the comprehensive feature vector and the architecture diagram design rationality judgment feature vector are in the same feature space for effective matching. Calculate the cosine similarity of each pair of feature vectors to evaluate their semantic matching degree.

[0106] The judgment result generation module 380 is used to determine whether the architecture diagram to be analyzed meets the overall planning requirements based on the comparison between the semantic matching coefficients between feature sequences and the preset threshold. All semantic matching coefficients are counted and analyzed, the overall matching degree of the architecture diagram is comprehensively evaluated, and a detailed review report is generated, including overall matching degree statistics, parts that meet the planning requirements, parts that do not meet the requirements and their specific reasons, improvement suggestions, etc.

[0107] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. The patent protection scope of the present invention shall be based on the claims. All equivalent structural changes made using the contents of the description and drawings of the present invention should also be included in the protection scope of the present invention.

Claims

1. A method for analyzing and judging the rationality of an architecture diagram based on multimodal feature fusion, characterized in that: The following steps are involved: S1. Obtain the architecture diagram to be analyzed, and obtain image data of the architecture diagram to be analyzed as input for subsequent processing; S2. Use the OCR module to perform text recognition on the architecture diagram to be analyzed, obtain the text part of the architecture diagram, and record the position of each text information; S3. Process the architecture diagram to be analyzed using the R-CNN model to obtain a set of feature maps of regions of interest of image elements in the architecture diagram, and record the position of each image element; S4, establish the position and logical information correspondence between the OCR module text information and the R-CNN model image elements, and record the association between them; S5, converting the positions and logical relationships of the image elements into text descriptions, and merging them with the text part of the architecture diagram to generate an expanded text part of the architecture diagram; S6. Performing semantic encoding based on word granularity on the expanded text portion of the architecture diagram to obtain a set of word granularity feature vectors describing the text of the architecture diagram; S7, performing semantic encoding based on word granularity on the architecture design rationality judgment text to obtain a set of architecture design rationality judgment text feature vectors; S8. Use CLIP model to perform multimodal feature fusion and generate comprehensive feature vector; The specific method is as follows: Feature extraction: Use the CLIP model to extract the feature vectors of the image elements in the region of interest in the architecture diagram; Feature space alignment: The CLIP model maps the set of granular feature vectors of the text description words in the architecture diagram and the feature vectors of the image element regions of interest in the architecture diagram to the same multimodal feature space, so that the text and image features with the same semantics are close in this space; feature Fusion: weighted fusion of the set of granular feature vectors of the text description words in the architecture diagram and the feature vectors of the image elements in the region of interest in the architecture diagram to generate a comprehensive feature vector; S9, matching the comprehensive feature vector with the architecture design rationality judgment text feature vector, and calculating the semantic matching coefficient; S10. Based on the comparison result of the semantic matching coefficient of the text feature vector and the rationality of the architecture design judgment text feature vector and the preset threshold, determine whether the architecture diagram to be analyzed meets the overall planning requirements.

2. The method for analyzing and judging the rationality of an architecture diagram based on multimodal feature fusion according to claim 1 is characterized in that: Establishing the position and logical information correspondence between the OCR module text information and the R-CNN model image elements includes the following steps: For each text box recognized by the OCR module, check whether it overlaps with any image element bounding box recognized by the R-CNN model, using the intersection-over-union ratio as the metric; If the IoU value between the text box and the image element bounding box exceeds the preset threshold, the association between them is recorded; The correspondence between the large bounding box and the small bounding box is detected through the intersection relationship, and the hierarchical relationship is recorded to ensure that the hierarchical structure and module relationship in the architecture diagram are captured.

3. The method for analyzing and judging the rationality of an architecture diagram based on multimodal feature fusion according to claim 2 is characterized in that: The OCR module and R-CNN model are used to extract the location information of text and image elements. The specific steps include: Use the OCR module to perform text recognition on the architecture diagram, extract all the text parts in the architecture diagram, and record the position of each text information in the architecture diagram; Use the R-CNN model to process the architecture diagram, identify and obtain the feature map set of the region of interest of the image element, and record the position of each image element; According to the position and logical information correspondence established between the OCR module and the R-CNN model, the position and logical information of the image elements are converted into text descriptions and merged with the text part of the architecture diagram to generate the expanded text part of the architecture diagram.

4. The method for analyzing and judging the rationality of an architecture diagram based on multimodal feature fusion according to claim 3 is characterized in that: Extracting a local semantic feature vector of an image element region of interest in the architecture diagram from the feature map of the image element region of interest in the architecture diagram specifically includes the following steps: Upsampling the feature map of the region of interest of the image element in the architecture diagram to obtain the feature map of the region of interest of the image element in the upsampled architecture diagram; Performing point convolution coding on the feature map of the region of interest of the image element in the upsampling architecture diagram to obtain the feature map of the region of interest of the image element in the channel modulation upsampling architecture diagram; Two-dimensional convolution coding is performed on the feature map of the image element region of interest in the channel modulation upsampling architecture diagram to obtain a local semantic feature vector of the image element region of interest in the architecture diagram.

5. The method for analyzing and judging the rationality of an architecture diagram based on multimodal feature fusion according to claim 4 is characterized in that: The text part of the architecture diagram is semantically encoded based on word granularity to obtain a set of word granularity feature vectors describing the architecture diagram text, specifically including the following steps: After word segmentation processing is performed on the text part of the architecture diagram, a set of granularity feature vectors of the text description words of the architecture diagram is obtained through a semantic encoder of a word embedding layer included in the large model; The text of the rationality judgment of the architecture design is semantically encoded based on word granularity to obtain a set of feature vectors of the word granularity describing the text of the rationality judgment of the architecture design, including: performing word segmentation processing on the text of the rationality judgment of the architecture design and then passing it through a semantic encoder of a word embedding layer included in a large model to obtain a set of feature vectors of the word granularity describing the text of the rationality judgment of the architecture design.

6. The method for analyzing and judging the rationality of an architecture diagram based on multimodal feature fusion according to claim 5 is characterized in that: After word segmentation processing is performed on the text portion of the architecture diagram, a set of word granularity feature vectors describing the text of the architecture diagram is obtained through a semantic encoder of the word embedding layer included in the large model, which specifically includes the following steps: Performing word segmentation processing on the architecture diagram and the architecture design rationality judgment text to convert the architecture diagram text portion into a word sequence consisting of multiple words; Using the embedding layer of the semantic encoder including the word embedding layer, each word in the word sequence is mapped into a word embedding vector to obtain a sequence of word embedding vectors; The converter including the semantic encoder of the word embedding layer performs global context semantic encoding on the sequence of the word embedding vectors based on the converter idea to obtain a set of granular feature vectors of the text description words of the architecture diagram.

7. The method for analyzing and judging the rationality of an architecture diagram based on multimodal feature fusion according to claim 6 is characterized in that: Based on a comparison between a semantic matching coefficient between feature sequences and a preset threshold, determining whether the architecture diagram to be analyzed meets the overall planning requirements includes: in response to the semantic matching coefficient between the feature sequences being greater than the preset threshold, determining that the architecture diagram to be analyzed meets the overall planning requirements.

Images