An engineering drawing intelligent auditing method and system based on a multi-modal fusion model

Abstract

The application discloses an engineering drawing intelligent auditing method and system based on a multi-modal fusion model, and belongs to the technical field of computer-aided design. The application collects multi-modal data of all factors such as drawing geometry, labels, texts and process symbols based on a CAD software open API and analyzes the data in a structured manner; a multi-modal fusion verification model combining a convolutional neural network and a Transformer is built, a multi-head cross-modal attention mechanism is adopted to identify unstructured graph elements with high precision and infer cross-modal hidden errors between graphs and texts; a visual configurable rule engine is built based on an extensible markup language and an industrial design knowledge graph, national standards, industry standards and enterprise dynamic updating drawing standards are adapted; drawing error early risk warning is realized by relying on a gradient boosting decision tree algorithm, a forced verification interception mechanism is set at a CAD drawing saving node, and a full-link drawing quality closed-loop management and control is built. The application can be deeply integrated with CAD software and industrial systems such as PLM and MES, is suitable for the standardized automatic auditing of engineering drawings in the fields of automobiles, aerospace and high-end mechanical equipment, and can greatly reduce the cost of manual drawing auditing and reduce design and production rework.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of computer-aided design (CAD) software secondary development and industrial artificial intelligence. Specifically, it relates to a method and system for achieving standardized automatic review of engineering drawings by secondary development based on CAD software open API, integrating visual and text features, and building a multimodal fusion verification model. Background Technology Current state of technology

[0002] Engineering drawings are the core technical carrier connecting the three-dimensional design of products with the processing and manufacturing in the workshop. The standardization of elements such as drawing annotation specifications, view expression, tolerance matching, process symbols, and layer line types directly determines the processing accuracy of parts, the assembly quality of products, and the overall cost of research and development and production.

[0003] In existing technologies, engineering drawing review mainly falls into four technical categories:

[0004] (1) Technical route based on built-in rule verification tools in CAD software. Commercial CAD software such as Siemens NX and SolidWorks rely on fixed logic to perform hard rule verification on single elements, and can only identify basic errors such as layers, line types, and fonts. Chinese patent CN116227124A discloses a method for rapid modeling and drafting of heat exchangers. Its drawing inspection module can only complete the basic error verification of component drawings and assembly drawings, such as missing views and incorrect welding point annotations. It cannot establish the correlation logic between the visual features of the drawing graphics and the semantics of the annotation text, and it is difficult to identify unstructured elements such as section lines, welding symbols, and thread annotations.

[0005] (2) Drawing review technology based on deep learning image recognition. The CADTransformer framework disclosed at CVPR 2022 uses the Transformer structure to mark up drawing primitives and realizes semantic recognition of line-level symbols. It has good recognition performance on the FloorPlanCAD dataset. This type of solution only uses the drawing image as a single input source and cannot read the structured raw data such as the CAD bottom layer, object identifiers, and part attributes. The error localization accuracy is insufficient and it does not have the ability to recognize cross-modal semantic association errors.

[0006] (3) Patent CN121352737A discloses a method for reviewing engineering design specifications based on a multimodal large model. This method integrates multimodal data from drawings, specification texts, design tables, and calculation sheets to construct structured information for compliance review. However, this solution is not deeply integrated with the underlying CAD API and cannot read native data such as part attributes, view identifiers, and layer structures within the drawings. It cannot achieve precise positioning of drawing elements, in-situ error reporting on the CAD interface, or mandatory control at the design end, and lacks the closed-loop control capability for the entire design-verification-data accumulation process.

[0007] (4) Knowledge Graph-Based Drawing Management Technology Approach: Existing power engineering drawing knowledge graph solutions rely on engineering drawing recognition intelligence to identify transmission line drawings and build a cost knowledge graph to complete the engineering cost compilation. This type of solution focuses on cost statistics and digital management of archives after drawing recognition, but does not cover pre-design quality control links such as real-time verification during the drawing design stage and mandatory interception of drawing storage. Problems with existing technology

[0008] Based on the above-mentioned existing technologies, there are six major technical pain points in the current field of engineering drawing review.

[0009] (1) Manual drawing review is inefficient: a single complex assembly drawing takes tens of minutes to complete review, and the review cycle for a large number of drawings is long, which delays the product development and iteration progress.

[0010] (2) Inconsistent implementation of manual review standards: Different engineers have different understandings of national standards, industry norms and internal enterprise process standards, resulting in inconsistent judgments of errors and omissions in similar drawings.

[0011] (3) High rate of manual omissions: Errors such as dimensional chain closure, view and annotation logic conflict, tolerance matching abnormality, and missing process symbols are easily missed, which leads to batch rework after the drawings are sent to the workshop.

[0012] (4) Traditional CAD rule-based inspection tools have limited capabilities: they can only identify basic errors in layers, line types, and fonts, but cannot analyze the correlation logic between graphic visual features and annotation text semantics, and cannot identify unstructured elements such as section lines, welding symbols, and thread annotations.

[0013] (5) Existing multimodal AI drawing recognition solutions are not well integrated with CAD software: they only independently complete drawing image recognition, do not deeply connect with the underlying CAD API, cannot read the internal structured data of the drawings, and are difficult to accurately locate errors and achieve on-site control at the design end.

[0014] (6) Lack of end-to-end closed-loop management capability for design-verification-data accumulation: Existing solutions cannot establish a complete management chain encompassing real-time design assistance, pre-event risk warning, in-event intelligent verification, forced interception of data storage, and post-event knowledge accumulation and iteration. In existing technologies, multimodal visual-text fusion AI technology and CAD secondary development technology are disconnected, and there is currently no integrated solution to achieve integrated acquisition of multimodal information in the native CAD environment and cross-platform multimodal fusion verification of models. Summary of the Invention

[0015] A method for intelligent review of engineering drawings based on a multimodal fusion model includes the following steps.

[0016] Step S1: Based on the CAD software's open API, complete the collection and structured parsing of all elements of the engineering drawings to construct a standardized drawing dataset. Call the CAD software's open API, relying on the unique identifiers of internal CAD objects (NX software tags), to traverse all native elements of the drawing, collecting multimodal information such as geometric view elements, dimensional tolerance annotations, technical requirement annotations, welding / thread / surface roughness process symbols, projected view features, part material properties, layers, line types, line widths, and font styles. Classify and structure the collected visual image data, text semantic data, and drawing structural data, synchronously recording the unique coordinate tags of various elements and their annotations in the CAD drawing, achieving accurate in-situ location of erroneous elements. The information collection module runs silently in the background, relying on the CAD software's event listening mechanism and background service process execution, without occupying CAD front-end design resources or interfering with the designer's drawing operations.

[0017] Step S2: Construct a multimodal fusion verification model based on convolutional neural networks and Transformers to complete multimodal feature extraction of drawings and cross-modal association error inference. The multimodal fusion verification model consists of three parts: an industrial primitive recognition sub-model, a text semantic analysis sub-model, and a cross-modal fusion verification sub-model. (1) The industrial primitive recognition sub-model adopts a lightweight convolutional neural network to complete feature extraction and classification recognition of unstructured drawing primitives such as section lines, datum symbols, threads, welding symbols, and geometric tolerance frames; the sub-model is trained based on the labeled industrial drawing sample dataset. (2) The text semantic analysis sub-model is based on the Transformer encoder and is combined with the industrial NLP dedicated corpus to parse technical requirements, size text, material notes, title block semantics, and extract tolerance values, processing requirements, assembly constraints key semantic feature vectors. (3) The cross-modal fusion verification sub-model introduces a multi-head cross-modal attention mechanism, mapping the visual feature vectors output by the convolutional neural network and the textual semantic feature vectors output by the Transformer to the same low-dimensional feature space. Through feature similarity and association weights, it completes multi-modal feature fusion inference and identifies cross-modal logical errors such as non-closed dimension chains, mismatch between tolerances and view structures, missing view annotations, and conflicts between process symbols and technical requirements. The model is equipped with a lightweight inference engine, which enables local deployment through model quantization, knowledge distillation, and network pruning, reducing the computing power requirements of the equipment.

[0018] Step S3: Construct a configurable rule engine based on XML and an industrial design knowledge graph. Combine this with a multimodal fusion verification model to generate drawing compliance judgments, error classifications, and intelligent correction suggestions. The national standard GB / T 4458.4 for mechanical drawing, industry drawing specifications, and enterprise-customized process standards are encapsulated into an XML structured file. The XML file defines check item identifiers, verification logic, applicable drawing types, standard clauses, and default error levels. An industrial design knowledge graph is built, with entity nodes including standard clause entities, drawing error type entities, standard rectification scheme entities, component category entities, and designer entities. Relationships between entities include constraint relationships, applicability relationships, substitution relationships, and rectification relationships. The rule engine is configured with a front-end visual interface, allowing staff to dynamically add or delete check items, adjust error risk levels, modify verification logic, and update new enterprise drawing standards. Configuration changes take effect via hot updates through XML files, without requiring modification of the underlying code or recompiling the system. The primitive recognition results and cross-modal anomaly information output by the multimodal fusion verification model are input into the rule engine. The rule engine matches the knowledge graph standard constraints to complete the compliance judgment, locates the erroneous primitives, classifies them into three levels of error: general, serious, and fatal, and outputs standardized rectification suggestions based on knowledge graph retrieval.

[0019] Step S4: Construct a drawing error trend prediction model based on historical drawing review big data to achieve early warning of high-risk design errors. Collect all historical drawing review data from the enterprise, extract multi-dimensional features such as error type, error frequency, designer number, component type, product module, and drawing complexity, and construct a structured training dataset. Use the gradient boosting decision tree algorithm to train the error trend prediction model; the model input is the current drawing feature vector, and the output is the predicted probability value of various errors. During the designer's drawing process, the system calls the prediction model in real time, and for high-frequency fatal errors and high-risk annotation pop-up warnings, it simultaneously pushes standard annotation templates and similar historical error cases to avoid batch design defects in advance during the drawing stage.

[0020] Step S5: Embed a mandatory verification mechanism in the CAD drawing saving node to build a closed-loop control of the entire drawing quality chain, and simultaneously complete the accumulation of review experience and knowledge and model self-iteration.

[0021] (1) Listen to the FileSave drawing saving event of CAD software and automatically start the complete intelligent review process of S1~S3; if the rule engine determines that the drawing has fatal or serious non-compliant errors, directly intercept the drawing saving, highlight all error element Tag coordinates in the native CAD interface, and display the error level, violation standard, and standardized correction plan; after all errors in the drawing are rectified and the second verification is passed, the saving restriction is lifted; (2) Store the audit record, error type, rectification plan, and basic information of drawings into the industrial design knowledge graph to expand the standard case library; retrain the multimodal fusion verification model and error trend prediction model by using the newly added audit data increment on a monthly / quarterly basis to continuously improve the model recognition accuracy and realize the autonomous iterative update of the rule library.

[0022] The above steps form a full-chain intelligent quality control system for drawings, which includes "real-time design assistance, pre-event risk prediction, in-event AI intelligent verification, storage of mandatory interception verification, and post-event review experience accumulation and iteration". Beneficial effects

[0023] (1) Significantly improved review efficiency: Relying on the local deployment of a lightweight AI inference engine, the time required for a single review of complex assembly drawings is greatly reduced, the overall review efficiency of drawings is improved by more than 90%, and the review cycle of large batches of drawings is significantly shortened.

[0024] (2) Significantly improved audit accuracy: Relying on the multimodal fusion verification model to identify cross-element logical association errors through cross-modal fusion reasoning, the rate of missed defects in drawings has been greatly reduced, achieving full coverage audit of all elements of drawings and solving the problems of missed defects in manual drawing review and inconsistent standard execution.

[0025] (3) Flexible and convenient standard adaptation: The rule engine supports front-end visual configuration, and enterprises can independently update national standards, industry standards and internal process standards without the need for secondary development by developers, adapting to the enterprise's standard iteration needs.

[0026] (4) Unique cross-modal error recognition capability: Breaking through the limitations of traditional single rule verification, it relies on the convolutional neural network and Transformer multi-head cross-modal attention fusion mechanism to identify latent errors such as text-image logic conflict, filling the technical gap of existing CAD inspection tools.

[0027] (5) Full-process source quality control: covering a complete closed loop of pre-drawing warning, real-time design assistance, forced interception of storage, and post-event knowledge self-iteration, reducing drawing defects from the design source and reducing workshop rework, material scrapping, and cross-departmental communication costs.

[0028] (6) Strong software compatibility and non-intrusive operation: It is developed based on the official open API of CAD, deeply integrated with the internal program of CAD, does not tamper with the underlying software code, does not interfere with the designer's drawing, modeling and output operations, and is compatible with mainstream CAD software versions on the market.

[0029] (7) Highly scalable and adaptable to multiple industries: It has reserved standardized interfaces to connect with upstream and downstream industrial software such as PLM, MES, and CAPP, and is suitable for standardized drawing review scenarios in multiple fields such as automobiles, aerospace, engineering machinery, and precision equipment. Attached Figure Description

[0030] Figure 1. Schematic diagram of the five-layer overall architecture of the intelligent auditing system of the present invention.

[0031] Figure 2. Schematic diagram of the internal structure of the multimodal fusion verification model of the present invention.

[0032] Figure 3. Schematic diagram of the closed-loop process of intelligent review of drawings throughout the entire process of this invention. Detailed Implementation

[0033] Example 1: Intelligent review of standardized NX engineering drawings for automotive parts.

[0034] Step 1: Multimodal Information Acquisition. A background acquisition plugin was developed based on the NX Open C++ API. The plugin starts silently in the background with the NX software, without pop-up windows interfering with the designer's drawing. Utilizing the unique identifier of NX's internal Object Tag, it traverses all objects in the drawing: three-view drawings, sectional views, enlarged details, geometric elements; dimensions, geometric tolerances, surface roughness annotations; title blocks, technical requirements, material notes; process symbols such as datum, threads, and welding; layers, line types, fonts, and automotive component material attributes. Image pixels, text strings, element tag coordinates, and layer parameters are collected and uniformly converted into JSON structured data, stored in a Redis cache, and persistently stored in a MySQL database.

[0035] Step 2: Multimodal fusion verification and inference employs a lightweight convolutional neural network to train a dedicated primitive recognition model for automotive component drawings. The training dataset contains tens of thousands of cross-sectional, datum, and welding symbol samples from body, chassis, and transmission drawings. An industrial-grade fine-tuned Transformer is used to build an NLP semantic parsing module to extract keywords related to tolerance ranges, heat treatment requirements, and assembly constraints. Visual and textual feature vectors are fused through a multi-head cross-modal attention mechanism to identify implicit errors such as non-closed dimension chains, mismatches between views and annotations, and conflicts between process symbols and technical requirements. The system is equipped with a local lightweight inference engine, and the complete verification of a complex automotive chassis assembly drawing with 400 annotations takes approximately 42 seconds.

[0036] Step 3: Configurable Rule Engine Compliance Judgment. Import GB / T 4458.4 national standard for mechanical drawing, automotive industry drawing specifications, and enterprise component tolerance standards XML rule files to build a dedicated industrial design knowledge graph for automotive components, storing standard constraints, error types, and corresponding rectification plans. A drag-and-drop visual configuration page allows process engineers to add specific verification rules such as "mandatory annotation of roughness for automotive stamping parts" and "view alignment of chassis welding symbols," and customize three levels of errors: fatal, severe, and general. Abnormal data output from the multimodal fusion verification model is fed into the rule engine for matching standards, automatically generating rectification suggestions. Example: "The main view lacks the datum A annotation; it is recommended to add a datum symbol to the bottom contour and match the corresponding dimensional tolerance."

[0037] Step 4: Error Trend Pre-warning Model Training. Collect nearly 20,000 historical automotive parts drawings from the past 3 years for review, extract multi-dimensional features such as vehicle model, part category, designer number, error type, and error frequency, and train a gradient boosting decision tree risk prediction model. The model is called in real time during the designer's drawing process. If the current drawing is similar in features to historical high-frequency scrapped parts drawings, a pop-up warning will be issued: "82% of this type of shell drawing has missing geometric tolerances. It is recommended to check the geometric annotations in advance." The automotive industry standard annotation template will be pushed simultaneously.

[0038] Step 5: NX Saves Forced Verification and Knowledge Accumulation Iteration. Monitor the NX software's FileSave event; clicking save automatically initiates the entire review process. Drawings with fatal errors such as missing tolerances or views are directly intercepted and saved; all non-compliant element coordinates are highlighted in red in the NX drawing window. Saving only occurs after all errors have been rectified and secondary verification has passed. The review record, error cases, and rectification plans are automatically written into the knowledge graph. The multimodal fusion verification model is incrementally retrained monthly to continuously improve drawing recognition accuracy.

Claims

1. A method for intelligent review of engineering drawings based on a multimodal fusion model, characterized in that, Includes the following steps: S1. Multimodal information acquisition steps: Based on the open API of CAD software, call the underlying interface of CAD, rely on the unique identifier of the internal object of CAD to traverse all elements of the drawing, collect multimodal information of geometric figures, dimension annotations, annotation text, process symbols, projection views, part attributes, and layer line types, complete the accurate coordinate positioning and structured analysis of elements, and generate a standardized drawing dataset. S2. Multimodal fusion verification steps: Construct a multimodal fusion verification model based on convolutional neural networks and Transformers; extract visual features of unstructured primitives in drawings using lightweight convolutional neural networks; parse drawing annotations and dimension text semantics using Transformers combined with an industrial natural language processing module; fuse visual features and text semantic features using a multi-head cross-modal attention mechanism to infer and identify cross-modal association errors such as non-closed dimension chains, mismatch between tolerances and views, and conflict between graphic and textual expressions; S3. Configurable rule engine compliance judgment steps: Build a configurable rule engine based on extensible markup language files and industrial design knowledge graph, and input national standards, industry drawing specifications, and enterprise process customization standards; The rule engine receives the verification results output by the multimodal fusion verification model, compares them with the standard constraints in the knowledge graph to complete the compliance judgment of the drawings, the location of error elements, the classification of error risk levels, and outputs standardized intelligent correction suggestions; S4. Pre-warning steps for drawing errors: Based on the company's historical drawing review data, extract multi-dimensional features such as error type, drawing type, designer, and component category, and use the gradient boosting decision tree algorithm to train the error trend prediction model; During the design drawing stage, the model is called in real time to output high-risk error warning information, and design optimization guidance is pushed simultaneously; S5. CAD Save Forced Verification and Knowledge Accumulation Steps: Listen for CAD software drawing save trigger events and automatically start the complete intelligent review process from steps S1 to S3; If a drawing contains non-compliant items, the drawing will be intercepted and saved, and all erroneous elements will be highlighted in the native CAD interface. The drawing can only be saved after rectification and verification are passed. The audit data, error cases, and rectification plans will be stored in the industrial design knowledge graph. The multimodal fusion verification model and error trend prediction model will be retrained using newly added audit data at preset cycles to achieve self-iterative optimization of the models. Through the above steps, a full-chain drawing quality control system of "real-time design assistance, pre-risk warning, in-process intelligent verification, forced interception of saving, and post-event knowledge storage and iteration" will be constructed.

2. The intelligent review method for engineering drawings based on a multimodal fusion model according to claim 1, characterized in that: The CAD software open API mentioned in step S1 is the NX Open API, and the unique identifier of the object is the NX internal tag identifier; the acquisition process runs silently in the background and is implemented through the NX event listening mechanism or background service process, without interfering with the normal design operation of the CAD software.

3. The intelligent review method for engineering drawings based on a multimodal fusion model according to claim 1, characterized in that: The lightweight convolutional neural network mentioned in step S2 is a MobileNetV3, EfficientNet, or ResNet series network; the Transformer is BERT, RoBERTa, or an industrial-specific pre-trained language model; the multi-head cross-modal attention mechanism maps visual primitive feature vectors and text semantic feature vectors to the same low-dimensional feature space, and realizes cross-modal fusion inference by calculating feature similarity and association weights.

4. The intelligent review method for engineering drawings based on a multimodal fusion model according to claim 1, characterized in that: The entity nodes of the industrial design knowledge graph mentioned in step S3 include standard clause entities, drawing error type entities, standard rectification scheme entities, and component category entities; the relationships between entities include constraint relationships, applicability relationships, substitution relationships, and rectification relationships; the rule engine supports front-end visual configuration, and configuration changes take effect through hot updates of extensible markup language files.

5. The intelligent review method for engineering drawings based on a multimodal fusion model according to claim 1, characterized in that: The gradient boosting decision tree algorithm mentioned in step S4 is XGBoost, LightGBM, or CatBoost; the error trend prediction model takes the feature vector of the current drawing as input and outputs the predicted probability of each error type.

6. The intelligent review method for engineering drawings based on a multimodal fusion model according to claim 1, characterized in that: The interception of drawing saving operation in step S5 is achieved by listening to the FileSave event of the CAD software, and the unique identification position of the erroneous element is marked with a highlight color in the native CAD interface; the preset period is monthly or quarterly.

7. The intelligent review method for engineering drawings based on a multimodal fusion model according to claim 1, characterized in that: The drawing review results output by the method support the automatic generation of standardized review reports. The reports include screenshots of drawing errors, non-compliant standard clauses, and rectification plans, and can be directly archived to the PLM drawing management system. The multimodal fusion verification model output data supports seamless interaction and integration with CAD, CAPP, CAM, and MES business systems.

8. An intelligent review system for engineering drawings based on a multimodal fusion model, characterized in that, include: The multimodal information acquisition module is used to acquire multimodal information from drawings based on the open API of CAD software and complete structured parsing. The multimodal fusion verification module is used to build and run a multimodal fusion verification model based on convolutional neural networks and Transformers to identify cross-modal association errors. The configurable rule engine module is used to complete drawing compliance judgment, error classification and correction suggestion output based on extensible markup language files and industrial design knowledge graph; The error trend prediction module is used to train a machine learning model based on historical map review data and output early warning information. The CAD save mandatory verification module is used to monitor CAD software save events and perform audit interception. The knowledge graph accumulation and iteration module is used to store audit data, error cases, and rectification plans, and supports periodic retraining of the model; the system executes the intelligent audit method according to any one of claims 1 to 7 when it runs.

9. The intelligent auditing system according to claim 8, characterized in that: The system has a built-in lightweight AI inference engine that enables local offline multimodal inference calculations of drawings through model quantization, knowledge distillation, or network pruning techniques, without the need for cloud computing power. The system is natively and deeply integrated with CAD software and is compatible with various mainstream CAD software versions.

10. The intelligent auditing system according to claim 8, characterized in that: The system adopts a five-layer architecture, from top to bottom: presentation layer, business logic layer, data access layer, multimodal AI intelligence layer, and knowledge layer. The presentation layer is a native integrated interface of CAD software, providing one-click approval, visual rule configuration, and approval report export functions. The business logic layer is responsible for approval process scheduling, rule execution, mandatory verification triggering, and account permission management. The data access layer connects to a persistent MySQL database and a high-speed Redis cache. The multimodal AI intelligence layer carries out primitive visual extraction, text semantic parsing, cross-modal feature fusion, and error risk prediction and reasoning. The knowledge layer stores industrial design knowledge graphs, drafting standard libraries, and historical drawing review experience cases.

11. A computer-readable storage medium having a computer-executable program stored thereon, characterized in that: When the program is run by the processor, it executes all the steps of the intelligent review method for engineering drawings based on a multimodal fusion model as described in any one of claims 1 to 7.

Images