Image recognition-based drawing intelligent review and maintenance auxiliary decision method and system

Abstract

This invention relates to an intelligent drawing review and maintenance support decision-making method and system based on image recognition, specifically as follows: Drawing input and preprocessing: Receiving multi-format drawings and performing adaptive optimization to provide high-quality data for subsequent recognition; Multimodal element extraction: Extracting graphic components, text information, and attribute parameters from the drawings to establish a multimodal dataset; Multi-dimensional review and defect identification: Reviewing drawings from three dimensions: compliance, logical consistency, and maintenance relevance, locating and classifying defects; Defect-maintenance correlation analysis: Establishing a mapping relationship between design defects and maintenance plans, quantifying the impact of defects; Maintenance plan optimization generation: Selecting the optimal maintenance plan through a multi-objective optimization algorithm; Result output and model iteration: Outputting a standardized review report and maintenance plan, and optimizing the model based on feedback; This invention has the advantages of a multimodal fusion architecture, a closed-loop process, adaptive and self-learning capabilities, and the ability to achieve quantitative decision support.

Description

Technical Field

[0001] This invention belongs to the field of engineering digitization and intelligent inspection technology, specifically relating to a method and system for intelligent review of drawings and maintenance auxiliary decision-making based on image recognition. Background Technology

[0002] With the advancement of smart construction, engineering drawings, as the core basis for design, construction, and maintenance, directly determine the safety and economy of a project. Currently, the industry has seen the emergence of AI-based construction drawing evaluation software and automated review systems. These systems rely on OCR recognition, CAD parsing engines, and rule engines to achieve standardized drawing checks (such as dimensioning and compliance verification). These products have been implemented in scenarios such as initial review within design institutes and government approval assistance, initially replacing some manual review work and improving efficiency by 60%-80% compared to traditional manual methods. However, although existing intelligent review products possess basic automation capabilities, significant shortcomings remain in functional completeness and structural design, as detailed below:

[0003] I. The disconnect between image recognition and semantic understanding leads to poor adaptability to complex scenarios: Existing systems mostly adopt a simple architecture of "OCR text extraction + rule matching", which can only recognize independent text and graphic elements and cannot establish a deep association between graphics and semantics. For example, for the tolerance mark "Ø25H7" in mechanical drawings, only the character itself can be recognized, but the corresponding hole structure and datum surface information cannot be associated. This results in a failure rate of more than 30% in the review of irregular structures and cross-layer related components. The core reason is the lack of a multimodal fusion model and the failure to build a semantic mapping relationship between graphics, text and specifications, which cannot simulate the "reading-understanding-reasoning" logic of human engineers.

[0004] Second, the review dimensions are too narrow and do not cover the analysis of the relationship between maintenance and repair: The existing products only focus on compliance checks in the design stage (such as size conflicts and matching of specification clauses), without considering the relationship between drawing defects and subsequent maintenance. For example, if the load-bearing wall thickness in the building drawings is insufficient, the existing system will only prompt "violation of GB50011 specification", but it cannot analyze the impact of the defect on subsequent structural reinforcement and pipeline modification. As a result, maintenance decisions still need to rely on human experience, forming a data gap between "review and maintenance". Structurally, it lacks a maintenance knowledge graph and risk transmission model, and only achieves "error identification" without achieving a closed loop of "decision assistance" function.

[0005] Third, the rule base is fixed and lacks adaptability: The existing system's review rules are mostly stored in "hard-coded" form, relying on manual conversion of industry standards into conditional expressions. This makes it unable to adapt to the drawing habits of different design institutes (such as differences in line type annotations and personalized drawing frame formats) and dynamically updated standard specifications. For example, when local standards are revised, technical personnel need to rewrite the rule code, with an update cycle of 1-2 months, resulting in a lag in the system's response to the new regulations. At the same time, the recognition accuracy of blurry drawings (such as noise in scanned documents and low-resolution images) drops by more than 50% due to the lack of adaptive preprocessing modules and transfer learning capabilities.

[0006] IV. Weak Decision Support and Lack of Quantitative Maintenance Plans: The existing system only outputs a "problem list + specification clauses," without providing actionable maintenance suggestions. For example, after detecting a collision between a pipe and a beam / column, it only indicates "collision conflict exists," but does not calculate the optimal avoidance path, material replacement costs, or the impact on the construction period. This forces engineers to spend a significant amount of time developing maintenance plans. The core deficiency is the lack of a multi-objective optimization algorithm for cost, risk, and schedule, leaving the decision-making process at a "qualitative prompt" level rather than "quantitative guidance." Therefore, it is essential to provide a multi-modal fusion architecture, a closed-loop process, adaptive and self-learning capabilities, and a drawing-based intelligent review and maintenance auxiliary decision-making method and system that enables quantitative decision support. Summary of the Invention

[0007] (a) Technical issues

[0008] In view of the above-mentioned existing technology, this application mainly addresses the following technical problems:

[0009] 1. How to solve the problem of the separation between image recognition and semantic understanding in existing systems, realize multimodal correlation reasoning of graphics, text, and specifications, and improve the review accuracy of complex drawings and blurry images;

[0010] 2. How to break through the limitations of existing products that "only review, do not make decisions", establish a correlation model between design defects and maintenance solutions, and realize full-process assistance from defect identification to maintenance decision-making;

[0011] 3. How to solve the problem of rigid rule base, build an adaptive rule engine, and support dynamic updates of specifications and standards as well as personalized adaptation to drawing habits;

[0012] 4. How to compensate for the lack of quantification in maintenance plans, and improve the scientific nature of decision-making by using multi-objective optimization algorithms to output maintenance suggestions with the best cost and lowest risk.

[0013] (II) Technical Solution

[0014] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for intelligent review and maintenance auxiliary decision-making based on image recognition, which features a multimodal fusion architecture, a closed-loop process, adaptive and self-learning capabilities, and quantitative decision support.

[0015] The objective of this invention is achieved as follows: Firstly, a method for intelligent review and maintenance assistance decision-making based on image recognition of drawings, the method being as follows:

[0016] S1, Drawing Input and Preprocessing: Receives multi-format drawings and performs adaptive optimization to provide high-quality data for subsequent recognition;

[0017] S2, Multimodal Element Extraction: Extract graphic components, text information and attribute parameters from drawings to establish a multimodal dataset;

[0018] S3, Multi-dimensional review and defect identification: Review drawings from three dimensions: compliance, logical consistency, and maintenance relevance, locate defects and classify them;

[0019] S4, Defect-Maintenance Correlation Analysis: Establish the mapping relationship between design defects and maintenance plans, and quantify the impact of defects;

[0020] S5, Maintenance plan optimization generation: The optimal maintenance plan is selected from the candidate plans through a multi-objective optimization algorithm;

[0021] S6, Results Output and Model Iteration: Output standardized review report and maintenance plan, and optimize the model based on feedback.

[0022] Furthermore, the drawing preprocessing steps in S1 are as follows:

[0023] Format parsing: Through the CAD parsing engine and PDF vector extraction module, drawings of different formats are uniformly converted into standardized image tensors and vector data;

[0024] Preprocessing:

[0025] 1) Blurred image restoration: A super-resolution reconstruction algorithm based on generative adversarial networks is used to reduce noise and improve resolution of scanned documents and low-resolution drawings;

[0026] 2) Layer separation and standardization: Automatically extract the frame, graphics, annotation and text layers, unify the line type, color and font format, and adapt to the subsequent recognition model.

[0027] In this invention, the multi-format drawings received in S1 include: vector graphics or scanned drawings in PDF, DWG, and DXF formats, project types, and industry standard versions.

[0028] Furthermore, the adaptive optimization in S1.2 includes grayscale conversion, binarization, denoising, and enhancement operations; wherein the binarization adopts an adaptive thresholding method; the denoising adopts a median filtering method to remove noise in the image while preserving the edge information of the image; the enhancement operation adopts a histogram equalization method to enhance the contrast of the image, making the target object in the image clearer.

[0029] Furthermore, the multimodal element extraction in S2 is as follows:

[0030] Graphic component recognition: An improved YOLOv8 network is used to detect and classify components, and output the component type, coordinate position, and geometric parameters; the network introduces an attention mechanism to enhance the recognition ability of small components;

[0031] Text information extraction: The OCR engine is used to extract the labeled text and annotations. The named entity recognition technology of NLP is used to classify the text into four types of structured data: size parameters, material properties, technical requirements, and specification references.

[0032] Multimodal association: Construct a preliminary semantic graph, associate graphical components with corresponding text annotations and specification clauses to form triplet data of "component-attribute-specification".

[0033] Furthermore, the compliance review in S3 is as follows:

[0034] Automatic specification parsing: Industry standards are transformed into machine-executable logical rules using NLP technology. The rule base supports version management and custom import.

[0035] Comparison and verification: The component parameters extracted in step 2 are matched with the rule base. Fuzzy matching algorithm is used to process parameter errors and output compliance results.

[0036] Furthermore, the logical consistency check in S3 specifically involves: analyzing the spatial relationships and logical conflicts between components based on the GNN model, and calculating the conflict probability through the adjacency matrix of the graph nodes, as shown in the following formula: ,in, For component nodes; Spatial distance; The conflict threshold; The weighting parameter is used for the maintenance correlation review, which specifically involves: analyzing the impact of identified defects on subsequent maintenance through a maintenance knowledge graph, and calculating the maintenance priority of defects using a risk transmission algorithm.

[0037] Furthermore, the goal of the risk transmission algorithm is to quantify the cascading impact of design defects on maintenance, and the calculation formula is as follows: ,in: Total risk value; For the first Risk level of each influencing factor; For the first The probability of occurrence of each factor; For the first The weights of each factor.

[0038] Furthermore, the defect-maintenance correlation analysis in S4 includes the following steps:

[0039] Knowledge graph construction: Integrate industry maintenance cases, component attributes, material properties, and construction process data to construct a four-layer knowledge graph containing "defect-impact-measures-cost";

[0040] Impact Quantitative Analysis: The chain reaction of defects was calculated using a Bayesian network model;

[0041] Solution matching: Based on similarity retrieval using a knowledge graph, match the historical repair solutions that best fit the current defect and output 3-5 candidate solutions.

[0042] Furthermore, the multi-objective optimization algorithm in S5 employs the NSGA-III multi-objective optimization algorithm to perform multi-objective optimization on candidate solutions, outputting a Pareto optimal solution set, and providing three priority solutions: "optimal cost," "lowest risk," and "shortest construction period," as detailed below:

[0043] S5.1, Initialize the population: Generate One candidate solution for maintenance;

[0044] S5.2, Fast Non-Dominated Sort: Based on the objective function value, candidate solutions are divided into different non-dominated layers;

[0045] S5.3, Crowding Calculation: Quantify the distribution density of solutions in the same layer to avoid solution set concentration;

[0046] S5.4, Reference point association: Select the optimal solution by using reference points to ensure the diversity and convergence of the solution set.

[0047] Secondly, a system for intelligent review and maintenance auxiliary decision-making based on image recognition of drawings includes an input layer, a core processing layer, a knowledge support layer, and an output layer. The system is used to execute the intelligent review and maintenance auxiliary decision-making based on image recognition of drawings as described above.

[0048] The input layer includes a drawing upload module and a preprocessing module; the core processing layer includes an identification and extraction module, a review and reasoning module, a correlation analysis module, and an optimization decision-making module; the knowledge support layer includes a standard rule base and a maintenance knowledge graph; and the output layer includes a visualization module and an interaction module.

[0049] The input layer is used to receive multi-format drawings and perform adaptive optimization to provide high-quality data for subsequent recognition.

[0050] The identification and extraction module includes a multimodal element extraction module, which is used to extract graphic components, text information and attribute parameters from drawings to establish a multimodal dataset;

[0051] The review and reasoning module includes a multi-dimensional defect identification module, which is used to review drawings from three dimensions: compliance, logical consistency, and maintenance relevance, to locate and classify defects;

[0052] The correlation analysis module includes a defect-maintenance mapping module, which is used to establish the mapping relationship between design defects and maintenance plans, and to quantify the impact of defects;

[0053] The optimization decision module includes a multi-objective scheme generation module, which is used to select the optimal maintenance scheme from candidate schemes through a multi-objective optimization algorithm;

[0054] The output layer is used to output standardized review reports and maintenance plans, based on a feedback optimization model.

[0055] (III) Beneficial Effects

[0056] 1. Enhanced Review Accuracy and Efficiency: This invention achieves deep correlation between graphics, text, and specifications through a multimodal fusion model and semantic graph. The false negative rate in complex drawing reviews is reduced to below 5%, and the accuracy of fuzzy image recognition is improved to over 92% (40% higher than existing technologies). The review cycle is shortened from 2-3 days manually to within 10 minutes, improving efficiency by over 95%. The core reason is that the multimodal model simulates the human "image recognition-reasoning" logic, overcoming the recognition limitations of existing technologies.

[0057] 2. Achieve a closed-loop "review-maintenance" process: For the first time, maintenance knowledge is integrated into the review process. Through a defect risk transmission model, not only are design errors identified, but the impact of defects on maintenance (such as the percentage increase in cost and the number of days of delay) can also be quantitatively analyzed, and the optimal maintenance plan can be output. For example, when insufficient floor slab thickness is detected, the system automatically generates a specific plan of "carbon fiber reinforcement + load redistribution", including material usage, construction procedures and cost budget, which shortens the maintenance decision time by more than 70%.

[0058] 3. Adaptive rule base updates significantly enhance adaptability: Based on NLP and transfer learning technologies, automatic parsing and rule generation of standard texts are achieved, shortening the new rule update cycle from 1-2 months to within 24 hours; through an adaptive preprocessing module, noise reduction and super-resolution reconstruction are performed on fuzzy drawings, maintaining a recognition accuracy of over 85%, adapting to the drawing habits of different design institutes, and solving the pain points of existing systems such as "fixed rules and poor adaptability";

[0059] 4. Quantitative maintenance decision-making to reduce implementation costs: The multi-objective optimization algorithm can simultaneously balance maintenance costs, safety risks, and schedule requirements, and output quantitative solutions (such as optimal modification paths, material replacement lists, and cost-saving ratios). Tests have shown that after adopting this invention, the cost of developing maintenance plans is reduced by 30%-40%, and the rework rate caused by unreasonable plans is reduced by 60%, significantly improving the economic efficiency of the project. Attached Figure Description

[0060] Figure 1 This is a flowchart of the method of the present invention.

[0061] Figure 2 This is a system block diagram of the present invention. Detailed Implementation

[0062] This invention specifically involves the cross-application of image recognition, deep learning, natural language processing, and industrial / construction engineering review. It is applicable to the automated review, defect location, and intelligent decision-making of engineering drawings (such as construction drawings and manufacturing drawings) in industries such as construction, machinery, and energy. It can be directly applied to the drawing quality control and maintenance process optimization scenarios of design institutes, construction units, regulatory departments, and operation and maintenance companies.

[0063] The present invention will be further described below with reference to the embodiments and / or accompanying drawings.

[0064] Example 1

[0065] like Figure 1 As shown, the intelligent drawing review and maintenance auxiliary decision-making method based on image recognition is as follows:

[0066] Step 1: Drawing Input and Preprocessing: Receive drawings in multiple formats and perform adaptive optimization to provide high-quality data for subsequent recognition;

[0067] In this embodiment, the drawing formats received in this step include: PDF, DWG, and DXF format drawings (including vector graphics and scanned copies), project type (architecture / mechanical / electrical), and industry standard version.

[0068] As a specific implementation method, the input drawing image information can also come from digital images of the drawings obtained by scanning or photographing paper drawings using devices such as scanners or high-definition cameras; and it is necessary to ensure that the images are clear, complete, and free from obvious blurring, distortion or obstruction.

[0069] Specifically, the drawing input and preprocessing steps are as follows:

[0070] Step 1.1, Format Parsing: Using the CAD parsing engine (supporting DWG kernel) and PDF vector extraction module, drawings of different formats are uniformly converted into standardized image tensors (resolution 2048×2048) and vector data;

[0071] Step 1.2, Adaptive preprocessing:

[0072] 1) Blurred Image Restoration: A super-resolution reconstruction algorithm based on Generative Adversarial Networks (GANs) is used to reduce noise and improve resolution in scanned documents and low-resolution drawings. The formula is as follows: ,in, For generator networks; For network parameters; The weight for perceived loss is set to 0.8. This is the perceptual loss function used to ensure the visual consistency of the restored image;

[0073] 2) Layer separation and standardization: Automatically extract layers such as frame, graphics, annotation, and text, and unify line type, color, and font format to adapt to subsequent recognition models.

[0074] Specifically, the multimodal fusion recognition algorithm achieves semantic association between graphics, text, and specifications, improving the recognition accuracy of complex drawings. ,in: This is a multimodal fusion feature; Graphical features extracted by CNN (such as component shape and position); Text features extracted for NLP (such as dimension annotations and technical requirements); These are the normative features of a knowledge graph (such as compliance clauses and related components). These are the weighting coefficients (determined through cross-validation, with default values ​​of 0.4, 0.3, and 0.3 respectively).

[0075] As another possible implementation, preprocessing also includes operations such as grayscale conversion, binarization, denoising, and enhancement to improve image quality and facilitate subsequent image recognition and analysis.

[0076] ① Grayscale conversion: Converting a color image to a grayscale image reduces the amount of image data and improves processing speed; the formula is: ,in, These are the red, green, and blue components of a color image, respectively. This is the grayscale value.

[0077] ② Binarization: Converting a grayscale image into a black-and-white binary image to highlight the target object in the image; an adaptive thresholding method is used for binarization, the formula is: ,in, For point The threshold at; For point The average gray value of the surrounding neighborhood; For point The standard deviation of gray levels in the surrounding neighborhood; This is the adjustment coefficient.

[0078] ③ Noise Removal: Median filtering is used to remove noise from the image while preserving its edge information; the formula for median filtering is: ,in, The filtered image; Original image; This is the filtering window.

[0079] ④ Enhancement: Histogram equalization is used to enhance image contrast, making target objects in the image clearer; the formula for histogram equalization is: ,in, For the first part of the original image grayscale levels; For the equalized first grayscale levels; The original image has gray levels of 1. The number of pixels; This represents the total number of pixels in the original image.

[0080] S2, Multimodal Element Extraction: Extract graphic components, text information and attribute parameters from drawings to establish a multimodal dataset;

[0081] In this embodiment, the core of this step is CNN+OCR+knowledge graph initialization.

[0082] Specifically, the steps for extracting multimodal elements are as follows:

[0083] Step 2.1, Graphic Component Recognition: An improved YOLOv8 network is used to detect and classify building / mechanical components (walls, beams, columns, pipes, gears, etc.), outputting component type, coordinate position, and geometric parameters (such as length, thickness, and hole diameter). The network incorporates a focus-attention mechanism (CBAM) to enhance the recognition capability of small components (such as bolts and pipes), as shown in the following formula: ,in, The feature map is after attention weighting; These are the convolution weights; Use the Sigmoid activation function; This is the original feature map; This is for element-wise multiplication.

[0084] Step 2.2, Text Information Extraction: A high-precision OCR engine is used to extract the labeled text (such as dimensions, tolerances, and material specifications) and annotations. Through NLP's Named Entity Recognition (NER) technology, the text is classified into four types of structured data: "Dimensional Parameters", "Material Properties", "Technical Requirements", and "Specification References".

[0085] Step 2.3, Multimodal Association: Construct a preliminary semantic graph, associate graphic components (such as “load-bearing wall”) with corresponding text labels (such as “thickness 200mm”) and standard clauses (such as “GB50011”) to form triple data of “component-attribute-standard” (such as thickness, 200mm, standard, GB50011).

[0086] As a specific and feasible implementation method, drawing element recognition can also adopt the following scheme:

[0087] Image recognition algorithms are used to identify various elements in the preprocessed drawing images, including electrical components, circuits, building structures, and mechanical parts; deep learning-based object detection algorithms, such as Faster R-CNN, are used to locate and classify the elements in the drawings.

[0088] Furthermore, the Faster R-CNN algorithm consists of three main parts: a convolutional neural network (CNN), a region proposal network (RPN), and a classification network.

[0089] Specifically, first, a CNN is used to extract features from the input image to obtain a feature map; then, the RPN generates a series of region proposals that may contain the target on the feature map; finally, a classification network classifies and regresses these region proposals to determine the category and location of the target.

[0090] S3, Multi-dimensional review and defect identification: Review drawings from three dimensions: compliance, logical consistency, and maintenance relevance, locate defects and classify them;

[0091] In this embodiment, the core of this step is adaptive rule engine + graph neural network (GNN) inference;

[0092] Specifically, the multi-dimensional review and defect identification steps are as follows:

[0093] Step 3.1, Compliance Review:

[0094] 1) Automatic specification parsing: Industry specifications (such as GB50016 and ISO standards) are converted into machine-executable logical rules (such as "load-bearing wall thickness ≥ 240mm") through NLP technology. The rule base supports version management and custom import.

[0095] 2) Comparison and verification: Match the component parameters extracted in step 2 with the rule base, use fuzzy matching algorithm to process parameter errors (e.g., "235mm" is judged as "close to violation"), and output the compliance result.

[0096] Step 3.2, Logical Consistency Review: Based on the GNN model, analyze the spatial relationships and logical conflicts between components (such as collisions between pipes and beams / columns, and non-closed dimensional chains). Calculate the conflict probability using the adjacency matrix of the graph nodes, as shown in the following formula: ,in, For component nodes; Spatial distance; The conflict threshold; These are the weight parameters.

[0097] Step 3.3, Inspection Relevance Review: Through the inspection knowledge graph, analyze the impact of the identified defects on subsequent inspections (e.g., "pipeline collision" → "affects pipeline modification" → "requires demolition and reconstruction"). Use the risk transmission algorithm to calculate the inspection priority of the defects (Category A: emergency repair; Category B: rectification within a time limit; Category C: optimization suggestion).

[0098] Specifically, the goal of the risk transmission algorithm is to quantify the cascading impact of design defects on maintenance. The calculation formula is as follows: ,in: The total risk value is 0-10, with higher values ​​indicating higher risk. For the first Risk levels of each influencing factor (1-5); For the first The probability of occurrence of each factor (0-1); For the first The weights of each factor (e.g., safety risk weight 0.5, cost risk weight 0.3, schedule risk weight 0.2).

[0099] In summary, this step primarily aims to achieve drawing review rule matching: A corresponding review rule library is established based on the drawing review specifications and standards of different industries; the identified drawing elements are matched against the rules in the review rule library to determine if the drawings contain design errors or non-compliance with specifications; for example, for electrical drawings, review rules may include whether the parameters of electrical components meet requirements, whether wiring connections are correct, etc.; then, based on the rule matching results, a detailed drawing review report is generated, including error type, location, and suggested modification schemes; the review report is presented to the user in an intuitive way, such as by marking error locations through a graphical interface and providing detailed textual explanations.

[0100] S4, Defect-Maintenance Correlation Analysis: Establish the mapping relationship between design defects and maintenance plans, and quantify the impact of defects;

[0101] In this embodiment, the core of this step lies in the maintenance of the knowledge graph and the risk transmission model.

[0102] Specifically, the steps for defect-maintenance correlation analysis are as follows:

[0103] Step 4.1 Knowledge Graph Construction: Integrate data such as industry maintenance cases, component attributes, material characteristics, and construction techniques to construct a four-layer knowledge graph containing "defect-impact-measures-cost".

[0104] Step 4.2, Impact Quantification Analysis: The cascading effects of the defect are calculated using a Bayesian network model, as shown in the following formula: ,in, For defect events (such as "insufficient floor slab thickness"); To influence the outcome (e.g., "increased reinforcement costs" or "construction delays"); Parent nodes (such as material prices, construction difficulty).

[0105] Step 4.3, Solution Matching: Based on similarity retrieval using a knowledge graph, match the historical repair solutions that best fit the current defect and output 3-5 candidate solutions.

[0106] S5, Maintenance plan optimization generation: The optimal maintenance plan is selected from the candidate plans through a multi-objective optimization algorithm;

[0107] In this embodiment, the core of this step is the NSGA-Ⅲ multi-objective optimization algorithm;

[0108] Specifically, the steps for generating the maintenance plan optimization are as follows:

[0109] Step 5.1, Define the objective function:

[0110] Cost target: ;

[0111] Risk objectives: ,in, Risk level; The probability of occurrence;

[0112] Project timeline target: .

[0113] Step 5.2, Setting Constraints: Standard requirements (such as meeting the bearing capacity standard after reinforcement), resource constraints (such as material supply cycle), and construction feasibility (such as site space constraints).

[0114] Step 5.3, Algorithm Solution: The NSGA-III algorithm is used to perform multi-objective optimization on the candidate solutions, output the Pareto optimal solution set, and provide three priority solutions: "cost-optimal", "risk-lowest" and "shortest construction period", and label the quantitative indicators of each solution (such as "cost-optimal solution: saves 15% of cost and extends the construction period by 3 days").

[0115] Specifically, the goal of the NSGA-Ⅲ multi-objective optimization algorithm is to select the optimal maintenance plan under the constraints of cost, risk, and schedule. Its key steps are as follows:

[0116] 1. Initialize the population: Generate One candidate solution for maintenance (based on knowledge graph matching results);

[0117] 2. Fast Non-Dominated Sort: Based on the objective function value, candidate solutions are divided into different non-dominated layers;

[0118] 3. Crowding Calculation: Quantify the distribution density of solutions at the same level to avoid solution set concentration;

[0119] 4. Reference point association: Select the optimal solution by using reference points to ensure the diversity and convergence of the solution set;

[0120] Output: Pareto optimal solution set, containing 3-5 differentiated priority solutions (optimal cost, lowest risk, shortest time).

[0121] As a feasible implementation method, the maintenance plan generation also requires the construction of maintenance auxiliary decision-making: combining the results of drawing review and historical equipment data, data analysis algorithms are used to provide auxiliary decisions for equipment maintenance; for example, based on the equipment failure type and drawing design information, the causes of failure are analyzed, and the optimal maintenance method and replacement parts are recommended; a decision tree algorithm is used for maintenance decision-making, and the construction process of the decision tree is as follows:

[0122] ① Data preparation: Collect historical equipment data and drawing review results, including information such as equipment failure type, failure time, failure cause, repair method, and replaced parts.

[0123] ② Feature selection: Select features relevant to maintenance decisions from the collected data, such as equipment model, years of operation, and failure frequency.

[0124] ③ Decision Tree Construction: Construct the decision tree using the ID3 or C4.5 algorithm; the ID3 algorithm selects features based on information gain, and the formula for calculating information gain is: ,in, For dataset; As a characteristic; Features The number of possible values; Dataset Chinese characteristics Values A subset of; For dataset The information entropy is calculated using the following formula: ,in, For dataset The number of categories; For dataset The Middle The proportion of samples of each class.

[0125] ④ Decision tree pruning: To avoid overfitting of the decision tree, the constructed decision tree is pruned; pre-pruning or post-pruning methods are used to prune the decision tree according to certain pruning criteria (such as the minimum description length criterion).

[0126] S6, Results Output and Model Iteration: Output standardized review report and maintenance plan, and optimize the model based on feedback.

[0127] In this embodiment, the result output and model iteration steps are as follows:

[0128] Step 6.1, Result Visualization:

[0129] 1) Drawing annotation: Highlight the defect location on the original drawing and indicate the defect level (red / A, yellow / B, blue / C).

[0130] 2) Report generation: Automatically generates standardized reports containing "defect list, compliance analysis, maintenance plan, cost budget, and schedule impact", and supports export to Word / PDF / Excel.

[0131] Step 6.2, Model Iteration: Users mark misjudged / missed cases, and the system updates the weights of the multimodal recognition model and rule base through transfer learning. For every 100 feedback cases accumulated, the model accuracy improves by 3%-5%, forming a closed loop of "use-feedback-optimization".

[0132] Among them, the decision output and execution: the maintenance auxiliary decision results are output to the user, and the user carries out equipment maintenance work according to the decision results; at the same time, the system records the maintenance process and results, providing data support for subsequent decision-making.

[0133] This invention relates to an intelligent drawing review and maintenance support decision-making method and system based on image recognition. In use, this invention constructs a multimodal fusion model integrating image recognition, CNN, text understanding, NLP, canonical reasoning, and knowledge graph, achieving a leap from element recognition to semantic understanding and adapting to complex scenarios. This invention integrates review and maintenance decision-making, establishing a closed-loop process of defect identification, risk assessment, solution generation, and effect verification, solving the data gap problem in existing technologies. Through transfer learning and a dynamic rule engine, this invention achieves automatic standard parsing, adaptive drawing habits, and self-learning capabilities, reducing manual maintenance costs. This invention introduces a multi-objective optimization algorithm to transform qualitative problems into quantitative solutions, achieving quantitative decision support and improving the scientific rigor and operability of decisions. This invention possesses the advantages of a multimodal fusion architecture, a closed-loop process, adaptive and self-learning capabilities, and quantitative decision support.

[0134] Example 2

[0135] like Figure 2 As shown, the system of intelligent drawing review and maintenance auxiliary decision-making method based on image recognition includes an input layer, a core processing layer, a knowledge support layer and an output layer. The system is used to execute the intelligent drawing review and maintenance auxiliary decision-making method based on image recognition as described above.

[0136] The input layer includes a drawing upload module and a preprocessing module; the core processing layer includes an identification and extraction module, a review and reasoning module, a correlation analysis module, and an optimization decision-making module; the knowledge support layer includes a standard rule base and a maintenance knowledge graph; and the output layer includes a visualization module and an interaction module.

[0137] The input layer is used to receive multi-format drawings and perform adaptive optimization to provide high-quality data for subsequent recognition.

[0138] As a specific and feasible implementation method, ① the drawing upload module supports uploading multiple drawing formats such as PDF, DWG, DXF, and scanned documents, provides batch upload and project classification management functions, and connects to both cloud storage and local deployment modes to ensure data security.

[0139] ② The preprocessing module includes an adaptive image optimization module, which performs noise reduction, super-resolution reconstruction, layer separation, and format standardization operations. It repairs blurred images using a GAN model, adapts to drawings of different qualities and formats, and provides high-quality data for subsequent recognition.

[0140] Furthermore, the preprocessing module is also used to perform preprocessing operations such as grayscale conversion, binarization, noise reduction, and enhancement on the acquired drawing images to improve image quality, and then transmit the processed image to the recognition and extraction module.

[0141] The collected drawing images can come from the drawing data information of the image acquisition module. The image acquisition module is responsible for scanning or photographing the paper drawings using devices such as scanners or high-definition cameras to obtain digital images of the drawings and then transmitting the images to the preprocessing module.

[0142] The identification and extraction module includes a multimodal element extraction module, which is used to extract graphic components, text information and attribute parameters from drawings to establish a multimodal dataset;

[0143] Specifically, the multimodal element extraction module integrates an improved YOLOv8 network and a high-precision OCR engine to achieve component recognition, text extraction, and semantic association, and outputs "component-attribute-specification" triple data to support multi-dimensional review.

[0144] Furthermore, the multimodal element extraction module can be replaced by the drawing element recognition module; the drawing element recognition module uses a deep learning-based target detection algorithm (such as the Faster R-CNN algorithm) to identify and locate various elements in the preprocessed drawing image, and transmits the recognition results to the review and reasoning module.

[0145] The review and reasoning module includes a multi-dimensional defect identification module, which is used to review drawings from three dimensions: compliance, logical consistency, and maintenance relevance, to locate and classify defects;

[0146] Specifically, the multi-dimensional defect identification module is based on an adaptive rule engine (automatic parsing of specifications) and a GNN inference model to complete compliance, logical consistency, and maintenance relevance reviews, and output a defect list and risk level.

[0147] Furthermore, the multi-dimensional defect identification module can be replaced by the review rule matching module; the review rule matching module establishes a review rule library based on the drawing review specifications and standards of different industries, matches the identified drawing elements with the rules in the review rule library, determines whether there are design errors or non-compliance with specifications in the drawings, and transmits the review results to the review result display module and the maintenance decision module.

[0148] The correlation analysis module includes a defect-maintenance mapping module, which is used to establish the mapping relationship between design defects and maintenance plans, and to quantify the impact of defects;

[0149] Specifically, the defect-maintenance mapping / association module calls upon the maintenance knowledge graph and Bayesian risk model to analyze the chain reaction of defects on maintenance, quantify costs, risks, and schedule losses, and match candidate maintenance solutions.

[0150] The optimization decision module includes a multi-objective scheme generation module / multi-objective optimization decision module, which is used to select the optimal maintenance scheme from candidate schemes through a multi-objective optimization algorithm;

[0151] Specifically, the multi-objective solution generation module / multi-objective optimization decision module runs the NSGA-III algorithm, optimizes candidate solutions in combination with constraints (standards, resources, feasibility), and outputs the Pareto optimal solution set and quantitative indicators.

[0152] Furthermore, the optimization decision module also includes a maintenance decision module; the maintenance decision module combines the review results of the review rule matching module and the equipment's historical data, uses data analysis algorithms (such as decision tree algorithms) to provide auxiliary decisions for equipment maintenance, generates maintenance plans, and transmits the decision results to the user.

[0153] The output layer is used to output standardized review reports and maintenance plans, based on a feedback optimization model.

[0154] As a specific and feasible implementation method, ① the normative rule base in the knowledge support layer has a built-in dynamic update engine to store national / local / industry norms, supports automatic NLP parsing and dynamic updates, provides rule version management and custom import functions, and adapts to new rules and personalized needs.

[0155] ② The maintenance knowledge graph in the knowledge support layer includes case studies and process data, which are used to integrate maintenance cases, process parameters, material costs and other data to build a four-layer network of "defect-impact-measures-cost" to provide knowledge support for solution matching.

[0156] As another feasible implementation method, ① the visualization output module (drawing annotation + report generation) in the output layer is used to highlight the location and level of defects on the original drawings and automatically generate standardized reports (including defect analysis, repair plan and cost budget), and supports export in multiple formats.

[0157] ② The interaction and iteration module (user feedback + model iteration) in the output layer is used to provide a user feedback entry point (marking misjudged / missed cases), and to update the recognition model and rule base weights through transfer learning to achieve system self-optimization.

[0158] The image recognition-based intelligent drawing review and maintenance auxiliary decision-making system also includes a data storage module. The data storage module is responsible for storing information such as drawing images, review results, equipment historical data, and maintenance decision results, providing data support for the system's operation and decision-making. At the same time, it provides data query and management functions to facilitate users' access to and operation of the data.

[0159] In summary, this invention automatically reviews drawings using image recognition technology, significantly shortening review time and improving work efficiency. Utilizing advanced image recognition algorithms and pre-set review rules, it accurately identifies design errors and non-compliance with specifications in drawings, reducing misjudgments and omissions. By combining drawing review results with historical equipment data, this invention employs data analysis algorithms to provide scientifically sound maintenance solutions, reducing maintenance costs and time. Furthermore, by establishing a unified data platform, this invention enables real-time data sharing and collaborative work during drawing review and maintenance decision-making processes, improving work efficiency and decision-making quality.

[0160] This invention relates to an intelligent drawing review and maintenance support decision-making method and system based on image recognition. In use, this invention constructs a multimodal fusion model integrating image recognition, CNN, text understanding, NLP, canonical reasoning, and knowledge graph, achieving a leap from element recognition to semantic understanding and adapting to complex scenarios. This invention integrates review and maintenance decision-making, establishing a closed-loop process of defect identification, risk assessment, solution generation, and effect verification, solving the data gap problem in existing technologies. Through transfer learning and a dynamic rule engine, this invention achieves automatic standard parsing, adaptive drawing habits, and self-learning capabilities, reducing manual maintenance costs. This invention introduces a multi-objective optimization algorithm to transform qualitative problems into quantitative solutions, achieving quantitative decision support and improving the scientific rigor and operability of decisions. This invention possesses the advantages of a multimodal fusion architecture, a closed-loop process, adaptive and self-learning capabilities, and quantitative decision support.

Claims

1. A method for intelligent review of drawings and auxiliary decision-making for maintenance based on image recognition, characterized in that: The method is as follows: S1, Drawing Input and Preprocessing: Receives multi-format drawings and performs adaptive optimization to provide high-quality data for subsequent recognition; S2, Multimodal Element Extraction: Extract graphic components, text information and attribute parameters from drawings to establish a multimodal dataset; S3, Multi-dimensional review and defect identification: Review drawings from three dimensions: compliance, logical consistency, and maintenance relevance, locate defects and classify them; S4, Defect-Maintenance Correlation Analysis: Establish the mapping relationship between design defects and maintenance plans, and quantify the impact of defects; S5, Maintenance plan optimization generation: The optimal maintenance plan is selected from the candidate plans through a multi-objective optimization algorithm; S6, Results Output and Model Iteration: Output standardized review report and maintenance plan, and optimize the model based on feedback iteration.

2. The intelligent drawing review and maintenance auxiliary decision-making method based on image recognition as described in claim 1, characterized in that: The drawing preprocessing steps in S1 are as follows: Format parsing: Through the CAD parsing engine and PDF vector extraction module, drawings of different formats are uniformly converted into standardized image tensors and vector data; Preprocessing: 1) Blurred image restoration: A super-resolution reconstruction algorithm based on generative adversarial networks is used to reduce noise and improve resolution of scanned documents and low-resolution drawings; 2) Layer separation and standardization: Automatically extract the frame, graphics, annotation and text layers, unify the line type, color and font format, and adapt to the subsequent recognition model.

3. The intelligent drawing review and maintenance auxiliary decision-making method based on image recognition as described in claim 2, characterized in that: The adaptive optimization in S1.2 includes grayscale conversion, binarization, denoising, and enhancement operations; wherein the binarization uses an adaptive thresholding method; the denoising uses median filtering to remove noise from the image while preserving the edge information of the image; and the enhancement operation uses histogram equalization to enhance the contrast of the image, making the target objects in the image clearer.

4. The intelligent drawing review and maintenance auxiliary decision-making method based on image recognition as described in claim 1, characterized in that: The multimodal element extraction in S2 is as follows: Graphic component recognition: An improved YOLOv8 network is used to detect and classify components, and output the component type, coordinate position, and geometric parameters; the network introduces an attention mechanism to enhance the recognition ability of small components; Text information extraction: The OCR engine is used to extract the labeled text and annotations. The named entity recognition technology of NLP is used to classify the text into four types of structured data: size parameters, material properties, technical requirements, and specification references. Multimodal association: Construct a preliminary semantic graph, associate graphical components with corresponding text annotations and specification clauses to form triple data of "component-attribute-specification".

5. The intelligent drawing review and maintenance auxiliary decision-making method based on image recognition as described in claim 1, characterized in that: The compliance review in S3 is as follows: Automatic specification parsing: Industry standards are transformed into machine-executable logical rules using NLP technology. The rule base supports version management and custom import. Comparison and verification: The component parameters extracted in step 2 are matched with the rule base. Fuzzy matching algorithm is used to process parameter errors and output compliance results.

6. The intelligent drawing review and maintenance auxiliary decision-making method based on image recognition as described in claim 5, characterized in that: The logical consistency check in S3 specifically involves: analyzing the spatial relationships and logical conflicts between components based on a GNN model, and calculating the conflict probability using the adjacency matrix of graph nodes, as shown in the following formula: ,in, For component nodes; Spatial distance; The conflict threshold; The weighting parameter is used for the maintenance correlation review, which specifically involves: analyzing the impact of identified defects on subsequent maintenance through a maintenance knowledge graph, and calculating the maintenance priority of defects using a risk transmission algorithm.

7. The intelligent drawing review and maintenance auxiliary decision-making method based on image recognition as described in claim 6, characterized in that: The goal of the risk transmission algorithm is to quantify the cascading impact of design defects on maintenance. The calculation formula is as follows: ,in: Total risk value; For the first Risk level of each influencing factor; For the first The probability of occurrence of each factor; For the first The weights of each factor.

8. The intelligent drawing review and maintenance auxiliary decision-making method based on image recognition as described in claim 1, characterized in that: The defect-maintenance correlation analysis in S4 involves the following steps: Knowledge graph construction: Integrate industry maintenance cases, component attributes, material properties, and construction process data to construct a four-layer knowledge graph containing "defect-impact-measures-cost"; Impact Quantitative Analysis: The chain reaction of defects was calculated using a Bayesian network model; Solution matching: Based on similarity retrieval using a knowledge graph, match the historical repair solutions that best fit the current defect and output 3-5 candidate solutions.

9. The intelligent drawing review and maintenance auxiliary decision-making method based on image recognition as described in claim 1, characterized in that: The multi-objective optimization algorithm in S5 uses the NSGA-III multi-objective optimization algorithm to perform multi-objective optimization on candidate solutions, outputting a Pareto optimal solution set, and providing three priority solutions: "optimal cost", "lowest risk", and "shortest construction period", as detailed below: S5.1, Initialize the population: Generate One candidate solution for maintenance; S5.2, Fast Non-Dominated Sort: Based on the objective function value, candidate solutions are divided into different non-dominated layers; S5.3, Crowding Calculation: Quantify the distribution density of solutions in the same layer to avoid solution set concentration; S5.4, Reference point association: Select the optimal solution by using reference points to ensure the diversity and convergence of the solution set.

10. A system for intelligent review and maintenance assistance decision-making of drawings based on image recognition, comprising an input layer, a core processing layer, a knowledge support layer, and an output layer, characterized in that: The system is used to execute the image recognition-based intelligent drawing review and maintenance auxiliary decision-making method as described in any one of claims 1-9; The input layer includes a drawing upload module and a preprocessing module; the core processing layer includes an identification and extraction module, a review and reasoning module, a correlation analysis module, and an optimization decision-making module; the knowledge support layer includes a standard rule base and a maintenance knowledge graph; and the output layer includes a visualization module and an interaction module. The input layer is used to receive multi-format drawings and perform adaptive optimization to provide high-quality data for subsequent recognition. The identification and extraction module includes a multimodal element extraction module, which is used to extract graphic components, text information and attribute parameters from drawings to establish a multimodal dataset; The review and reasoning module includes a multi-dimensional defect identification module, which is used to review drawings from three dimensions: compliance, logical consistency, and maintenance relevance, to locate and classify defects; The correlation analysis module includes a defect-maintenance mapping module, which is used to establish the mapping relationship between design defects and maintenance plans, and to quantify the impact of defects; The optimization decision module includes a multi-objective scheme generation module, which is used to select the optimal maintenance scheme from candidate schemes through a multi-objective optimization algorithm; The output layer is used to output standardized review reports and maintenance plans, based on a feedback iterative optimization model.

Images