Visual large model defect identification method and system based on text prompt

Abstract

The invention provides a visual large model defect identification method and system based on text prompt, and the system comprises a data collection module which collects a high-resolution image of the surface of a building curtain wall through an unmanned plane or other equipment, and carries out the matching of a related text description; the visual feature extraction module is used for extracting high-level semantic features; the text feature modeling module is used for converting the text description into an embedded vector; the multi-modal fusion module is used for combining visual features and text features; the defect classification module is used for inputting the fused features into a classification layer and outputting probability distribution of each potential defect category; the model optimization module is used for improving generalization ability and detection accuracy; and the result output module is used for generating a visual image and a detailed report. According to the invention, the defects of the prior art in the aspects of detection efficiency, application range and recognition precision are overcome, the safety and convenience of the whole detection process are improved through an intelligent means, and powerful technical support is provided for maintenance of modern building curtain walls.

Description

Technical Field

[0001] The present invention relates to the technical field of building curtain wall defect detection, and in particular to a visual large model defect recognition method and system based on text prompts. Background Art

[0002] As a modern architectural design element, building curtain walls are widely used in high-rise and large buildings due to their aesthetics and functionality; however, with the passage of time and the influence of environmental factors, curtain walls may develop various types of surface defects, such as cracks, deformation, breakage, etc. These defects not only affect the appearance of the building, but may also pose a threat to structural safety; therefore, it is crucial to detect and repair surface defects of building curtain walls in a timely and accurate manner.

[0003] The existing building curtain wall surface defect detection methods mainly include the following: Manual inspection: Relies on the inspector’s experience and visual judgment to identify defects.

[0004] Robotic Inspection: Inspection is performed by an automatic or semi-automatic robotic system.

[0005] Drone inspection: Use drones to quickly collect images and assist in inspection work.

[0006] Sensor detection: monitor physical information such as temperature, humidity, and displacement through sensors installed on the curtain wall.

[0007] Deficiencies of existing technology: 1. Manual inspection: Low efficiency: Relying on inspectors to check one by one, it is time-consuming and costly.

[0008] Highly subjective: The results are affected by the inspectors’ experience and vision, which can easily lead to misjudgment.

[0009] Safety hazards: Especially in high-rise buildings, workers are required to work at high altitudes, which poses great risks.

[0010] 2. Robot detection: Limited movement: Complex curtain wall structures may make it difficult for the robot to move around and cover all areas.

[0011] High cost: Equipment procurement and maintenance costs are high, making it difficult to promote and apply on a large scale.

[0012] 3. Drone inspection: Unstable image quality: It is easily affected by external factors such as lighting and weather, resulting in uneven quality of collected images.

[0013] Heavy data processing burden: Relying on traditional visual algorithms for image analysis makes it difficult to efficiently identify complex defects.

[0014] 4. Sensor detection: Complex deployment: The installation and maintenance of sensors are relatively cumbersome, which increases the difficulty of implementation.

[0015] Insufficient accuracy and real-time performance: Data analysis still faces challenges and cannot meet real-time and high-precision requirements.

[0016] Therefore, the prior art has deficiencies and needs further improvement. Summary of the invention

[0017] In view of the problems existing in the prior art, the present invention provides a method and system for visual large model defect recognition based on text prompts.

[0018] To achieve the above object, the specific scheme of the present invention is as follows: The present invention provides a visual large model defect recognition system based on text prompts, the system comprising: A data acquisition module is used to collect high-resolution images of the building curtain wall surface through drones or other equipment, and provide relevant text descriptions; The visual feature extraction module uses a pre-trained deep convolutional neural network (CNN) to perform multi-layer convolution, pooling, and non-linear activation function processing on the input image to extract high-level semantic features; The text feature modeling module converts text descriptions into embedding vectors and uses the self-attention mechanism to extract text features and generate text representations with contextual information; The multimodal fusion module combines visual features and text features in a weighted fusion or concatenation manner, and uses the self-attention mechanism to dynamically adjust the weights between the two to enhance the importance of related features; The defect classification module inputs the fused features into the classification layer, performs defect detection and classification, and outputs the probability distribution of each potential defect category; Model optimization module, which uses self-supervised learning and data enhancement techniques to fine-tune the model, reduce dependence on large-scale labeled data, and improve generalization ability and detection accuracy; The result output module generates visual images and detailed reports, provides real-time feedback and alerts, and assists managers in analysis and decision-making.

[0019] Furthermore, the data acquisition module also performs data enhancement, including image rotation, scaling, and brightness adjustment, to increase the diversity and reliability of the data set.

[0020] Furthermore, the text feature modeling module further includes a pre-trained language model, including BERT or GPT, to improve the ability to understand complex text descriptions.

[0021] Furthermore, the text feature modeling module adopts the self-attention mechanism in the Transformer architecture to improve the efficiency and accuracy of text feature extraction.

[0022] Furthermore, the multimodal fusion module also adopts a multi-head attention mechanism (Multi-HeadAttention) or a graph neural network (GNN) to more effectively process complex multimodal data.

[0023] The present invention also provides a method for visual large model defect recognition based on text prompts. Based on the above system, the method comprises the following steps: S1, data collection: collect high-resolution images of the building curtain wall surface through drones or other equipment, and provide relevant text descriptions for each image; S2, visual feature extraction: Use a pre-trained deep convolutional neural network (CNN) to process the input image and learn the high-level semantic features of the image through multiple layers of convolution, pooling, and non-linear activation functions; S3, text feature extraction: convert the words or phrases in the text description into word embedding vectors to form the text embedding matrix, and use the self-attention mechanism in the Transformer architecture to extract text features; S4, feature fusion: Combine visual features and text features in a weighted fusion or concatenation manner, and use the self-attention mechanism to dynamically adjust the weight between the two to ensure that the model can focus on the visual and text information most relevant to defect detection; S5, defect detection and classification: the fused features are input into the classification layer to perform defect detection and classification tasks, and the probability distribution of each potential defect category is output; S6, model optimization: use self-supervised learning methods to conduct preliminary training on the model, and fine-tune it for specific building curtain wall defect detection tasks through transfer learning, and combine multiple data enhancement techniques to improve the generalization ability of the model; S7, Result Output: Generates visual images and detailed reports containing defect location and classification information, provides real-time feedback and alerts, helps managers promptly identify and notify areas that need to be addressed, and assists in formulating repair plans.

[0024] Furthermore, step S1, the data collection step, also includes enhancing the collected data, including image rotation, scaling, and brightness adjustment, so as to expand the diversity of the training data set.

[0025] Furthermore, in step S3, in the text feature extraction step, a pre-trained language model, including BERT or GPT, is used to improve the understanding and representation capabilities of complex text descriptions.

[0026] Furthermore, in step S4, in the feature fusion step, in addition to weighted fusion or concatenation, a multi-head attention mechanism (Multi-Head Attention) or a graph neural network (GNN) is also used to more efficiently process complex multimodal data.

[0027] Furthermore, in step S7, in the result output step, the generated visualization images and detailed reports support export in multiple formats, including PDF, JPEG, and CSV, to meet the needs of different users and facilitate subsequent analysis and archiving.

[0028] The technical solution of the present invention has the following beneficial effects: 1. Improve defect recognition accuracy Multimodal learning: This invention integrates image and text information, uses deep convolutional neural networks (CNN) to extract image features, and processes text descriptions through the self-attention mechanism, so that the model can understand the specific description of defects (such as cracks, deformation, etc.), thereby performing more accurate classification and positioning.

[0029] Adaptability to complex scenes: It breaks through the limitation of traditional methods that rely only on single visual information, and improves the ability to identify subtle or hidden defects, especially in complex scenes.

[0030] 2. Enhance automation and real-time performance Large-scale automated testing: The system can automatically capture high-resolution images and generate corresponding text descriptions without human intervention, realizing full process automation from data collection to result output.

[0031] Real-time feedback and alerts: Through real-time feedback mechanisms, managers can be promptly identified and notified of areas that need to be addressed, helping to quickly respond and develop repair plans.

[0032] 3. Reduce reliance on large amounts of labeled data Self-supervised learning and fine-tuning: Self-supervised learning methods can be used to effectively train with a small amount of data and fine-tune specific tasks through transfer learning, reducing the need for large-scale labeled data and lowering data preparation costs.

[0033] Data enhancement technology: Combining multiple data enhancement technologies (such as image rotation, scaling, brightness adjustment, etc.) further enhances the generalization ability of the model and ensures its stable performance in different environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a text annotation schematic diagram of the present invention; Figure 2 It is a schematic diagram of the annotation model of the present invention; Figure 3 is a system block diagram of the present invention; Figure 4 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments; it can be understood that the specific embodiments described herein are only used to explain the present invention rather than to limit the present invention; it should also be noted that, for ease of description, only the parts related to the present invention rather than all are shown in the accompanying drawings.

[0036] Combination Figure 1-Figure 4 As shown, the present invention provides a visual large model defect recognition system based on text prompts, the system comprising: A data acquisition module is used to collect high-resolution images of the building curtain wall surface through drones or other equipment, and provide relevant text descriptions; The visual feature extraction module uses a pre-trained deep convolutional neural network (CNN) to perform multi-layer convolution, pooling, and non-linear activation function processing on the input image to extract high-level semantic features; The text feature modeling module converts text descriptions into embedding vectors and uses the self-attention mechanism to extract text features and generate text representations with contextual information; The multimodal fusion module combines visual features and text features in a weighted fusion or concatenation manner, and uses the self-attention mechanism to dynamically adjust the weights between the two to enhance the importance of related features; The defect classification module inputs the fused features into the classification layer, performs defect detection and classification, and outputs the probability distribution of each potential defect category; Model optimization module, which uses self-supervised learning and data enhancement techniques to fine-tune the model, reduce dependence on large-scale labeled data, and improve generalization ability and detection accuracy; The result output module generates visual images and detailed reports, provides real-time feedback and alerts, and assists managers in analysis and decision-making.

[0037] The data acquisition module also performs data enhancement, including image rotation, scaling, and brightness adjustment, to increase the diversity and reliability of the data set.

[0038] The text feature modeling module further includes a pre-trained language model, including BERT or GPT, to improve the ability to understand complex text descriptions.

[0039] The text feature modeling module adopts the self-attention mechanism in the Transformer architecture to improve the efficiency and accuracy of text feature extraction.

[0040] The multimodal fusion module also adopts a multi-head attention mechanism (Multi-Head Attention) or a graph neural network (GNN) to more effectively process complex multimodal data.

[0041] The present invention provides a method for visual large model defect recognition based on text prompts, the method comprising the following steps: S1, data collection: collect high-resolution images of the building curtain wall surface through drones or other equipment, and provide relevant text descriptions for each image; S2, visual feature extraction: Use a pre-trained deep convolutional neural network (CNN) to process the input image and learn the high-level semantic features of the image through multiple layers of convolution, pooling, and non-linear activation functions; S3, text feature extraction: convert the words or phrases in the text description into word embedding vectors to form the text embedding matrix, and use the self-attention mechanism in the Transformer architecture to extract text features; S4, feature fusion: Combine visual features and text features in a weighted fusion or concatenation manner, and use the self-attention mechanism to dynamically adjust the weight between the two to ensure that the model can focus on the visual and text information most relevant to defect detection; S5, defect detection and classification: the fused features are input into the classification layer to perform defect detection and classification tasks, and the probability distribution of each potential defect category is output; S6, model optimization: use self-supervised learning methods to conduct preliminary training on the model, and fine-tune it for specific building curtain wall defect detection tasks through transfer learning, and combine multiple data enhancement techniques to improve the generalization ability of the model; S7, Result Output: Generates visual images and detailed reports containing defect location and classification information, provides real-time feedback and alerts, helps managers promptly identify and notify areas that need to be addressed, and assists in formulating repair plans.

[0042] Step S1, the data collection step, also includes enhancing the collected data, including image rotation, scaling, and brightness adjustment, to expand the diversity of the training data set.

[0043] In step S3, in the text feature extraction step, a pre-trained language model, including BERT or GPT, is used to improve the understanding and representation capabilities of complex text descriptions.

[0044] In step S4, in the feature fusion step, in addition to weighted fusion or concatenation, a multi-head attention mechanism (Multi-Head Attention) or a graph neural network (GNN) is also used to process complex multimodal data more efficiently.

[0045] In step S7, in the result output step, the generated visualization images and detailed reports support export in multiple formats, including PDF, JPEG, and CSV, to meet the needs of different users and facilitate subsequent analysis and archiving.

[0046] The text-prompt-based visual large-model building curtain wall defect detection system of the present invention mainly includes the following components: Data acquisition module: This module is responsible for acquiring high-resolution images of the building curtain wall surface through drones or other appropriate equipment. At the same time, it also collects text descriptions associated with these images, such as information about crack type, size, location, etc. This information together constitutes a multimodal dataset for training and testing models.

[0047] Visual feature extraction module: The input image is processed using a pre-trained deep convolutional neural network (CNN) to extract high-level visual features. CNN can capture complex patterns and structures in images through a series of operations such as convolutional layers, pooling layers, and activation function learning.

[0048] Text feature modeling module: The self-attention mechanism is used to convert the features and text information extracted from the image into an embedded vector, and further generate a contextual representation of the text. This step uses the technology in the Transformer architecture to effectively understand and represent text content.

[0049] Multimodal fusion module: Combine visual features and text features to form a richer and more accurate feature representation. This module can achieve this through weighted fusion or concatenation, and may also use a self-attention mechanism to enhance the importance of related features, ensuring that the final output features can better reflect the relationship between the image and the text.

[0050] Defect Classification Module: Input the fused features obtained in the previous stage into the fully connected layer or other classifiers to complete the specific classification task of the surface defects of the building curtain wall. In this process, the system will output the probability distribution of each potential defect category to help determine the most likely defect type.

[0051] Model optimization module: In order to adapt the model to specific application scenarios and improve its performance, self-supervised learning methods and various data augmentation techniques are used for fine-tuning. This helps reduce the need for large-scale labeled data and improves the model's generalization ability in different environments.

[0052] Result output and visualization module: The system not only provides defect location and classification results, but also generates intuitive visualization images and detailed report documents. These output forms can help building managers understand the inspection results more easily and take necessary maintenance measures accordingly.

[0053] The above parts work together to enable the entire system to meet the needs of modern building curtain wall surface defect detection on the basis of automation, efficiency and accuracy.

[0054] The method for detecting defects of a large visual model building curtain wall based on text prompts of the present invention comprises the following detailed steps: Data collection and preparation: Using drones or other appropriate equipment, collect high-resolution images of the building's curtain wall surface.

[0055] Each image is accompanied by relevant text descriptions (such as crack type, size, location, etc.) to form a multimodal dataset.

[0056] The collected data are manually annotated and data augmented to increase the diversity and reliability of the data set.

[0057] Visual feature extraction based on textual cues: The input image is processed by a pre-trained deep convolutional neural network (CNN). The CNN model learns the high-level semantic features of the image through multiple layers of convolution, pooling, and non-linear activation functions, and outputs image feature representation.

[0058] Text feature extraction: Convert words or phrases in the text description into word embedding vectors to form an embedding matrix for the text.

[0059] The self-attention mechanism in the Transformer architecture is used to extract text features and generate text representations with contextual information.

[0060] Fusion of visual and textual features: The visual features extracted from the image and the text features extracted from the text are input into a multimodal fusion module.

[0061] The two features are combined by weighted fusion or concatenation, and the weight between the two is dynamically adjusted through the self-attention mechanism to ensure that the model can focus on the visual and textual information most relevant to defect detection.

[0062] Defect Detection and Classification: The fused features are input into a fully connected layer or other classification modules to perform defect detection and classification tasks.

[0063] Output a probability distribution for each potential defect class to determine the most likely defect type.

[0064] Model fine-tuning and optimization: The model is initially trained using a self-supervised learning approach and fine-tuned for a specific building curtain wall defect detection task via transfer learning.

[0065] Combine multiple data enhancement techniques (such as image rotation, scaling, brightness adjustment, etc.) to improve the generalization ability of the model.

[0066] Perform hyperparameter adjustment and training strategy optimization to further improve model performance and increase detection accuracy and efficiency.

[0067] Result output and visualization: The system generates visual images and detailed reports with defect location and classification information.

[0068] Real-time feedback and alert mechanisms help managers promptly identify and notify areas that need to be addressed, assisting in developing remediation plans.

[0069] These steps together constitute a complete text-based visual large-model building curtain wall defect detection process, which aims to achieve efficient, accurate, and intelligent automated detection to meet the needs of modern buildings for curtain wall maintenance.

[0070] Working principle of the present invention: The present invention is a visual large-scale model building curtain wall defect detection system and method based on text prompts, which aims to achieve efficient, accurate and intelligent curtain wall defect detection by combining image and text information. The following is a detailed working principle: 1. Data Collection and Preparation Image acquisition: Use drones or other appropriate equipment to capture high-resolution images of the building curtain wall surface. Drones can cover large areas and obtain images at different angles and heights to ensure comprehensive data collection.

[0071] Text description pairing: Each image is paired with relevant text descriptions, such as crack type, size, location, etc. These text descriptions can be manually input or automatically generated labels to assist in subsequent feature extraction and classification.

[0072] 2. Visual feature extraction Pre-trained CNN model: The collected images are input into a pre-trained deep convolutional neural network (CNN). CNN learns high-level semantic features in images through multi-layer convolution, pooling, and non-linear activation function processing.

[0073] Image feature output: After the above processing, CNN outputs high-level feature representations of the image. These features capture the complex patterns and structures in the image and provide a basis for subsequent fusion and classification.

[0074] 3. Text feature modeling Word embedding vector: Convert the words or phrases in the text description into word embedding vectors to form an embedding matrix for the text. The embedding vector of each word contains its semantic information in the vocabulary.

[0075] Self-attention mechanism: Use the self-attention mechanism in the Transformer architecture to extract text features and generate text representations with contextual information. This step can capture long-distance dependencies in text descriptions and improve the ability to understand complex texts.

[0076] 4. Fusion of visual and textual features Weighted fusion or splicing: The visual features extracted from the image and the text features extracted from the text are input into a multimodal fusion module. The two features are combined by weighted fusion or splicing, and the weight between the two is dynamically adjusted through the self-attention mechanism to enhance the importance of related features.

[0077] Self-attention mechanism: Attention weights are calculated based on the similarity of visual features and text features to ensure that the model can focus on the visual and text information most relevant to defect detection.

[0078] 5. Defect detection and classification Classification layer input: The fused features are input to a fully connected layer or other classification module to perform defect detection and classification tasks. This module predicts the type of defects present in the input image by learning patterns from historical data.

[0079] Probability distribution output: The classification module outputs the probability distribution of each potential defect category to determine the most likely defect type. This helps with subsequent repair decisions and management.

[0080] 6. Model fine-tuning and optimization Self-supervised learning: With a small amount of data, the model can be initially trained through self-supervised learning methods to reduce the dependence on large-scale labeled data.

[0081] Transfer learning and fine-tuning: For specific building curtain wall defect detection tasks, transfer learning is used to fine-tune the model to improve its generalization ability and detection accuracy.

[0082] Data enhancement technology: Combine multiple data enhancement technologies (such as image rotation, scaling, brightness adjustment, etc.) to further improve the robustness and adaptability of the model.

[0083] 7. Result output and visualization Visual image generation: The system generates visual images containing defect location and classification information to help managers intuitively understand the inspection results.

[0084] Detailed report: In addition to the visual images, the system also generates detailed report documents to record the inspection process and results. It supports export in multiple formats such as PDF, JPEG, and CSV for easy archiving and sharing.

[0085] Real-time feedback and alerts: Through the real-time feedback mechanism, managers can be promptly identified and notified of areas that need to be addressed, assisting in the formulation of repair plans.

[0086] In summary, the working principle of the present invention covers the complete process from data collection to result output. Through multimodal learning, self-attention mechanism, self-supervised learning and data enhancement technology, efficient, accurate and intelligent building curtain wall defect detection is achieved to meet the needs of modern building maintenance.

[0087] Embodiment 1: 1. Application Scenarios The present invention is suitable for detecting defects in building curtain walls, especially in large high-rise buildings. It can perform full coverage and high-efficiency detection of the curtain wall surface in an automated manner, avoiding the inefficiency and safety hazards of manual inspection, and solving the limitations of the application scope and data processing capabilities of robot and drone inspections.

[0088] 2. Process 1. Data collection and preparation: This solution uses drones or other equipment to collect high-resolution images of the building curtain wall surface, and adds relevant text descriptions (such as crack type, size, location, etc.) to form a multimodal dataset. The images and text descriptions in the dataset will serve as the basic data for model training, and the diversity and reliability of the data will be further improved through manual annotation and data enhancement.

[0089] 2. Visual macro model based on text prompts: This paper adopts a self-developed visual macro model based on text prompts, which combines deep convolutional neural network (CNN) for image feature extraction and self-attention mechanism for text feature modeling. Specifically, first, a pre-trained CNN is used to extract high-level visual features in the image. The specific process is as follows: Input image: Assume that the input image is Where 𝐻 and 𝑊 are the height and width of the image respectively, and 3 represents the RGB channel. Image feature extraction: through the pre-trained CNN model Extract image features. The model learns the high-level semantic features of the image through multiple layers of convolution, pooling, and nonlinear activation functions. The output is the image features. , where d is the feature dimension

[0090] Next, the features extracted from the image and text information (such as defect description, repair suggestions, etc.) are converted into embedded vectors, and then feature extraction is performed through the self-attention mechanism to generate a vector representation of the text. Assume that the text description is ,in is a word or phrase. Text input: Translate each word Convert to word embedding vector , where 𝑘 is the dimension of word embedding. Then, the word embeddings of all words are combined into the embedding matrix of the text . Text feature extraction: Use the self-attention mechanism in Transformer to extract text features and output the context information representation of the text , where d is the feature dimension.

[0091]

[0092] 3. Fusion of visual and textual features: Visual and textual features need to be combined through a fusion strategy so that both image and textual information can be considered comprehensively in the final model output. The self-attention mechanism is used here, which enhances feature representation by calculating the similarity between visual features and textual features. Fusion mechanism: Image features are combined into and text features Input into a multimodal fusion module and fused by weighted fusion or splicing. Assume that the fused feature is , the relevant features can be enhanced by calculating the attention weights. Weighted fusion:

[0093] Where 𝛼 and 𝛽 are weight parameters that control the importance of visual and text features.

[0094] The symbol “;” indicates the concatenation of features. Self-attention mechanism: The self-attention mechanism dynamically adjusts the fused weights based on the similarity of visual and text features to ensure that the model can focus on the visual and text information most relevant to defect detection.

[0095]

[0096] Where 𝐴 is the attention matrix and d is the dimension of the feature.

[0097] 4. Defect detection and classification: The fused features obtained in the previous step It is input into a fully connected layer or other classification module for defect detection and classification. Assuming there are N types of defects, the classification process can be expressed as:

[0098] Among them, 𝑦 is the defect classification result, and are the weights and biases of the classification layer, respectively, and the output is the probability distribution of each defect category.

[0099] 5. Model fine-tuning and optimization: The self-developed visual large model adopts a self-supervised learning method, which can be effectively trained with a small amount of data, and fine-tuned for specific building curtain wall defect detection tasks through transfer learning. During the fine-tuning process, we combine a variety of data enhancement techniques (such as image rotation, scaling, brightness adjustment, etc.) to enhance the generalization ability of the model and ensure that the model can adapt to different types of curtain wall structures and defect characteristics. Through fine-tuning hyperparameters and training strategies, the performance of the model is optimized to improve the accuracy and efficiency of detection.

[0100] 6 Result output and visualization: The output of the model includes not only the location and classification of defects, but also the generation of visual images and detailed reports for further analysis by building managers. Through real-time feedback and alarm mechanisms, the system can promptly identify and notify managers of areas that need to be addressed, helping to develop repair plans.

[0101] The present invention solves the problems of inefficiency, poor accuracy and limited applicability of existing technologies in building curtain wall defect detection through a self-developed large visual model based on text prompts. Traditional methods mostly rely on single visual information and cannot effectively identify complex defects. The present invention significantly improves the accuracy of defect recognition by fusing image and text information and using multimodal learning, especially in complex scenarios. The text prompt mechanism enables the model to understand the specific description of defects (such as cracks, deformations, breakages, etc.), breaking through the limitations of traditional methods and improving the accuracy of classification and positioning. In addition, the model adopts self-supervised learning and fine-tuning strategies, reducing dependence on a large amount of labeled data and enhancing the generalization ability of the model on different building curtain walls and defect types. Finally, the present invention realizes efficient, accurate and intelligent curtain wall defect detection, meeting the needs of modern building automation, real-time and full coverage.

[0102] The key point of the present invention is the self-developed large visual model based on text prompts, which realizes efficient and accurate detection and classification of building curtain wall defects by combining image and text information. Its innovation lies in the use of multimodal learning, which breaks through the limitations of single visual information and improves the recognition accuracy of complex defects. Through the text prompt mechanism, the model can understand the specific description of the defects (such as cracks, deformations, etc.) and accurately classify them. In addition, self-supervised learning and fine-tuning strategies are adopted to reduce dependence on labeled data and improve the generalization ability of the model. The technical features to be protected include the fusion method of image and text features, the defect detection and classification method based on text prompts, and the fine-tuning process of self-supervised learning, to ensure the applicability and intelligence in various building curtain walls and defect scenarios.

[0103] Example 2: Crack detection of high-rise building curtain wall Background Description A large commercial complex includes several high-rise office buildings with more than 20 floors. Due to long-term exposure to the external environment, cracks of varying degrees appeared on the surfaces of some curtain walls, and comprehensive defect detection was required to ensure building safety.

[0104] Implementation steps Data collection: Use drones to capture high-resolution images of all building facades along a predetermined path, ensuring coverage of every floor and facade.

[0105] Each image is accompanied by a brief text description, such as "north facade, 8th to 15th floors", so that the location information can be accurately associated during subsequent processing.

[0106] Visual feature extraction: The collected images are input into a pre-trained deep convolutional neural network (CNN), and processed through multi-layer convolution, pooling, and non-linear activation functions to extract high-level visual features and capture the crack patterns in the images.

[0107] Text feature modeling: The paired text descriptions are converted into word embeddings, and the self-attention mechanism in the Transformer architecture is used to extract text features and generate text representations with contextual information to help the model understand the specific location and possible severity of the cracks.

[0108] Feature Fusion: The visual features are combined with the text features through weighted fusion, and the self-attention mechanism is used to dynamically adjust the weights between the two to enhance the importance of related features and ensure that the model can focus on the visual and textual information most relevant to crack detection.

[0109] Defect Detection and Classification: The fused features are input into a fully connected layer to perform crack detection and classification tasks, output the probability distribution of each potential crack category, and determine the most likely crack type and its location.

[0110] Model fine-tuning and optimization: A self-supervised learning method is used to initially train the model, and transfer learning is used to fine-tune the model for a specific high-rise building curtain wall crack detection task to improve the generalization ability and detection accuracy.

[0111] Combined with a variety of data enhancement techniques (such as image rotation, scaling, brightness adjustment, etc.), the robustness and adaptability of the model can be further improved.

[0112] Result output and visualization: The system generates visual images containing crack location and classification information, intuitively showing the location and severity of the cracks.

[0113] Generate detailed report documents to record the inspection process and results, and support export in multiple formats such as PDF, JPEG, and CSV for easy archiving and sharing.

[0114] Provides real-time feedback and alerts to promptly notify managers of areas that need to be addressed and assist in developing remediation plans.

[0115] Effect evaluation Multiple hidden cracks that were difficult to find using traditional methods were successfully identified, improving detection efficiency and accuracy.

[0116] Automatically generated visualizations and detailed reports empower maintenance teams and speed up repair decision-making.

[0117] Example 3: Deformation detection of complex curtain wall structure Background Description A modern art museum has a complex curved glass curtain wall design. Over time, some areas have slightly deformed, affecting the building's appearance and posing a potential safety hazard. An efficient method is needed to detect these subtle changes.

[0118] Implementation steps Data collection: A professional-grade drone equipped with a high-resolution camera was used to take detailed photos of the museum's exterior from different angles and heights, ensuring coverage of the entire curved glass curtain wall.

[0119] Equipped with a laser rangefinder to accurately measure the distance of each point, provide additional spatial geometric information, and assist in image analysis.

[0120] Visual feature extraction: The collected images are input into the pre-trained CNN model, and processed through multi-layer convolution, pooling and non-linear activation functions to extract high-level visual features and capture subtle deformations on the surface of the glass curtain wall.

[0121] Text feature modeling: The paired text descriptions (for example, "in the southeast corner, near the entrance, the glass panel has a slight dent") are converted into word embeddings, and the self-attention mechanism is used to extract text features and generate text representations with contextual information, helping the model understand the specific location and form of the deformation.

[0122] Feature Fusion: Visual features are combined with text features by splicing, and the self-attention mechanism is used to dynamically adjust the weight between the two to enhance the importance of related features and ensure that the model can focus on the visual and textual information most relevant to deformation detection.

[0123] Defect Detection and Classification: The fused features are input into a fully connected layer to perform deformation detection and classification tasks, outputting the probability distribution of each potential deformation category and determining the most likely deformation type and its location.

[0124] Model fine-tuning and optimization: A self-supervised learning method is used to preliminarily train the model, and transfer learning is used to fine-tune the model for specific complex curtain wall structure deformation detection tasks to improve the generalization ability and detection accuracy.

[0125] Combined with a variety of data enhancement techniques (such as image rotation, scaling, brightness adjustment, etc.), the robustness and adaptability of the model can be further improved.

[0126] Result output and visualization: The system generates visual images containing deformation location and classification information, intuitively showing the location and severity of the deformation.

[0127] Generate detailed report documents to record the inspection process and results, and support export in multiple formats such as PDF, JPEG, and CSV for easy archiving and sharing.

[0128] Provides real-time feedback and alerts to promptly notify managers of areas that need to be addressed and assist in developing remediation plans.

[0129] Effect evaluation Multiple subtle deformations that are difficult to detect are accurately identified, especially those on complex curved glass curtain walls, improving detection accuracy.

[0130] Automatically generated visualizations and detailed reports provide valuable references for maintenance teams, facilitating rapid response and repairs to keep buildings safe and beautiful.

[0131] These two specific embodiments demonstrate the application effects of the present invention in different scenarios and verify its effectiveness and superiority in detecting cracks in high-rise building curtain walls and deformation of complex curtain wall structures.

[0132] The above description is only a preferred embodiment of the present invention, and does not limit the scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the protection scope of the present invention.

Claims

1. A visual large model defect recognition system based on text prompts, characterized in that: The system includes: A data acquisition module is used to collect high-resolution images of the building curtain wall surface through drones or other equipment, and provide relevant text descriptions; The visual feature extraction module uses a pre-trained deep convolutional neural network to perform multi-layer convolution, pooling, and non-linear activation function processing on the input image to extract high-level semantic features; The text feature modeling module converts text descriptions into embedding vectors and uses the self-attention mechanism to extract text features and generate text representations with contextual information; The multimodal fusion module combines visual features and text features in a weighted fusion or concatenation manner, and uses the self-attention mechanism to dynamically adjust the weights between the two to enhance the importance of related features; The defect classification module inputs the fused features into the classification layer, performs defect detection and classification, and outputs the probability distribution of each potential defect category; Model optimization module, which uses self-supervised learning and data enhancement techniques to fine-tune the model, reduce dependence on large-scale labeled data, and improve generalization ability and detection accuracy; The result output module generates visual images and detailed reports, provides real-time feedback and alerts, and assists managers in analysis and decision-making.

2. The system according to claim 1, characterized in that The data acquisition module also performs data enhancement, including image rotation, scaling, and brightness adjustment, to increase the diversity and reliability of the data set.

3. The system according to claim 1, characterized in that The text feature modeling module further includes a pre-trained language model, including BERT or GPT, to improve the ability to understand complex text descriptions.

4. The system according to claim 1, characterized in that The text feature modeling module adopts the self-attention mechanism in the Transformer architecture to improve the efficiency and accuracy of text feature extraction.

5. The system according to claim 1, characterized in that The multimodal fusion module also adopts a multi-head attention mechanism or a graph neural network to more effectively process complex multimodal data.

6. A method for visual large model defect recognition based on text prompts, based on the system according to any one of claims 1 to 5, characterized in that: The method comprises the following steps: S1, data collection: collect high-resolution images of the building curtain wall surface through drones or other equipment, and provide relevant text descriptions for each image; S2, visual feature extraction: Use a pre-trained deep convolutional neural network to process the input image and learn the high-level semantic features of the image through multi-layer convolution, pooling and non-linear activation functions; S3, text feature extraction: convert the words or phrases in the text description into word embedding vectors to form the text embedding matrix, and use the self-attention mechanism in the Transformer architecture to extract text features; S4, feature fusion: Combine visual features and text features in a weighted fusion or concatenation manner, and use the self-attention mechanism to dynamically adjust the weight between the two to ensure that the model can focus on the visual and text information most relevant to defect detection; S5, defect detection and classification: the fused features are input into the classification layer to perform defect detection and classification tasks, and the probability distribution of each potential defect category is output; S6, model optimization: use self-supervised learning methods to conduct preliminary training on the model, and fine-tune it for specific building curtain wall defect detection tasks through transfer learning, and combine multiple data enhancement techniques to improve the generalization ability of the model; S7, Result Output: Generates visual images and detailed reports containing defect location and classification information, provides real-time feedback and alerts, helps managers promptly identify and notify areas that need to be addressed, and assists in formulating repair plans.

7. The method according to claim 6, characterized in that Step S1, the data collection step, also includes enhancing the collected data, including image rotation, scaling, and brightness adjustment, to expand the diversity of the training data set.

8. The method according to claim 6, characterized in that In step S3, in the text feature extraction step, a pre-trained language model, including BERT or GPT, is used to improve the understanding and representation capabilities of complex text descriptions.

9. The method according to claim 6, characterized in that In step S4, in the feature fusion step, in addition to weighted fusion or concatenation, a multi-head attention mechanism or graph neural network is also used to process complex multimodal data more efficiently.

10. The method according to claim 6, characterized in that In step S7, in the result output step, the generated visualization images and detailed reports support export in multiple formats, including PDF, JPEG, and CSV, to meet the needs of different users and facilitate subsequent analysis and archiving.

Images