A method for determining the conformity of a building engineering component part based on image processing

Abstract

The application relates to the technical field of building engineering quality detection, and discloses a building engineering component standardization determination method based on image processing. The method first extracts structured determination triplets in the standard text by using a large language model, and establishes a mapping relationship between the structured determination triplets and target components in a building information model based on semantic similarity. Then, a site image is acquired, and a pose parameter of a collection terminal relative to the model is calculated. The target depth is compared with the full-scene depth by using a double-channel depth buffer technology, a dynamic three-dimensional projection mask consistent with a viewpoint is generated, and an unobstructed visible area of the target component is accurately identified. Finally, the mask is used to extract a region of interest, cross-modal reasoning is performed by combining a standard constraint condition and a visual language model, and a compliance determination result is output. The application effectively solves the problem of obstruction interference in a complex construction environment, realizes automatic quality acceptance based on semantic understanding, and improves the accuracy and intelligent level of determination.

Description

Technical Field

[0001] This invention relates to the field of building engineering quality inspection technology, specifically to a method for determining the standardization of building engineering components and parts based on image processing. Background Technology

[0002] With the deepening application of Building Information Modeling (BIM) technology throughout the entire lifecycle of building projects, digital quality acceptance has become an important development direction for intelligent construction. In the traditional project acceptance process, quality inspectors typically need to check each component and part completed on-site according to construction quality acceptance specifications, design drawings, or BIM models. This method is not only time-consuming and labor-intensive, but also highly dependent on the professional experience of personnel, making it prone to subjective judgment errors.

[0003] To improve acceptance efficiency, several automated inspection technologies based on computer vision have emerged. These technologies use cameras or mobile devices to capture on-site images and employ deep learning algorithms to identify surface defects such as cracks and honeycomb-like pitting. However, these existing technologies face significant challenges in complex construction site environments. Due to the chaotic nature of construction sites, target components to be inspected are often obscured by scaffolding, construction machinery, temporary materials, or other building structures. Most existing image analysis methods process images in two dimensions, lacking the ability to perceive three-dimensional depth information and thus failing to automatically distinguish target components from foreground obstructions. This often leads to algorithms incorrectly analyzing the surface of obstructions as part of the target component, or resulting in false positives or false negatives due to the inability to locate the effective visible area of ​​the target component.

[0004] Furthermore, existing automated acceptance systems have limitations in handling the correlation between standard texts and digital models. Construction quality acceptance standards are typically in the form of natural language text, while BIM models are structured geometric data, creating a significant semantic gap. Existing technologies usually rely on manually establishing mapping rules between standard clauses and model components, or writing fixed detection algorithms for specific single defects, lacking the ability to understand and logically reason about natural language standards. This means that when acceptance standards change or non-standardized appearance compliance requirements arise, existing systems struggle to adapt flexibly and cannot achieve truly automated judgment based on standard texts. Therefore, effectively solving the target extraction problem in complex occlusion environments and establishing intelligent linkage between standard texts, digital models, and on-site visual information are pressing technical challenges in the field of intelligent acceptance of building engineering projects. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for determining the standardization of building components based on image processing. This method solves the problems of low efficiency in manual comparison, difficulty in handling complex occlusion relationships by traditional augmented reality technology, and lack of semantic understanding in automated determination during the existing building construction quality acceptance process.

[0006] To achieve the above objectives, this invention provides the following technical solution: a method for determining the standardization of building components based on image processing. This method includes the following steps: First, acquiring the text of the building construction quality acceptance standard; then, using a large language model to deeply understand the semantics of the text; and extracting structured decision triples containing the target component name, detection attributes, and compliance constraints. To establish the association between the standard and the digital model, the high-dimensional semantic similarity between the target component name in the structured decision triples and the component attributes in the building information model is calculated, thereby determining the globally unique identifier of the target component in the model.

[0007] Secondly, the on-site images captured by the acquisition terminal are obtained, and the precise pose parameters of the acquisition terminal relative to the world coordinate system of the building information model at the time of capture are calculated.

[0008] Furthermore, based on the geometric data of the target component corresponding to the pose parameters and globally unique identifiers, a dynamic 3D projection mask that strictly matches the viewpoint of the on-site image is generated using depth buffering techniques in computer graphics. The core function of this dynamic 3D projection mask is to identify, at the pixel level, the effective visible area of ​​the target component in the on-site image that is not occluded by foreground objects.

[0009] Finally, a dynamic 3D projection mask is used to extract the region of interest from the on-site image, focusing visual attention on the solid part of the target component. Combining the detection attributes and compliance constraints in the structured decision triplet, a visual language model is used for cross-modal reasoning to output the compliance determination result of the target component.

[0010] Furthermore, in determining the globally unique identifier of the target component, this invention employs a semantic matching strategy based on a vector space model. After parsing the standardized text into triples using a large language model, a set of candidate names is extracted from the Building Information Modeling (BIM) database. By converting the target component name into a high-dimensional semantic feature vector, the cosine similarity between the target component name and the name vectors in the candidate name set is calculated. Components whose similarity meets a preset screening threshold are then bound to the model. This allows the system to automatically handle inconsistencies between standardized terminology and modeling naming, achieving intelligent mapping.

[0011] Furthermore, in calculating pose parameters, this invention employs a registration scheme based on virtual synthetic data. A virtual camera is set up in the Building Information Model (BIM) to render and generate a virtual reference image and a depth index map. A matching relationship is established by extracting feature descriptors from the site image and the virtual reference image, constructing a 2D-3D point pair set. Perspective is then utilized... The point algorithm is used to calculate the initial pose and construct the reprojection error function. The Levenberg-Marquardt algorithm is then used to perform nonlinear iterative optimization of the pose parameters until the reprojection error converges, thereby obtaining high-precision six-degree-of-freedom pose parameters.

[0012] Furthermore, the core innovation of this invention lies in using depth buffering technology to generate dynamic 3D projection masks to solve the occlusion problem. Specifically, the system separates the independent geometric mesh of the target component based on a globally unique identifier, and constructs a view matrix and a perspective projection matrix by combining pose parameters. Dual-pass rendering is performed in the graphics rendering pipeline: the first pass only draws the independent geometric mesh, writing the generated fragment depth values ​​into the target depth buffer; the second pass draws the full scene geometry model including environment objects and enables depth testing, writing the nearest fragment depth value into the full scene depth buffer.

[0013] Furthermore, by comparing the absolute value of the difference between the target depth value and the total scene depth value pixel by pixel, the system can accurately determine the visibility of a pixel. If the absolute value of the difference is less than a preset depth tolerance threshold, it indicates that the target component at that pixel location is the nearest object in the scene and is not occluded; otherwise, it is determined to be occluded. The two-dimensional mask matrix generated based on this logic is then filled with holes through morphological closing operations to form the final dynamic three-dimensional projection mask.

[0014] Furthermore, in the reasoning and judgment stage, this invention utilizes a mask to perform Hadamard product operations on the on-site image, generating a region of interest image containing only the visible portion of the target component, effectively filtering out background noise interference. This image is input into the visual encoder of the visual language model, while the detected attributes and compliance constraints are transformed into prompt word sequences and input into the text encoder. By calculating the matching probability distribution of visual features and text features in the same feature space, automated compliance judgment is achieved.

[0015] Furthermore, to enhance the interpretability of the judgment, when the judgment result is unqualified, the system uses the backpropagation mechanism to calculate the gradient of the attention map relative to the input image, generates a visual heat map and overlays it on the on-site image, intuitively identifying the specific non-compliant areas (such as cracks, rust or missing parts) that caused the unqualification.

[0016] This invention provides a method for determining the standardization of building components based on image processing. It has the following beneficial effects:

[0017] 1. This invention utilizes a large language model to extract structured decision triples from standard texts and automatically maps them to component attributes in a building information model based on semantic similarity calculation in vector space. This solves the industry problem of inconsistencies between design model naming and acceptance specification terminology, and realizes intelligent association from unstructured standards to digital components, reducing the time cost and error rate of manually configuring acceptance rules.

[0018] 2. This invention generates a dynamic three-dimensional projection mask by using a difference comparison technique based on a dual-channel depth buffer. This invention can accurately distinguish target components from complex obstructions (such as scaffolding, pipes, etc.) at the construction site, achieving pixel-level visibility judgment. This ensures that the visual analysis algorithm focuses only on the actual exposed surface of the target component, eliminates the interference of environmental background noise on image analysis, and improves the accuracy and robustness of judgment in complex obstruction environments.

[0019] 3. By combining dynamic mask-guided region of interest extraction with the cross-modal reasoning capability of visual language models, this invention not only achieves automated semantic judgment of component compliance, breaking through the limitation of traditional algorithms that can only identify specific defects, but also uses the backpropagation mechanism to generate visual heat maps, intuitively locating the specific non-compliant areas that lead to non-compliance, providing interpretable and traceable visual evidence for engineering quality acceptance. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the occlusion culling logic based on a depth buffer according to the present invention; Figure 3 This is a schematic diagram comparing the VLM inference performance under the guidance of the dynamic mask of the present invention; Figure 4 This is a schematic diagram of the pixel-level visibility determination logic based on a depth buffer according to the present invention. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] This invention includes an image processing-based system for determining the standardization of building components and parts. The system may include a data acquisition terminal, a server, and a building information model (BIM) database. The data acquisition terminal, server, and BIM database establish a communication connection and transmit data via a communication network.

[0023] The data acquisition terminal is equipped with an optical imaging device. This device is used to acquire two-dimensional image data of the target area at the construction site, defining this data as a site image. The data acquisition terminal is also equipped with a wireless communication module for uploading the acquired site images to a server. The data acquisition terminal can be a mobile computing device equipped with a camera, including smartphones, tablets, augmented reality glasses, or industrial inspection robots.

[0024] Building Information Modeling (BIM) databases are used to store digital model data for the entire lifecycle of building projects. This digital model data includes geometric and topological information as well as non-geometric attribute information for building components. Geometric and topological information includes the 3D vertex coordinates, triangular facet indices, and normal vectors of the components. Non-geometric attribute information includes globally unique identifiers, component category names, material parameters, and design specification parameters. BIM databases are deployed on local server storage media or in cloud storage services.

[0025] The server is equipped with a central processing unit (CPU), system memory, and a graphics processing unit (GPU). The system memory stores computer-executable instructions, which the CPU executes to run the determination method of this invention. The GPU is used to perform parallel tensor operations on deep neural network models, providing computational support for reasoning in large language models and visual language models.

[0026] The server is logically divided into a specification parsing module, a spatial registration module, a mask generation module, and an inference and judgment module. These modules interact with each other via an internal bus or shared memory.

[0027] The specification parsing module is used to process unstructured construction quality acceptance specification texts for building engineering projects. This module loads a large language model fine-tuned with engineering domain data. It performs natural language processing on the input specification text to extract structured data containing target component names, detection attributes, and compliance constraints. The specification parsing module also calculates the semantic similarity between the target component name and component attribute data in the Building Information Modeling (BIM) database, thereby determining a globally unique identifier for the component to be detected.

[0028] The spatial registration module connects to the data acquisition terminal and the Building Information Model (BIM) database. It receives site images and retrieves corresponding 3D scene data from the BIM database. The spatial registration module extracts 2D visual feature points from the site images and 3D geometric feature points from the 3D scene data. Based on the correspondence between the 2D visual feature points and the 3D geometric feature points, it calculates the pose parameters of the data acquisition terminal relative to the BIM world coordinate system at the time of image capture. These pose parameters include a rotation matrix and a translation vector.

[0029] The mask generation module connects to the spatial registration module and the Building Information Modeling (BIM) database. Based on a globally unique identifier, the mask generation module extracts the independent geometric mesh of the target component from the BIM database. It then constructs a virtual projection matrix using pose parameters and performs rasterization rendering on the independent geometric mesh of the target component. Finally, the module calculates the occlusion relationship between the target component and other components in the scene using depth buffering technology, generating a binarized image with the same resolution and viewpoint as the actual scene image. This binarized image is defined as a dynamic 3D projection mask.

[0030] The inference and judgment module is connected to the mask generation module and the specification parsing module. The inference and judgment module receives the on-site image, the dynamic 3D projection mask, and text prompts generated from detection attributes and compliance constraints. The inference and judgment module performs pixel-level multiplication on the on-site image and the dynamic 3D projection mask to filter out background pixels in the on-site image, obtaining the region of interest (ROI) image. The inference and judgment module loads a visual language model to extract the visual feature vectors of the ROI image and the text feature vectors of the text prompts, calculates the matching probability between the visual feature vectors and the text feature vectors, and outputs the compliance judgment result of the component based on this matching probability.

[0031] See attached document Figure 1 This invention provides a method for determining the standardization of building components based on image processing. The method includes the following steps: First, the system performs a mapping step between structured specifications and building information model (BIM) components. This step receives unstructured construction quality acceptance specification text as input data. The system uses a pre-trained large language model to extract entity relations from the specification text, outputting a structured decision triple containing the target component name, visual inspection attributes, and compliance constraints. The system reads the component attribute list from the BIM database, converts the target component name into a high-dimensional text embedding vector, and calculates the cosine similarity between this high-dimensional text embedding vector and the component name vector in the component attribute list. The system selects components with similarity scores exceeding a preset threshold, extracts their globally unique identifiers, and thus establishes a logical index relationship between the specification text description and the 3D digital model entities.

[0032] Next, the system performs spatial pose registration between the site images and the Building Information Model (BIM). The system acquires two-dimensional site images and extracts key two-dimensional feature points from these images. Simultaneously, based on the spatial region corresponding to the aforementioned globally unique identifier, the system extracts the corresponding three-dimensional geometric feature points from the BIM. The system calculates the extrinsic parameter matrix of the camera in the BIM world coordinate system by solving the Perspective-n-Point problem. This extrinsic parameter matrix consists of a rotation matrix. Translation vector Composition. The system incorporates the camera's intrinsic parameter matrix. Constructing from three-dimensional world coordinates to the pixel coordinates of a two-dimensional image The projection transformation model satisfies the following relation: ,in This represents the equality relationship in homogeneous coordinates. The system iteratively optimizes the rotation matrix by minimizing the reprojection error function. Translation vector The reprojection error continues until it is less than the preset convergence threshold, thus completing the geometric alignment of the virtual and real spaces.

[0033] Subsequently, the system executes a dynamic 3D projection mask generation step based on semantic indexing. Using globally unique identifiers, the system extracts the geometric mesh data of the target component from the full dataset of the Building Information Model (BIM). The system utilizes the already calculated rotation matrix... Translation vector A virtual camera viewpoint is set, and the geometric mesh data of the target component is rasterized and rendered. During the rendering process, the system performs a depth buffer test (Z-buffering). The system calculates the depth value of the target component at each pixel location. And the depth value of the full-scene model System comparison and The system determines that the target component corresponding to the pixel is not occluded by the environment only when the absolute value of the difference between the two is less than the depth tolerance threshold. The system sets the pixel value of the unoccluded pixel that belongs to the projection area of ​​the target component to the first value (e.g., 1), and sets the pixel value of other areas to the second value (e.g., 0), thereby generating a dynamic 3D projection mask with the same resolution as the on-site image.

[0034] Finally, the system executes a mask-guided visual language reasoning and judgment step. The system performs a Hadamard product operation on the on-site image and a dynamic 3D projection mask. This operation retains pixel information in the on-site image where the mask value is for the first region, and replaces pixel information in the on-site image with background fill values ​​for the region where the mask value is for the second region, generating a region of interest (ROI) image. The system inputs the ROI image into a visual encoder to extract visual feature vectors, and simultaneously converts visual detection attributes and compliance constraints into natural language prompt word sequences, which are input into a text encoder to extract text feature vectors. The system maps the visual feature vectors and text feature vectors to the same feature space, calculates their feature matching degree, or generates a compliance prediction probability through a decoder. The system compares this prediction probability with a preset judgment threshold. If the prediction probability meets the compliance conditions, a qualified conclusion is output; otherwise, an unqualified conclusion is output, and the non-compliant region in the image is marked.

[0035] This embodiment details how to use a large language model to transform unstructured natural language specifications into machine-executable building information model index instructions.

[0036] The system first loads an electronic text document of the Construction Engineering Quality Acceptance Standard as input data. This text document contains unstructured natural language paragraphs. The standard parsing module within the system is configured with a predefined prompt word template. This prompt word template includes a task instruction field, an input text field, and an output format constraint field. The task instruction field defines the entity extraction rules, requiring the model to identify the detection objects, detection dimensions, and specific acceptance indicators in the text. The output format constraint field specifies that the model outputs the results in a machine-readable data exchange format (e.g., JSON). The system then fills the input data into the input text field of the prompt word template, combining them to form a complete input sequence.

[0037] The system feeds the input sequence into the large language model for inference generation. The large language model outputs a structured sequence of decision triples. For each canonical entry, this decision triple is defined as follows: .in, The semantic name representing the target component, such as the anchor plate of an embedded part. Represents the visual attribute to be detected, such as surface smoothness. This represents logical constraints for compliance, such as no obvious indentation and an indentation depth of less than 2 millimeters. This process transforms ambiguous natural language into structured data with clear logical boundaries.

[0038] The system then performs a semantic mapping operation to determine the semantic name of the target component in the triple. This is linked to specific component entities in the Building Information Modeling (BIM) database. The system iterates through the BIM database, extracting the attribute name and type name fields of all components to construct a candidate name set. The system then calls a text embedding model to embed the semantic name of the target component. Transform into high-dimensional semantic feature vectors and each name in the candidate name set Transform into the corresponding high-dimensional candidate feature vector .

[0039] System Calculation With each Cosine similarity between them. The calculation formula is as follows: ; Where · represents the vector dot product, This represents the Euclidean norm of the vector. This calculation process measures the semantic similarity between the standard descriptive vocabulary and the architectural model naming vocabulary in the vector space, thereby resolving the inconsistency between standard terminology (such as fire hydrant) and model parametric naming (such as Hydrant_Type_A_001).

[0040] The system sets a similarity filtering threshold. The system filters out components corresponding to candidate names whose cosine similarity is greater than this threshold and obtains the globally unique identifier (GUID) of these components. The system then extracts the decision triplet. Bind to the selected globally unique identifier to generate a pair of tasks to be detected. The task to be detected serves as the driving data for subsequent image processing flows and is stored in the task queue. Through the above steps, the system achieves a precise logical index from abstract text specifications to specific three-dimensional digital components.

[0041] This embodiment describes in detail how to establish a geometric correspondence between two-dimensional on-site observation data and three-dimensional digital model data through computer vision algorithms, thereby determining the precise position and orientation of the camera in the architectural coordinate system.

[0042] The spatial registration module first acquires the on-site images uploaded by the data acquisition terminal and reads the camera's intrinsic parameter matrix. Intrinsic parameter matrix Includes focal length, principal point coordinates, and pixel tilt factor, used to describe the optical imaging characteristics inside the camera. To establish feature correspondence, the system sets up a virtual camera in the Building Information Model (BIM) based on the pose estimate of the previous frame or preset coarse positioning information (such as construction area markers), and renders a virtual reference image and a corresponding depth index map. This depth index map records the three-dimensional coordinates of each pixel in the virtual reference image in the BIM world coordinate system.

[0043] The system utilizes a feature extraction network to process on-site images and virtual reference images in parallel. The feature extraction network performs multi-scale convolution operations on the images, detecting corners, edges, or texture-rich regions as keypoints, and calculating a high-dimensional feature descriptor for each keypoint. The system establishes feature point matching pairs by calculating the Euclidean distance between the feature descriptors of the on-site image and the virtual reference image. For each successfully matched feature point pair, the system extracts the two-dimensional pixel coordinates of the feature points from the on-site image. Based on the pixel positions of corresponding feature points in the virtual reference image, the depth index map is indexed to obtain the corresponding three-dimensional spatial coordinates. Therefore, the system constructs a set of... A set of 2D-3D point pairs with elements.

[0044] The system employs a Perspective-n-Point (PnP) algorithm combined with a Random Sample Consensus (RANSAC) strategy to solve for camera pose. The system randomly selects a subset from the set of point pairs, assuming they satisfy the projection transformation equation, and calculates the assumed rotation matrix. Translation vector The projection transformation equation is described as follows: ; in, The scaling factor is used. The system uses the remaining point pairs to verify the correctness of the hypothetical model, counts the number of inliers, iterates repeatedly to eliminate mismatched points generated during feature matching, and selects the model parameter with the largest number of inliers as the initial pose estimate.

[0045] To further improve registration accuracy, the system constructs a nonlinear least squares optimization problem. The system defines a reprojection error function, which characterizes the geometric distance between the pixel coordinates of all interior points on the observation plane and the predicted coordinates of the 3D points projected back to the image plane via the current pose. The system uses the Levenberg-Marquardt algorithm to iteratively optimize the reprojection error function. The system fine-tunes the rotation matrix. Translation vector The value of this parameter minimizes the sum of the reprojection error functions. When the optimization process converges, the system outputs the final pose parameters. These pose parameters precisely define the 3D position of the camera's optical center in the building information model coordinate system and the spatial orientation of the optical axis at the moment of image acquisition, providing a geometric reference for accurately mapping 3D semantic information back to the 2D image.

[0046] This embodiment details how the mask generation module prepares the geometric data foundation and virtual projection environment for generating dynamic masks.

[0047] The mask generation module first performs a geometric data retrieval operation based on globally unique identifiers. This module establishes a data connection with the Building Information Modeling (BIM) database and, based on the globally unique identifier of the target component determined in the preceding steps, queries the detailed geometric attributes of the component. The system parses the BIM data structure, extracting a list of 3D vertex coordinates constituting the surface of the target component and a list of vertex indices defining the connection relationships of triangular facets.

[0048] Since the geometric data of components in Building Information Modeling (BIM) is usually stored in the component's own local coordinate system, in order to achieve alignment with the site scene, the system reads the spatial transformation matrix of the component in the overall building model, i.e., the model matrix. The system extracts the coordinates of each 3D vertex. By applying model matrix transformation, it is converted into absolute coordinates in the world coordinate system of Building Information Modeling. The transformation calculation follows the formula. ,in This includes the translation, rotation, and scaling parameters of the component during design. Through this step, the system constructs an independent geometric mesh object containing only the target component. The object's location in the virtual space is consistent with its physical location in the actual building.

[0049] While preparing the geometric data, the system initializes a virtual camera object in the graphics rendering pipeline. This virtual camera is designed to simulate the optical imaging process of a physical data acquisition terminal. The system first reads the intrinsic parameter matrix of the data acquisition terminal. The intrinsic parameter matrix Including focal length and principal point coordinates .

[0050] The system is based on the intrinsic parameter matrix Constructing the perspective projection matrix This matrix defines the view frustum shape of the virtual space and maps 3D spatial points to the normalized device coordinate system (NDC). Simultaneously, the system sets the virtual camera's resolution parameters to match the pixel resolution of the on-site image. Strict consistency is required to ensure that the subsequently generated mask corresponds to the on-site image at the pixel level.

[0051] The system further reads the extrinsic parameter matrix calculated by the spatial registration module, namely the rotation matrix. Translation vector The system constructs the view matrix based on this extrinsic parameter matrix. The view matrix is ​​used to transform geometric points in the world coordinate system to the virtual camera coordinate system. Defined as the inverse of the extrinsic parameter matrix or its equivalent transformation form. The system will... and Loaded into the shader program of the graphics rendering pipeline, it serves as a globally unified variable for subsequent rasterization rendering. Through the above settings, the system constructs a digital twin observation environment in computer memory that is completely synchronized with the physical capture moment, providing a mathematical basis for generating semantic masks that accurately cull occlusions.

[0052] This embodiment describes in detail the computation process by which the mask generation module uses a graphics processing unit (GPU) to convert three-dimensional geometric data into a two-dimensional discrete fragment sequence.

[0053] The system first loads the target component's independent geometric mesh data and the entire scene's geometric mesh data extracted in the preceding steps into the graphics processing unit's video memory. The system then configures the programmable rendering pipeline and activates the vertex shader and fragment shader. The system passes pre-calculated model matrix, view matrix, and perspective projection matrix as unified variables to the vertex shader.

[0054] After the rendering command is issued, the vertex shader performs a coordinate space transformation on each input 3D vertex. The vertex shader multiplies the local coordinates of the vertex sequentially by the model matrix, the view matrix, and the perspective projection matrix, outputting clipping space coordinates. The system hardware performs a primitive assembly operation, assembling the transformed vertices into triangular facet primitives according to the index list. The system performs a view frustum clipping operation on the triangular facet primitives, removing primitive portions located outside the virtual camera's view frustum range, and performs perspective division on the retained primitives to convert the coordinates to normalized device coordinates (NDC).

[0055] Subsequently, the system performs a viewport transformation operation, linearly mapping the normalized device coordinates to a screen pixel coordinate system consistent with the resolution of the on-site image. The rasterizer scans and transforms each triangular facet in screen space, discretizing it into a sequence of fragments covering specific pixel locations. During this process, the rasterizer uses a barycentric coordinate interpolation algorithm to calculate the depth value (Z-value) of each generated fragment based on the depth information of the triangle vertices.

[0056] To distinguish target components from the environment, the system employs a marker-based shading strategy in the fragment shader stage. When rendering the geometric mesh of the target component, the fragment shader is configured to output a first marker value (e.g., pure white or a logical value of 1) to the color buffer; when rendering the entire scene environment mesh, the fragment shader is configured to output only depth data to the depth buffer or output a second marker value (e.g., pure black or a logical value of 0). Through this rasterization process, the system transforms continuous 3D geometric surfaces into a 2D discrete data stream with screen coordinates, depth information, and semantic markers, providing a pixel-level data foundation for subsequent occlusion detection.

[0057] See attached document Figure 2 This embodiment describes in detail how the mask generation module uses depth information to determine the pixel-level visibility of the target component, thereby eliminating areas occluded by the foreground.

[0058] The mask generation module allocates two independent floating-point frame buffers in video memory, defined as the target depth buffer and the full scene depth buffer, respectively. The resolution of these two buffers is configured to be strictly consistent with the scene image. The system initializes these two buffers by setting the initial depth value of all pixel positions to infinity or the maximum value in the normalized depth space (e.g., 1.0).

[0059] The system executes the first rendering pass, drawing only the individual geometric meshes of the target component. During rasterization, the system writes the depth value of each fragment of the target component to the corresponding pixel position in the target depth buffer. For pixel regions not covered by the target component, the target depth buffer retains its initial maximum value. The depth map generated in this step is denoted as... .

[0060] The system executes a second rendering pass, drawing a full-scene geometry model including the target component and all environmental objects (such as walls, pipes, and supports). During this process, the system enables the DepthTest function. For each pixel location on the screen, the system retains only the fragment depth value closest to the virtual camera and writes it to the full-scene depth buffer. The depth map generated in this step is... . It reflects the closest visible depth of all object surfaces in the scene from the current viewpoint.

[0061] The system initiates the occlusion culling calculation kernel, which calculates the coordinates of each pixel on the image plane. The visibility determination logic is executed in parallel. The system reads data from the same coordinate position. and The system first checks If the value is equal to the initial maximum value, it indicates that the target component has no projection at that pixel position, and the system directly determines that the pixel is invisible.

[0062] like For the effective depth value, the system calculates the absolute value of the difference between the target depth value and the total scene depth value. The system introduces a small depth tolerance threshold. (Bias) is used to offset numerical jitter caused by precision errors in floating-point calculations and quantization errors in the depth buffer. The system performs a comparison and judgment: like This indicates that at this pixel location, the surface depth of the target component is basically consistent with the nearest surface depth in the entire scene, meaning that the target component is the nearest object in the current viewing direction and is not occluded.

[0063] like And under normal circumstances This indicates that there is another environmental object with a smaller depth (i.e., closer to the camera) at that pixel location, and the target component is located behind this environmental object and is in an occluded state.

[0064] Through the pixel-by-pixel depth comparison logic described above, the system can accurately distinguish between the self-occlusion and mutual occlusion states of the target component, ensuring that only physically visible surface areas are retained for subsequent image analysis.

[0065] This embodiment details how the mask generation module transforms the logical results of depth buffer comparison into a two-dimensional binary matrix for image processing.

[0066] The system initializes a single-channel two-dimensional matrix in memory as a mask container. The number of rows in this matrix... Number of columns It is configured to have a strictly equal vertical and horizontal pixel resolution to the scene image. The system sets the initial value of all elements in the matrix to the second value (e.g., logic 0 or grayscale value 0), representing the default background area.

[0067] The system iterates through the coordinates of every discrete pixel on the image plane. Based on the depth comparison results in the aforementioned embodiments, a state assignment operation is performed. The system defines a mask generation function. For any coordinates If the position satisfies both of the following conditions: First, the depth value in the target depth buffer. If the value is less than the preset infinity indicator value, then the pixel is located within the geometric projection range of the target component. Second, the absolute value of the difference between the target depth value and the total scene depth value. Less than the depth tolerance threshold This means that the pixel is not occluded by other objects in the scene; The system will then assign the coordinates The mask value at that location is set to the first value (e.g., logic 1 or grayscale value 255).

[0068] The above logical assignment process can be represented by the following piecewise function formula: ; in, This represents the maximum initial depth value of the depth buffer.

[0069] To eliminate jagged edges caused by insufficient sampling frequency and hole noise due to floating-point errors in depth calculation during rasterization, the system performs morphological filtering on the generated raw binary mask. The system defines a structuring element, such as a 3×3 square or circular kernel. First, the system performs a morphological closing operation on the raw mask, i.e., dilation followed by erosion, to fill the tiny holes inside the mask and smooth discontinuous boundaries. Subsequently, the system performs edge softening by convolving the binary boundaries of the mask with a Gaussian blur kernel, generating a soft mask with grayscale transitions, or maintaining hard thresholding for strict region clipping. The final generated... It is stored as a single-channel image tensor, which serves as a spatial attention map indicating the effective visual region of the target component, and is then transmitted to the subsequent inference and decision-making module.

[0070] This embodiment describes in detail how the reasoning and judgment module combines the spatial attention mechanism of a dynamic 3D projection mask with the semantic understanding capability of a visual language model to perform automated compliance verification of building components.

[0071] The inference and determination module first performs image localization processing on the region of interest (ROI). The system receives the original scene image. and the dynamic 3D projection mask generated in the aforementioned steps To eliminate visual interference from complex backgrounds (such as scaffolding, cluttered materials, and irrelevant personnel) on the judgment model, the system performs a Hadamard product operation, i.e., element-wise multiplication, on the on-site image and the projection mask.

[0072] The operation logic is as follows: For pixels with a mask value of the first value (effective area), the system retains their original RGB color information; for pixels with a mask value of the second value (background area), the system replaces them with a preset background average value. Or zero value. The tensor of the generated region of interest image. The calculation formula is: ; in, This is the Hadamard product (element-wise multiplication). Through this operation, the system forces the visual model's attention to be fully focused on the physically visible surface of the target component.

[0073] The system then constructs a multimodal input data stream. In the visual channel, the system will... The input is fed into a pre-trained visual encoder network. This network performs multi-layer convolutional or self-attention mechanisms to map the high-dimensional image pixel matrix into a low-dimensional dense visual feature vector. In the text channel, the system calls the structured data output by the specification parsing module, including detected attributes. and compliance constraints The system uses prompt word engineering templates to reorganize structured data into natural language queries. For example, "Do the detection attributes of the target component in the image meet compliance constraints?" The system will... The text is input into a text encoder network to extract the corresponding text feature vectors. .

[0074] The system performs cross-modal feature fusion and inference judgment. The system combines visual feature vectors... With text feature vectors The input is fed into a multimodal fusion layer. This fusion layer evaluates the semantic matching degree between visual representations and text descriptions by calculating vector dot products, cosine similarity, or performing nonlinear mapping through a multilayer perceptron (MLP). The system uses a softmax function to map the matched data into a normalized probability distribution. .

[0075] The decision logic is as follows: ; in, This represents the fusion inference function. The system sets the decision threshold. (For example, 0.8). If The probability value representing the compliance category is greater than If the system outputs a pass / fail result, it will output a fail / unqualified result.

[0076] Furthermore, to provide interpretable criteria for judgment, the system activates an attention backtracking mechanism when the judgment result is unsatisfactory. The system calculates the attention map of the last layer in the inference network relative to the input image. The gradient is used to generate a visual heatmap. This heatmap is overlaid on the original site image, highlighting the specific pixel areas that led to the non-compliance judgment (such as rust spots or crack locations). The final judgment, probability confidence level, and heatmap evidence are packaged and output to the user terminal.

[0077] See attached document Figure 3 and attached Figure 4 Specific application example: Automated acceptance of the appearance quality of precast shear walls.

[0078] This embodiment demonstrates how to use the method of the present invention to automatically inspect the surface appearance defects of a precast concrete shear wall on a construction site, especially in the case of scaffolding obstruction on site.

[0079] Component to be inspected: A precast shear wall with the number W_L3_E_04 in the BIM model.

[0080] Acceptance standard: GB50204, "Code for Acceptance of Construction Quality of Concrete Structures".

[0081] Site conditions: A row of temporary scaffolding steel pipes was placed in front of the wall, obscuring part of the wall. The wall surface actually has a honeycomb-like pitted defect.

[0082] Implementation steps: Step S1: Standardization and mapping.

[0083] Input: Input section 8.2.1 of GB50204 into the system.

[0084] LLM output (triple): (Target): Surface of concrete shear wall (Attribute): Appearance Quality (Constraints): There should be no serious defects (such as exposed reinforcement, holes, slag inclusions, looseness, cracks), and general defects should be repaired.

[0085] Mapping: The system calculates semantic similarity and automatically maps concrete shear walls to GUIDs in the BIM model.

[0086] Step S2: Spatial registration.

[0087] Data collection: The inspection robot moves in front of the wall to take pictures.

[0088] Solution: The system identifies image features and uses the PnP algorithm to calculate the position of the robot camera in the BIM coordinate system (12.5, 3.0, 1.6), facing due north.

[0089] Step S3: Dynamic 3D projection mask generation.

[0090] Dual-channel rendering: Channel A (Target): Renders only the shear wall with GUID 1f8a9c…. Depth Map The wall is shown to be about 3 meters away from the camera.

[0091] Channel B (Full Scene): Renders all models (including wall and construction facility models). Since the BIM includes scaffolding models (or depth maps from real-time site scans), the depth map... The depth at the scaffold location is 1.5 meters (closer to the wall), and at the wall location it is 3 meters.

[0092] Difference determination: In the pixel area obscured by the scaffolding: If it is determined to be an obstruction, the mask value is set to 0.

[0093] In the exposed areas of the wall: If the mask is deemed visible, its value is set to 1.

[0094] Result: A binary mask with the shape of a scaffold "hollowed out" is generated.

[0095] Step S4: Reasoning and determination.

[0096] ROI Extraction: Multiply the site photo with a mask. The scaffolding area turns black (background), retaining only the wall pixels.

[0097] VLM Inference: Input prompt "Detect whether there are honeycombs or exposed tendons in the image?".

[0098] Result: The model ignored the black scaffolding area, accurately identified the honeycomb pitting in the upper right corner of the wall, output "unqualified", and superimposed a red heat map on the result image.

Claims

1. A method for determining the standardization of building components and parts based on image processing, characterized in that, Includes the following steps: Obtain the text of the construction quality acceptance specification for building engineering, extract structured decision triples using a large language model, and calculate the semantic similarity between the target component name in the structured decision triples and the component attributes in the building information model to determine the globally unique identifier of the target component. Acquire on-site images captured by the acquisition terminal, and calculate the pose parameters of the acquisition terminal relative to the world coordinate system of the building information model at the time of capture; Based on the geometric data of the target component corresponding to the pose parameters and the globally unique identifier, a dynamic three-dimensional projection mask consistent with the viewpoint of the scene image is generated using depth buffering technology. The dynamic three-dimensional projection mask is used to identify the visible area of ​​the target component in the scene image that is not occluded. The region of interest is extracted from the scene image using the dynamic 3D projection mask. Combined with the detection attributes and compliance constraints in the structured decision triplet, the compliance determination result of the target component is output using a visual language model.

2. The method for determining the standardization of building components and parts based on image processing according to claim 1, characterized in that, The text of the construction quality acceptance specification for building engineering is input into the large language model, and the structured decision triplet containing the target component name, the detection attribute and the compliance constraint is output. Extract the attribute names of all components from the building information model database to construct a candidate name set; The target component name is converted into a high-dimensional semantic feature vector, and the cosine similarity between the high-dimensional semantic feature vector and each name vector in the candidate name set is calculated. Select the components corresponding to candidate names whose cosine similarity is greater than a preset screening threshold, and extract the globally unique identifier of the components.

3. The method for determining the standardization of building components and parts based on image processing according to claim 1, characterized in that, The steps for calculating the pose parameters of the acquisition terminal relative to the world coordinate system of the building information model at the time of shooting include: A virtual camera is set in the building information model to render and generate a virtual reference image and a depth index map that records three-dimensional coordinate information. Extract the two-dimensional key feature points of the scene image and the reference key feature points of the virtual reference image, and establish feature matching pairs based on the distance of the feature descriptors; Based on the feature matching pairs, obtain the two-dimensional pixel coordinates in the scene image and the corresponding three-dimensional spatial coordinates in the depth index map, and construct a 2D-3D point pair set; Using perspective The point algorithm processes the set of 2D-3D point pairs to obtain the pose parameters, which include rotation matrices and translation vectors.

4. The method for determining the standardization of building components and parts based on image processing according to claim 3, characterized in that, After obtaining the pose parameters including the rotation matrix and translation vector through the solution, the method further includes: Construct a reprojection error function, which is used to characterize the geometric distance between the predicted coordinates of the three-dimensional spatial coordinates in the 2D-3D point pair set after being projected onto the image plane by the rotation matrix and translation vector and the two-dimensional pixel coordinates. The reprojection error function is iteratively optimized using the Levenberg-Marquardt algorithm until its value is less than a preset convergence threshold. The optimized rotation matrix and translation vector are then used as the final pose parameters.

5. The method for determining the standardization of building components and parts based on image processing according to claim 1, characterized in that, The step of generating a dynamic 3D projection mask consistent with the viewpoint of the scene image using depth buffering technology, based on the geometric data of the target component corresponding to the pose parameters and the globally unique identifier, includes: The independent geometric mesh of the target component is separated from the building information model based on the globally unique identifier; A view matrix is ​​constructed using the pose parameters, and a perspective projection matrix is ​​constructed by combining the intrinsic parameter matrix of the acquisition terminal. In the graphics rendering pipeline, the view matrix and the perspective projection matrix are used to perform rasterization rendering on the independent geometric mesh and the full-scene geometric model of the building information model, respectively.

6. The method for determining the standardization of building components and parts based on image processing according to claim 5, characterized in that, The rasterization rendering and the generation process of the dynamic 3D projection mask specifically include: The first rendering pass is executed, drawing only the independent geometric mesh, and writing the generated fragment depth values ​​into the target depth buffer to obtain pixel-level target depth values; The second rendering pass is executed to draw the full scene geometry model and enable depth testing. The generated nearest fragment depth value is written to the full scene depth buffer to obtain the pixel-level full scene depth value. For each pixel position corresponding to the scene image, calculate the absolute value of the difference between the target depth value and the full scene depth value; If the absolute value of the difference is less than a preset depth tolerance threshold, the pixel position is determined to belong to the unoccluded area of ​​the target component.

7. The method for determining the standardization of building components and parts based on image processing according to claim 6, characterized in that, After determining that the pixel position belongs to the unobstructed area of ​​the target component, the method further includes: Initialize a two-dimensional mask matrix with the same resolution as the scene image; The values ​​of the pixel positions identified as the unmasked areas in the two-dimensional mask matrix are set to the first value, and the values ​​of the other pixel positions are set to the second value; A morphological closing operation is performed on the two-dimensional mask matrix to fill the holes, resulting in the final dynamic three-dimensional projection mask.

8. The method for determining the standardization of building components and parts based on image processing according to claim 1, characterized in that, The steps for extracting the region of interest from the scene image using the dynamic 3D projection mask include: Perform a Hadamard product operation on the on-site image and the dynamic 3D projection mask; The pixel information of the scene image corresponding to the region with the first value in the dynamic three-dimensional projection mask is retained; The pixel information of the on-site image corresponding to the region with the second value in the dynamic three-dimensional projection mask is replaced with a preset background fill value to generate a region of interest image that only contains the visible part of the target component.

9. The method for determining the standardization of building components and parts based on image processing according to claim 8, characterized in that, The step of using a visual language model to output the compliance determination result of the target component includes: The image of the region of interest is input into the visual encoder of the visual language model to extract visual feature vectors; The detection attributes and compliance constraints in the structured decision triples are converted into natural language prompt word sequences, which are then input into the text encoder of the visual language model to extract text feature vectors. Map the visual feature vector and the text feature vector to the same feature space, and calculate the feature matching probability distribution; If the probability value representing the compliance category in the feature matching probability distribution is greater than the preset judgment threshold, the qualified compliance judgment result is output.

10. The method for determining the standardization of building components and parts based on image processing according to claim 9, characterized in that, After outputting the compliance determination result of the target component, the following is also included: When the compliance determination result is unqualified, the gradient of the attention map of the visual language model relative to the region of interest image is calculated using backpropagation; A visual heatmap is generated based on the gradient and then overlaid on the on-site image to identify non-compliant areas.

Images