A method and system for quality inspection of real-scene 3D models integrating deep learning

Abstract

This invention provides a method and system for quality inspection of real-scene 3D models that integrates deep learning. It constructs a real-scene 3D model by acquiring multispectral imaging data of the target scene; optimizes the multispectral imaging data to generate standardized input data; based on this, a deep learning-driven semantic segmentation network is used to extract correlation features between multispectral bands; combined with the geometric structural features of the real-scene 3D model, pixel-level segmentation results are generated; and verification is performed based on preset topological continuity rules to generate a multispectral feature association dataset. The quality of the real-scene 3D model is then checked based on the comparison results of the geometric feature segmentation deviations between different bands with preset tolerance ranges, yielding the quality inspection results. This invention achieves refined quality inspection of real-scene 3D models, effectively identifying and locating abnormal regions with geometric structural breaks or distortions, and improving the geometric consistency and overall quality of real-scene 3D models under complex lighting conditions.

Description

Technical Field

[0001] This invention relates to the field of 3D model inspection technology, and in particular to a method and system for quality inspection of real-world 3D models that integrates deep learning. Background Technology

[0002] Geometric consistency checks on real-world 3D models under complex lighting conditions, especially in scenes with interlacing shadows and high dynamic range, are crucial for ensuring the accuracy and reliability of the models. These applications typically involve urban planning, virtual reality, and cultural heritage preservation, where high-quality 3D models are fundamental. Due to variations in natural light and the effects of shadows cast by buildings or other objects, acquired image data often exhibits high contrast and complex lighting conditions, posing challenges to 3D reconstruction algorithms. To ensure that the generated 3D models are not only realistic in appearance but also geometrically accurate, a technique capable of effectively handling data under these complex lighting conditions is needed.

[0003] The mainstream solution for this need is a method based on deep learning segmentation models driven by monospectral image data, combined with point cloud registration verification. This approach extracts geometric features from monospectral image data through a pre-trained semantic segmentation network, generates preliminary segmentation results, and then uses point cloud registration technology to compare the geometric consistency between the 3D model and the segmentation results, marking discrepancies as anomalies. However, this approach has some drawbacks. For example, relying on monospectral data makes it sensitive to lighting interference such as shadows and highlights, resulting in low feature extraction capabilities in dark or overexposed areas and a higher false detection rate. Point cloud registration verification cannot effectively distinguish between real structural fractures and texture differences caused by lighting, especially at the boundaries of multi-scale curved surfaces, where it is prone to missed detections. Summary of the Invention

[0004] This invention provides a method and system for quality inspection of real-world 3D models that integrates deep learning, in order to solve the problems of low feature extraction capability and high false detection rate in dark or overexposed areas in the prior art; inability to effectively distinguish between real structural fractures and texture differences caused by lighting; and the tendency to miss detection, especially at the intersection of multi-scale curved surfaces.

[0005] In a first aspect, the present invention provides a method for quality inspection of real-world 3D models that integrates deep learning, comprising:

[0006] Acquire multispectral imaging data of the target scene, and construct a real-world 3D model based on the multispectral imaging data;

[0007] The multispectral imaging data is optimized to generate standardized input data;

[0008] Based on the standardized input data, a deep learning-driven semantic segmentation network is used to extract the correlation features between multispectral bands, and combined with the triangular mesh connection relationship of the real-scene 3D model, pixel-level segmentation results are generated.

[0009] Based on preset topological continuity rules, the pixel-level segmentation results are verified to generate a multispectral feature association dataset.

[0010] Based on the comparison results of the geometric feature segmentation deviation between different multispectral bands in the multispectral feature association dataset and the preset tolerance range, the real-scene 3D model is subjected to quality inspection to obtain quality inspection results. The quality inspection results include abnormal areas with geometric structure fractures or geometric structure distortions.

[0011] Optionally, based on the standardized input data, a deep learning-driven semantic segmentation network is used to extract the correlation features between multispectral bands, and combined with the triangular mesh connection relationship of the real-scene 3D model, pixel-level segmentation results are generated, including:

[0012] The standardized input data is fed into a deep learning-driven geometrically constrained semantic segmentation network to extract the correlation features between multispectral bands;

[0013] Obtain the target-level feature map output by the geometrically constrained semantic segmentation network, and perform upsampling and feature concatenation on the target-level feature map to generate a fused feature map;

[0014] Based on the triangular mesh connection relationship of the real scene 3D model, the weighted relationship between the feature similarity and segmentation prediction difference of adjacent pixel pairs in the fused feature map is calculated to generate a constraint weight matrix.

[0015] The segmentation prediction probability distribution of the fused feature map is adjusted according to the constraint weight matrix so that the segmentation prediction values ​​of adjacent pixels satisfy the geometric continuity constraint, resulting in the adjusted fused feature map.

[0016] A pixel-by-pixel classification operation is performed on the adjusted fused feature map to generate pixel-level segmentation results.

[0017] Optionally, based on the triangular mesh connection relationship of the real-scene 3D model, a weighted relationship between the feature similarity and segmentation prediction difference of adjacent pixel pairs in the fused feature map is calculated to generate a constraint weight matrix, including:

[0018] Based on the connection relationship of the surface triangular mesh of the real-world 3D model, the adjacent pixel pairs corresponding to the edges of each triangular facet in the fused feature map are determined.

[0019] Extract the feature vector of each pixel in the multispectral channels of adjacent pixel pairs in the fused feature map to calculate the feature similarity;

[0020] Extract the segmentation prediction probability distribution of each pixel in the adjacent pixel pairs in the fused feature map to calculate the segmentation prediction difference.

[0021] Based on the preset weighting coefficients, the feature similarity and the segmentation prediction difference are normalized and linearly superimposed to generate the weighted constraint value corresponding to each pair of adjacent pixels.

[0022] The weighted constraint values ​​are mapped to the fused feature map to construct a constraint weight matrix with the same resolution as the fused feature map.

[0023] Optionally, the segmentation prediction probability distribution of the fused feature map is adjusted according to the constraint weight matrix to ensure that the segmentation prediction values ​​of adjacent pixels satisfy the geometric continuity constraint, resulting in an adjusted fused feature map, including:

[0024] The adjustment coefficient for each pixel is calculated based on the constraint strength values ​​of adjacent pixel pairs in the constraint weight matrix.

[0025] The segmentation prediction probability distribution of each pixel in the adjacent pixel pair in the fused feature map is adjusted using the adjustment coefficient to obtain the superimposed segmentation prediction probability distribution;

[0026] The superimposed segmentation prediction probability distribution is normalized to generate an updated segmentation prediction probability distribution.

[0027] The updated segmentation prediction probability distribution replaces the segmentation prediction probability distribution of the corresponding pixel in the fused feature map to generate an adjusted fused feature map, so that the segmentation prediction values ​​of adjacent pixels satisfy the geometric continuity constraint.

[0028] Secondly, the present invention provides a quality inspection system for real-world 3D models that integrates deep learning, comprising:

[0029] The acquisition module is used to acquire multispectral imaging data of the target scene and construct a real-scene 3D model based on the multispectral imaging data.

[0030] The optimization module is used to optimize the multispectral imaging data and generate standardized input data.

[0031] The extraction module is used to extract the correlation features between multispectral bands based on the standardized input data using a deep learning-driven semantic segmentation network, and combine the triangular mesh connection relationship of the real-scene 3D model to generate pixel-level segmentation results.

[0032] The verification module is used to verify the pixel-level segmentation results based on preset topological continuity rules and generate a multispectral feature association dataset.

[0033] The inspection module is used to perform quality checks on the real-world 3D model based on the comparison results of the geometric feature segmentation deviation between different multispectral bands in the multispectral feature association dataset with a preset tolerance range, and to obtain quality inspection results. The quality inspection results include abnormal areas with geometric structure fractures or geometric structure distortions.

[0034] Thirdly, the present invention provides a computing device including a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform a method for quality inspection of a real-world 3D model incorporating deep learning as described in any of the first aspects.

[0035] Fourthly, the present invention provides a computer storage medium storing computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the method for quality inspection of a real-world 3D model incorporating deep learning as described in any one of the first aspects.

[0036] In this invention, multispectral imaging data of a target scene is acquired, and a real-world 3D model is constructed based on the multispectral imaging data. The multispectral imaging data is optimized to generate standardized input data. Based on the standardized input data, a deep learning-driven semantic segmentation network is used to extract the correlation features between multispectral bands, and combined with the triangular mesh connection relationship of the real-world 3D model, pixel-level segmentation results are generated. Based on a preset topological continuity rule, the pixel-level segmentation results are verified to generate a multispectral feature association dataset. According to the comparison results of the geometric feature segmentation deviation between different multispectral bands in the multispectral feature association dataset with a preset tolerance range, the real-world 3D model is subjected to quality checks to obtain quality check results, which include abnormal regions with geometric structural breaks or distortions. The technical solution provided by this invention captures the complete reflection characteristics of scenes under complex lighting conditions using multispectral imaging data (including visible and non-visible light bands), compensating for the texture loss problem in shadow and highlight areas of single-spectral data, providing high-fidelity input for the construction of realistic 3D models, and enhancing the ability to restore geometric details; through frequency domain noise suppression and adaptive illumination equalization operations, it eliminates sensor noise and uneven illumination interference, generating standardized data with consistent illumination and controllable noise, solving the problem of unstable model feature extraction caused by data quality fluctuations in traditional methods; and it utilizes deep learning networks to extract correlation features between multispectral bands and combines them with the geometric structure of the 3D model. This system enables high-precision segmentation of pixel-level geometric categories (such as planes, edges, and holes), overcoming the bottleneck of missegmentation in complex geometric regions by traditional single-spectral models. It verifies the segmentation results through preset geometric topology rules (such as prohibiting planes and holes from being adjacent), and combines feature difference quantification analysis between multispectral bands to locate potential abnormal regions, solving the false detection / false negative problem caused by pure data-driven models ignoring physical structure rules. Through dynamic matching of difference values ​​and tolerance ranges in multispectral feature association datasets, it distinguishes between real geometric anomalies and lighting interference, outputting structured quality inspection results, achieving objectivity and interpretability of quality inspection results in complex lighting scenarios.Standardized input data is fed into a semantic segmentation network driven by a deep learning network to extract multispectral correlation features and generate a target-level feature map. Multi-scale information is fused by upsampling and shallow feature concatenation to construct a fused feature map. Based on the triangular mesh connection relationship of the real-world 3D model, the feature similarity and segmentation difference of adjacent pixel pairs are quantified to generate a constraint weight matrix. This matrix is ​​used to dynamically adjust the segmentation prediction probability distribution, forcing the predicted values ​​of adjacent pixels to meet geometric continuity. Finally, highly consistent pixel-level segmentation results are output through pixel-by-pixel classification. By dynamically embedding multispectral feature fusion and geometric topological constraints, the robustness of the segmentation model in detecting geometrically broken and distorted regions under complex lighting scenes is improved. This effectively solves the problem of high false detection rate caused by the limitations of single spectral data and lack of rule constraints in traditional methods, while ensuring that the pixel-level segmentation results are strictly consistent with the surface topological rules of the real-world 3D model.

[0037] These or other aspects of the invention will become more apparent from the following description of the embodiments. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a method for quality inspection of a real-world 3D model that integrates deep learning, provided as an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the structure of a real-scene 3D model quality inspection system that integrates deep learning, provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0043] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Figure 1 This invention provides a flowchart of a method for quality inspection of real-world 3D models that integrates deep learning, as shown in the following embodiment. Figure 1 As shown, the method includes:

[0046] To address the issues of insufficient segmentation accuracy and limited point cloud registration verification rules caused by the sensitivity of single-spectral image data-driven methods to shadows and highlights under complex lighting conditions, this invention proposes a collaborative verification mechanism that integrates multispectral data and geometric topology rules. By introducing multispectral imaging data (such as visible light, near-infrared, and thermal infrared bands) and utilizing the complementarity of different bands to illumination interference (such as near-infrared penetrating shadows and thermal infrared weakening highlights), combined with frequency domain noise suppression and dynamic illumination equalization preprocessing, standardized input data resistant to interference is generated. Furthermore, a geometrically constrained semantic segmentation network is designed, embedding the triangular mesh topological relationship of the 3D model into the deep learning feature extraction process. Through the fusion of inter-band correlation features and geometric structure features, pixel-level segmentation accuracy in complex illumination areas is improved. Finally, a multispectral feature association dataset is constructed. Based on the dual verification of inter-band geometric feature segmentation deviation and topological continuity rules, the inconsistencies of abnormal regions are quantified (such as significant differences in reflection between visible light and near-infrared bands in crack areas, violating the plane-hole adjacency rule), thereby accurately distinguishing between real geometric defects and illumination artifacts. This addresses the pain points of high false detection rates and multi-scale missed detections in traditional methods, achieving high robustness in checking the geometric consistency of 3D models under complex illumination scenes. Based on this, this invention provides a method for quality inspection of real-world 3D models that integrates deep learning, such as... Figure 1 ,include:

[0047] Step 101: Acquire multispectral imaging data of the target scene, and construct a real-world 3D model based on the multispectral imaging data;

[0048] In this step, the target scene requires a realistic physical environment for 3D modeling and quality inspection, such as building facades and terrain surfaces. Multispectral imaging data refers to a collection of images containing visible light and at least one non-visible light band (such as infrared), with each multispectral band recording the reflection or radiation characteristics of the scene at different wavelengths. The realistic 3D model refers to a 3D digital model reconstructed based on multispectral data, representing the geometric structure and surface texture of the target scene.

[0049] In this embodiment of the invention, visible light and non-visible light band (such as near-infrared and thermal infrared) data of the target scene are acquired using a multispectral imaging device (such as a multispectral camera or lidar) to generate multispectral imaging data containing multi-band texture information. Based on this data, the three-dimensional point cloud coordinates of the scene are calculated using a motion reconstruction algorithm or laser point cloud registration technology, and a real-world three-dimensional model with texture maps is generated using a surface reconstruction algorithm (such as Poisson reconstruction). This model characterizes the geometric structure and surface properties of the target scene, providing a spatial reference for subsequent quality inspection.

[0050] Step 102: Optimize the multispectral imaging data to generate standardized input data;

[0051] In this step, the standardized input data refers to the multispectral image after noise suppression, illumination equalization, and normalization, with its size and numerical range adapted to the input requirements of the deep learning model.

[0052] In this embodiment of the invention, frequency domain noise suppression (such as fast Fourier transform to filter out high-frequency noise) and adaptive illumination equalization (such as local histogram equalization) are performed on the multispectral imaging data to eliminate shadow and highlight interference, thereby obtaining the second target imaging data. Subsequently, the multi-band images in the second target imaging data are cropped to a preset size and normalized to a uniform numerical range (such as [0,1]) to generate standardized input data adapted to the deep learning network. This process ensures processing efficiency through dynamic resource allocation strategies (such as edge computing nodes adjusting computing power according to the amount of data).

[0053] Step 103: Based on the standardized input data, use a deep learning-driven semantic segmentation network to extract the correlation features between multispectral bands, and combine the triangular mesh connection relationship of the real-scene 3D model to generate pixel-level segmentation results;

[0054] In this step, the associated features refer to the joint representations between multispectral bands extracted by the semantic segmentation network driven by the deep learning network, such as the correlation between near-infrared and visible light reflectance characteristics. Pixel-level segmentation results refer to the pixel classification results output by the deep learning network, with each pixel labeled as a geometric structure type such as planar, edge, or hole. Triangular mesh connectivity refers to the edge connection topology between triangular facets on the surface of the real-world 3D model, including the three edges of each triangular facet and the vertices they connect.

[0055] In this embodiment of the invention, standardized input data is input into a deep learning-driven semantic segmentation network. The encoder extracts inter-spectral band correlation features (such as the correlation between near-infrared and visible light reflectance characteristics), and the decoder combines the triangular mesh connection relationship of the real-world 3D model (such as edge adjacency information) to output pixel-level segmentation results. The label of each pixel identifies its geometric structure type (such as planar, edge, or hole type).

[0056] Step 104: Based on the preset topological continuity rules, verify the pixel-level segmentation results and generate a multispectral feature association dataset;

[0057] In this step, the preset topological continuity rule refers to the predefined geometric adjacency constraints, such as allowing planar classes to be adjacent to edge classes, and prohibiting planar classes from being adjacent to hole classes.

[0058] In this embodiment of the invention, based on the geometric structure type label of each pixel in the adjacent pixel pair obtained from the pixel-level segmentation result, a segmentation label combination is generated, and it is verified whether the segmentation label combination conforms to a preset topological continuity rule (e.g., planes and holes are prohibited from being adjacent). For segmentation label combinations that violate the rule, the difference in reflection intensity of the corresponding adjacent pixel pairs in each band of the multispectral spectrum (e.g., the sum of squared differences in the near-infrared band) is calculated, and a weighted comprehensive difference value is generated. The position coordinates, comprehensive difference values, and preset channel weight coefficients of all segmentation label combinations and their corresponding adjacent pixel pairs are associated and stored as a multispectral feature association dataset.

[0059] Step 105: Based on the comparison results of the geometric feature segmentation deviation between different multispectral bands in the multispectral feature association dataset and the preset tolerance range, perform a quality check on the real-scene 3D model to obtain the quality check results. The quality check results include abnormal areas with geometric structure breakage or geometric structure distortion.

[0060] In this step, geometric feature segmentation deviation refers to the inconsistency in segmentation labels for the same location across different bands, reflecting geometric structural inconsistencies. The preset tolerance range refers to a set of anomaly detection thresholds pre-defined according to the scene type, used to dynamically adapt to anomaly detection standards. The comparison result refers to the comparison result (exceeding or not exceeding) between the sum of comprehensive difference values ​​and the regional tolerance value, used to trigger anomaly labeling. The quality inspection result refers to a detection report containing the location, type, and severity of geometric structural fractures or distortions.

[0061] In this embodiment of the invention, a layered difference threshold is defined based on the resolution level and scene type of the real-world 3D model. The comprehensive difference value in the multispectral feature dataset is compared with this threshold to filter target adjacent pixel pairs. These target pixel pairs are marked as anomaly regions to be confirmed, and their coordinates are mapped onto the surface of the 3D model for spatial adjacency traversal. Triangular faces sharing edges and vertices are merged to form candidate anomaly regions. The final anomaly region is determined by comparing the sum of the comprehensive difference values ​​of all adjacent pixel pairs within the candidate region with a preset tolerance value, including geometric structural breaks or distortions. Finally, the location, range, label type, and sum of difference values ​​of the final anomaly region are associated and stored to generate a quality inspection result. This process effectively identifies and records anomaly regions in the 3D model.

[0062] For example, taking the detection of cracks in the walls of ancient buildings as an example, firstly, visible light (texture) and near-infrared (material) data of the ancient building walls are collected using a multispectral camera to construct a real-scene 3D model; high-frequency noise in the infrared images in the near-multispectral imaging data is filtered out, and the brightness of the shadow areas in the visible light images is balanced to generate standardized input data; a semantic segmentation network driven by deep learning is used to identify the wall surface (planar type) and cracks (hole type), and the anomaly of adjacent planes and holes is verified; the difference in reflection intensity between cracks and walls in the near-infrared band is calculated to be significant (e.g., a difference of 0.8), and the comprehensive difference value is 0.72; the quality of cracks and walls is judged, and if the comprehensive difference value of 0.72 > the tolerance threshold of 0.6, it is marked as a crack anomaly, and the quality inspection results containing the abnormal area and the severity of the anomaly are output.

[0063] This invention addresses the problems of high false detection rates and missed topological fractures caused by lighting interference in traditional real-world 3D model quality inspection. Standardized input data improves model robustness, and multispectral feature correlation verification enhances the reliability of anomaly detection, enabling high-precision and automated geometric quality inspection in complex lighting scenarios.

[0064] This invention provides a specific embodiment. Step 103 involves extracting the correlation features between multispectral bands based on the standardized input data using a deep learning-driven semantic segmentation network, and combining this with the triangular mesh connection relationship of the real-world 3D model to generate pixel-level segmentation results. Specifically, this includes the following steps:

[0065] Step 301: Input the standardized input data into a deep learning-driven geometrically constrained semantic segmentation network to extract the correlation features between multispectral bands;

[0066] In this step, the associated features refer to the features generated by fusing multispectral band information through a geometrically constrained semantic segmentation network driven by a deep learning network, including joint representations of visible and non-visible light bands.

[0067] In this embodiment of the invention, the geometrically constrained semantic segmentation network performs cross-channel feature fusion on visible and non-visible light bands (such as near-infrared and thermal infrared) in the standardized input data through multi-layer superimposed feature extraction units. By sharing cross-channel weight parameters, it establishes the correlation between different bands and finally outputs the correlation features between multispectral bands.

[0068] Step 302: Obtain the target-level feature map output by the geometrically constrained semantic segmentation network, perform upsampling and feature concatenation on the target-level feature map, and generate a fused feature map;

[0069] In this step, the target-level feature map refers to the low-resolution feature map output from the deep layers of the semantic segmentation network, which contains abstracted relational features. The fused feature map refers to the feature map containing multi-scale geometric structural information generated through upsampling and feature concatenation operations.

[0070] In this embodiment of the invention, the target layer feature map is a low-resolution feature map output from the deep layer of the network (e.g., the resolution is 1 / 16 of the original input). It is gradually upsampled to a higher resolution (e.g., 1 / 8, 1 / 4, 1 / 1) through a bilinear interpolation algorithm. After each upsampling, the feature map of the current layer is spatially aligned with the intermediate feature map of the same resolution in the shallow layer of the network and spliced ​​along the channel dimension to finally generate a fused feature map that integrates multi-scale geometric structure information.

[0071] Step 303: Based on the triangular mesh connection relationship of the real scene 3D model, calculate the weighted relationship between the feature similarity of adjacent pixels and the segmentation prediction difference in the fused feature map, and generate a constraint weight matrix;

[0072] In this step, feature similarity refers to the similarity measure of adjacent pixels across multiple spectral channels, calculated using a cosine similarity algorithm. Segmentation prediction difference refers to the inconsistency measure of adjacent pixels on segmentation prediction labels, calculated based on class difference values ​​(such as cross-entropy). The weighted relationship refers to the calculation relationship of a linear combination of feature similarity and segmentation prediction difference according to a preset ratio. The constraint weight matrix refers to a matrix generated based on the weighted relationship, where the matrix element values ​​reflect the strength of the geometric continuity constraint between adjacent pixels.

[0073] In this embodiment of the invention, based on the triangular mesh connection relationship of the real-world 3D model, the adjacent pixel pairs corresponding to the edges of each triangular facet in the fused feature map are determined; the feature vector of each pixel in the multispectral channel is extracted, and the feature similarity is calculated; the segmentation prediction probability distribution vector of each pixel is extracted, and the segmentation prediction difference is calculated; according to the preset weighting coefficient, the feature similarity and the segmentation prediction difference are superimposed to generate the weighted constraint value corresponding to each adjacent pixel pair; the weighted constraint value is mapped to the fused feature map to construct a constraint weight matrix, and each element value in the matrix reflects the geometric continuity strength that should be satisfied between adjacent pixels.

[0074] Step 304: Adjust the segmentation prediction probability distribution of the fused feature map according to the constraint weight matrix so that the segmentation prediction values ​​of adjacent pixels satisfy the geometric continuity constraint, and obtain the adjusted fused feature map;

[0075] In this step, the segmentation prediction value refers to the class probability value of a single pixel in the segmentation prediction result (e.g., [0.2, 0.8] indicates an 80% probability of belonging to the second class). Geometric continuity constraint refers to the rule that forces the segmentation results of adjacent pixels to satisfy topological continuity, implemented through a constraint weight matrix. The adjusted fused feature map refers to the feature map after applying the geometric continuity constraint, whose segmentation prediction result has eliminated isolated outliers.

[0076] In this embodiment of the invention, the constraint weight matrix and the segmentation prediction probability distribution of the fused feature map are adjusted to force the prediction probability values ​​of adjacent pixels to be close to the same (e.g., when the weight of the edge region is higher, the probability of adjacent pixels being predicted to be of the same category increases), thereby eliminating isolated outliers in the segmentation results and making the segmentation prediction values ​​of adjacent pixels satisfy the geometric continuity constraint, thus obtaining the adjusted fused feature map.

[0077] Step 305: Perform a pixel-by-pixel classification operation on the adjusted fused feature map to generate pixel-level segmentation results.

[0078] In this embodiment of the invention, the maximum value point function is used to maximize the multi-class probability value of each pixel in the adjusted fused feature map to generate the final pixel-level segmentation result. The label value of each pixel corresponds to its geometric structure type (such as plane, edge, hole).

[0079] This invention addresses the problems of high false detection rates of geometric structures and missed detections of topological fractures caused by complex lighting interference in traditional 3D model quality inspection. By embedding physical geometric rules into the deep learning inference process through a constraint weight matrix, it improves the detection robustness in complex scenes (such as shadow-interlaced regions). Feature stitching operations preserve multi-scale geometric details and avoid the loss of small-scale structures. Pixel-by-pixel classification ensures high-precision spatial positioning of the detection results.

[0080] This invention provides a specific embodiment. Step 303 involves calculating the weighted relationship between the feature similarity and segmentation prediction difference of adjacent pixels in the fused feature map based on the triangular mesh connection relationship of the real-scene 3D model, and generating a constraint weight matrix. This specifically includes the following steps:

[0081] Step 311: Based on the connection relationship of the surface triangular mesh of the real scene 3D model, determine the adjacent pixel pairs corresponding to the edges of each triangular facet in the fused feature map;

[0082] In this step, a triangular patch edge refers to the boundary line segment of a single triangular patch in the real-world 3D model, formed by connecting two vertices, and is used to define the geometric association between adjacent pixel pairs. An adjacent pixel pair refers to two pixels that share the same triangular patch edge in the fused feature map, representing a spatial adjacency relationship defined based on the geometric structure of the real-world 3D model.

[0083] In this embodiment of the invention, the surface triangular mesh structure of the real-world 3D model is first analyzed, and the edge connection information of all triangular facets is extracted. Based on the spatial position of the triangular facet edges in the real-world 3D model, they are mapped to the two-dimensional pixel coordinate system of the fused feature map, and two pixels sharing the same side are determined to be adjacent pixel pairs. This process ensures that adjacent pixel pairs are strictly associated with the geometric topology of the 3D model.

[0084] Step 312: Extract the feature vector of each pixel in the multispectral channels of adjacent pixel pairs in the fused feature map to calculate the feature similarity;

[0085] In this step, a multispectral channel refers to a data channel containing visible light and at least one non-visible light band (such as thermal infrared), characterizing the reflection or radiation properties of different bands in a scene. A feature vector is a vector formed by arranging the feature values ​​of a single pixel in the multispectral channels in channel order, used to quantify the multispectral characteristics of a pixel. Feature similarity is a measure of the similarity of features between adjacent pixel pairs in the multispectral channels, obtained by calculating the reciprocal of the sum of squared channel differences; a larger value indicates more consistent spectral features.

[0086] In this embodiment of the invention, the feature values ​​of each pixel in the multispectral channels (such as visible light, near infrared, and thermal infrared) of the fused feature map are vectorized to generate feature vectors; for each pair of adjacent pixels, the reciprocal of the sum of squared differences between the feature vectors of the two pixels in each channel is calculated as the feature similarity, where the smaller the sum of squared differences, the more similar the spectral features, and the larger the result of the reciprocal calculation.

[0087] Step 313: Extract the segmentation prediction probability distribution of each pixel in the adjacent pixel pairs in the fused feature map to calculate the segmentation prediction difference.

[0088] In this step, the segmentation prediction probability distribution refers to the set of probability values ​​for each pixel in an adjacent pixel pair in the fused feature map, indicating that each pixel belongs to a different geometric structure category in the segmentation prediction result. This reflects the confidence of the deep learning model in predicting the pixel category. The segmentation prediction discrepancy refers to the measure of inconsistency between adjacent pixel pairs in the segmentation prediction result. It is calculated by summing the absolute values ​​of the category probability differences; a larger value indicates a more significant prediction contradiction.

[0089] In this embodiment of the invention, the segmentation prediction probability distribution of each pixel in the fused feature map is obtained. By traversing each pixel position in the fused feature map, its corresponding multi-dimensional feature vector is read, and this vector is converted into probability values ​​of each geometric category by a classifier to generate the segmentation prediction probability distribution of each pixel. For example, the probability distribution of pixel A is 0.7 for the planar category and 0.3 for the hole category, indicating that the probability of this pixel belonging to the planar region is 70%, and the probability of belonging to the hole region is 30%. For each pair of adjacent pixels, the sum of the absolute differences of the probability distribution vectors of the two pixels in each category is calculated as the segmentation prediction difference degree, where the larger the sum of the differences, the more inconsistent the segmentation prediction results are.

[0090] Step 314: Based on the preset weighting coefficients, normalize and linearly superimpose the feature similarity and the segmentation prediction difference to generate the weighted constraint value corresponding to each pair of adjacent pixels;

[0091] In this step, the preset weighting coefficient refers to the pre-defined weight ratio parameter between feature similarity and segmentation prediction difference, used to control the contribution ratio of the two to the final constraint strength. The weighted constraint value refers to the numerical value generated by normalized linear superposition, reflecting the geometric continuity constraint strength that adjacent pixel pairs should satisfy; the larger the value, the stronger the constraint.

[0092] In this embodiment of the invention, the feature similarity and segmentation prediction difference are normalized so that their values ​​are in the range of [0,1]. The normalized feature similarity is multiplied by a preset feature similarity weight coefficient, and the normalized segmentation prediction difference is multiplied by a segmentation prediction difference weight coefficient. The two are added together to generate a weighted constraint value.

[0093] Step 315: Map the weighted constraint values ​​to the fused feature map to construct a constraint weight matrix with the same resolution as the fused feature map;

[0094] In this embodiment of the invention, based on the row and column positions of adjacent pixel pairs corresponding to the triangular facet edges in the fused feature map, the row and column indices of each element in the initial constraint weight matrix are determined, wherein the row and column coordinates of the two pixels in each adjacent pixel pair are respectively used as the row and column indices of the matrix element; the weighted constraint values ​​of each adjacent pixel pair are simultaneously filled into the corresponding row and column indices and their symmetrical positions in the initial constraint weight matrix, such that the element values ​​of row index i, column index j are the same as those of row index e, column index f in the matrix; for pixel pairs in the fused feature map not associated with the triangular facet edges, preset default constraint values ​​are filled into the corresponding positions in the initial constraint weight matrix; after traversing all adjacent pixel pairs to complete the filling operation of the initial constraint weight matrix, a constraint weight matrix with the same resolution as the fused feature map is generated, wherein the value of each matrix element reflects the geometric continuity constraint strength that the corresponding pixel pairs need to satisfy.

[0095] This invention addresses the problems of high false detection rates and difficulties in verifying geometric consistency under complex lighting conditions caused by the separation of rules and data in traditional 3D model quality inspection. By deeply integrating the geometric structural features of the 3D model with the characteristics of multispectral data through a constraint weight matrix, the detection accuracy and robustness of fractured or distorted regions are improved.

[0096] For example, in the quality inspection of 3D models of roof tiles for ancient buildings, the process begins by traversing the edges of all triangular facets based on the edge relationships of the triangular mesh on the tile surface. Two pixels sharing the same edge are defined as adjacent pixel pairs, ensuring coverage of all tile seam areas. Secondly, for each adjacent pixel pair, feature vectors in the visible light (e.g., red, green, and blue bands) and near-infrared bands are extracted. The sum of squared differences in reflectance between adjacent pixels in each band is calculated. If the sum of squared differences is small (e.g., for intact tiles where the difference is less than the threshold of 5 due to consistent material), it is considered to have high feature similarity; if the sum of squared differences increases significantly (e.g., for detached tiles where the difference exceeds the threshold of 20 due to exposed substrate), the feature similarity decreases. Next, the segmentation prediction probability distribution vector of each pixel in the adjacent pixel pair is extracted (such as the probability of "hole type" and the probability of "planar type"). The sum of the absolute values ​​of the probability difference between the two is calculated. If the sum of the absolute values ​​of the difference exceeds the threshold of 0.7 (for example, if the probability of the detached area being predicted as "hole type" is 0.9 and the probability of the adjacent complete area being predicted as "planar type" is 0.85, then the sum of the absolute values ​​of the difference is 0.15 + 0.15 = 0.3, which does not exceed the threshold; while if the difference of the detached edge increases significantly to 0.8), it is judged as a high segmentation prediction difference. Subsequently, based on the preset weighting coefficients (feature similarity weight 0.6, segmentation prediction difference weight 0.4), the feature similarity and segmentation prediction difference are normalized to the [0,1] interval and then linearly superimposed to generate a weighted constraint value. For example, if the feature similarity of a pair of adjacent pixels is 0.8 and the segmentation prediction difference is 0.3, then the weighted constraint value is (0.8×0.6) + (1-0.3×0.4) = 0.72 (high constraint value); while if the feature similarity is 0.3 and the segmentation prediction difference is 0.8, then the weighted constraint value is (0.3×0.6) + (1-0.8×0.4) = 0.18 + 0.24 = 0.42 (low constraint value). Finally, the weighted constraint values ​​of all adjacent pixel pairs are mapped to a constraint weight matrix with the same resolution as the fused feature map. Regions with high constraint values ​​(such as the seams of complete tiles) correspond to high weight values ​​in the matrix, forcing the segmentation results to maintain continuity; regions with low constraint values ​​(such as detached edges) correspond to low weight values, allowing abrupt changes in the segmentation boundaries.

[0097] This invention provides a specific embodiment. Step 304 involves adjusting the segmentation prediction probability distribution of the fused feature map according to the constraint weight matrix, so that the segmentation prediction values ​​of adjacent pixels satisfy the geometric continuity constraint, thereby obtaining the adjusted fused feature map. This specifically includes the following steps:

[0098] Step 321: Calculate the adjustment coefficient for each pixel based on the constraint strength values ​​of adjacent pixel pairs in the constraint weight matrix;

[0099] In this step, the constraint strength value refers to the strength of the geometric continuity constraint on adjacent pixel pairs in the constraint weight matrix; a larger value indicates that the prediction results of adjacent pixels need to be more consistent. The adjustment coefficient refers to the adjustment magnitude calculated based on the constraint strength and probability difference, used to force the predicted values ​​of adjacent pixels to converge.

[0100] In this embodiment of the invention, for each pixel and all its neighboring pixels (determined based on triangular mesh connections), the constraint strength value of the corresponding pair of neighboring pixels is obtained from the constraint weight matrix; the absolute value of the probability difference between the pixel and each neighboring pixel in the same geometric category is calculated (e.g., probability difference for planar class = |0.7-0.2| = 0.5), and the constraint strength value is multiplied by the absolute value of the probability difference to generate an adjustment coefficient for each neighboring pixel and each category. For example, if the constraint strength is 0.9 and the probability difference is 0.5, then the adjustment coefficient = 0.9 × 0.5 = 0.45.

[0101] Step 322: Adjust the segmentation prediction probability distribution of each pixel in the adjacent pixel pair in the fused feature map using the adjustment coefficient to obtain the superimposed segmentation prediction probability distribution;

[0102] In this step, the superimposed segmentation prediction probability distribution refers to the unnormalized probability distribution after the adjustment coefficients are applied to the original segmentation prediction probability distribution, and the sum of possible probabilities is not equal to 1.

[0103] In this embodiment of the invention, the adjustment operation achieves convergence of prediction results for adjacent pixels by directly increasing or decreasing the probability value. Specifically, for each geometric category of each pixel, if the segmentation prediction probability distribution of the adjacent pixel in that category is higher than that of the current pixel, the adjustment coefficient is subtracted from the segmentation prediction probability distribution of the current pixel; otherwise, it is increased. For example, if the segmentation prediction probability distribution of the planar category for pixel A is 0.7, and its adjacent pixel B is 0.2 (probability difference 0.5), and the adjustment coefficient is 0.45, then after adjustment, the segmentation prediction probability distribution of the planar category for pixel A = 0.7 - 0.45 = 0.25, and the segmentation prediction probability distribution of the planar category for pixel B = 0.2 + 0.45 = 0.65.

[0104] Step 323: Normalize the superimposed segmentation prediction probability distribution to generate an updated segmentation prediction probability distribution;

[0105] In this step, the updated segmentation prediction probability distribution refers to the normalized legal probability distribution, with a probability sum of 1 and retaining the adjustment effect.

[0106] In this embodiment of the invention, the normalization process ensures that the sum of the adjusted probability values ​​is 1. Specifically, the superimposed segmentation prediction probability distributions of each pixel are summed by category, and the total value is calculated (e.g., 0.25 for planar class + 0.75 for hole class = 1.0). Then, the superimposed segmentation prediction probability distributions of each category are divided by the total value. For example, if the superimposed segmentation prediction probability distribution of pixel A is 0.25 for planar class and 0.75 for hole class, the normalized values ​​remain 0.25 and 0.75. If the superimposed segmentation prediction probability distribution is 0.5 for planar class and 0.6 for hole class (total 1.1), then the updated segmentation prediction probability distribution after normalization is 0.5 / 1.1≈0.45, 0.6 / 1.1≈0.55.

[0107] Step 324: Replace the segmentation prediction probability distribution of the corresponding pixel in the fused feature map with the updated segmentation prediction probability distribution to generate an adjusted fused feature map so that the segmentation prediction values ​​of adjacent pixels satisfy the geometric continuity constraint.

[0108] In this step, the adjusted fused feature map refers to the feature map after the updated probability distribution replaces the original segmentation prediction probability distribution, which is used for subsequent pixel-by-pixel classification to generate the final pixel-level segmentation result.

[0109] In this embodiment of the invention, the normalized and updated segmentation prediction probability distribution of each pixel is re-encoded into a multi-dimensional feature vector according to the category order, replacing the original feature vector at the corresponding position in the fused feature map. For example, if the original feature vector is [0.7, 0.3], and it is adjusted to [0.25, 0.75], then the original data is directly overwritten to form a new adjusted feature map. In the adjusted fused feature map, wall pixels tend to be more inclined to the crack class, and crack region pixels tend to be more inclined to the planar class. However, due to the high constraint strength, the prediction difference between the two is significantly reduced, which meets the geometric continuity requirement of the wall-crack transition region.

[0110] This invention addresses the problem of false detections due to breaks or distortions caused by ignoring topological rules in traditional models by dynamically adjusting the predicted probability distribution of adjacent pixels to force pixel-level segmentation results to meet geometric continuity constraints. The adjustment coefficients are adaptively generated based on the constraint strength and probability differences driven by multispectral processing, avoiding manual intervention. Normalization ensures the legality of the probabilities, ultimately improving the segmentation accuracy and structural rationality in complex scenarios.

[0111] This invention provides a specific embodiment. Step 104 involves verifying the pixel-level segmentation results based on preset topological continuity rules to generate a multispectral feature association dataset. This specifically includes the following steps:

[0112] Step 401: Obtain the geometric structure type label of each pixel in the adjacent pixel pair from the pixel-level segmentation result, and generate a segmentation label combination by combining the geometric structure type label;

[0113] In this step, the geometric structure type label refers to the geometric category identifier of each pixel in the pixel-level segmentation result, including planar category labels (continuous surfaces), edge category labels (structural boundaries), and hole category labels (broken or missing regions). Segmentation label combination refers to the pairing of geometric structure type labels of two pixels in adjacent pixel pairs, such as planar-hole category, edge-planar category.

[0114] In this embodiment of the invention, the geometric structure type label of each pixel is first extracted from the pixel-level segmentation result through morphological operations or image segmentation algorithms; all adjacent pixel pairs are traversed, and the geometric structure type labels of the pixels are recorded and combined in sequence (such as plane class - hole class) to form a segmentation label combination for subsequent topology rule verification.

[0115] Step 402: Determine whether the combination of segmentation labels meets the geometric continuity requirements according to the preset topological continuity rules. The preset topological continuity rules include allowing planar labels to be adjacent to edge labels and prohibiting planar labels from being adjacent to hole labels.

[0116] In this step, the geometric continuity requirement refers to the rules based on the geometric integrity of the surface of the real-world 3D model, which are used to determine whether the combination of segmentation labels conforms to the rationality of the physical structure.

[0117] In this embodiment of the invention, the compliance of label combinations is determined according to preset topological continuity rules. By matching pairs, if a segmented label combination violates a prohibition rule (prohibiting planes and holes from being adjacent), it is marked as non-compliant; otherwise, it is compliant. For example, the label "building-vegetation-hole" is judged as non-compliant because it violates the rule that planes and holes are prohibited from being adjacent, while "building-road-edge" complies with the rule.

[0118] Step 403: Assign valid identifiers to the segmentation label combinations that meet the geometric continuity requirements, and generate the target segmentation label combinations;

[0119] In this step, a valid identifier refers to a unique identifier that marks a combination of segmentation labels that meet the geometric continuity requirements, and is used to identify normal regions.

[0120] In this embodiment of the invention, the segmentation label combinations that are determined to be compliant are assigned a valid identifier (such as the value 1) to form a target segmentation label combination set. For example, after marking compliant segmentation label combinations, only valid combinations such as building-road edge are retained, while abnormal combinations such as building-vegetation hole are removed.

[0121] Step 404: For segmentation label combinations that are determined to be inconsistent with geometric continuity requirements, assign an abnormal identifier, generate non-target segmentation label combinations, calculate the reflection intensity difference between adjacent pixel pairs corresponding to the non-target segmentation label combinations, and generate a comprehensive difference value by combining the preset channel weight coefficient.

[0122] In this step, the anomaly identifier is a unique identifier that marks a combination of segmentation labels that violates geometric continuity requirements, used to identify the abnormal region to be checked. The reflection intensity difference refers to the absolute difference in reflection intensity values ​​of adjacent pixel pairs across multiple spectral channels, used to quantify the inconsistency of material properties. The preset channel weighting coefficient refers to the pre-defined contribution ratio parameter of each spectral channel in the difference calculation (e.g., near-infrared weight 0.7, visible light 0.3). The comprehensive difference value is a numerical value generated by weighting the reflection intensity differences of each channel, reflecting the intensity of the inconsistency of the abnormal region in the multispectral dimension.

[0123] In this embodiment of the invention, for segmentation label combinations determined to be "non-compliant", an anomaly identifier is assigned to generate non-target segmentation label combinations; the reflectance intensity values ​​of adjacent pixel pairs corresponding to the non-target segmentation label combinations in each channel of the multispectral imaging data are obtained; for each channel, the absolute difference in reflectance intensity values ​​of the two pixels in the adjacent pixel pair is calculated to generate reflectance intensity difference values ​​for each channel; the reflectance intensity difference values ​​for each channel are multiplied by a preset weighting coefficient (e.g., 0.7 for near-infrared and 0.3 for visible light) to generate weighted difference values ​​for each channel; the weighted difference values ​​for all channels are summed to generate a comprehensive difference value to quantify the spectral inconsistency of the abnormal region.

[0124] Step 405: The target segmentation label combination, the non-target segmentation label combination, the comprehensive difference value, the position coordinates of the adjacent pixel pairs corresponding to the segmentation label combination, and the preset channel weight coefficient are associated and stored to generate a multispectral feature association dataset containing a verification identifier, wherein the verification identifier includes a valid identifier and an abnormal identifier.

[0125] In this step, the verification identifier refers to the unique identifier of the combination type of segmentation label, including valid identifiers and abnormal identifiers.

[0126] In this embodiment of the invention, all data is stored as a structured dataset according to their association relationships. For example, each pixel pair record includes a building-road edge (valid identifier), a building-vegetation hole (abnormal identifier), a comprehensive difference value of 0.13, pixel location coordinates (100, 50) and (101, 50), and a preset channel weight coefficient, which are finally integrated into a multispectral feature association dataset for subsequent model optimization.

[0127] This invention addresses the problems of high false detection rates and poor adaptability to complex scenarios caused by reliance on a single data source in traditional 3D model quality inspection. By quantifying the spectral inconsistencies of abnormal regions through comprehensive difference values ​​and combining valid and abnormal identifiers to achieve full data analysis, it improves detection accuracy and efficiency, while also supporting traceability verification of abnormal regions.

[0128] This invention provides a specific embodiment. Step 105 involves performing a quality check on the real-world 3D model based on the comparison results between the geometric feature segmentation deviations and preset tolerance ranges between different multispectral bands in the multispectral feature association dataset, obtaining a quality check result. The quality check result includes abnormal regions with geometric structural breaks or distortions, specifically including the following steps:

[0129] Step 501: Define a layered difference threshold corresponding to the resolution level and scene type of the real-world 3D model based on a preset tolerance range;

[0130] In this step, resolution level refers to the precision level of the realistic 3D model, such as millimeter level (0.1mm / pixel) or centimeter level (1cm / pixel), which affects the setting of the layer difference threshold. Scene type refers to the application environment classification of the realistic 3D model, including indoor (stable lighting) and outdoor (complex lighting), used to adjust the threshold sensitivity. Layer difference threshold refers to the difference judgment threshold dynamically set according to the resolution level and scene type of the realistic 3D model, for example, high-resolution outdoor scene threshold = 0.6.

[0131] In this embodiment of the invention, the preset tolerance range includes the difference threshold matching rules for different model resolutions (such as millimeter level, centimeter level) and scene types (such as indoor, outdoor). The high-resolution model corresponds to a lower difference threshold (such as 0.5), the low-resolution model corresponds to a higher difference threshold (such as 0.8), and the outdoor scene requires an additional increase in the threshold due to the complexity of the lighting (such as threshold × 1.2).

[0132] Step 502: Compare the comprehensive difference value in the multispectral feature association dataset with the hierarchical difference threshold to filter out target adjacent pixel pairs;

[0133] In this step, the target adjacent pixel pair refers to the adjacent pixel pair whose overall difference value exceeds the hierarchical difference threshold, and is used as a candidate object for the abnormal region to be identified.

[0134] In this embodiment of the invention, the position coordinates of the anomaly region to be confirmed are converted into the coordinates of the corresponding triangular facet vertices in the surface triangular mesh, wherein each position coordinate is associated with the nearest triangular facet vertex through a nearest neighbor matching algorithm; based on the coordinates of the vertex, all triangular faces containing the vertex are extracted as an initial set of faces to be processed; the first triangular facet is selected as the current facet, marked as processed, and added to a temporary set of faces for candidate anomalies; all adjacent triangular faces sharing edges or vertices with the current facet are traversed, and if an adjacent triangular facet is not marked as processed but is included in the initial set of faces to be processed, it is marked as processed and added to the temporary set of faces, and this traversal operation is repeated until no new triangular faces are added; all triangular faces in the temporary set of faces are merged into a candidate anomaly region, and the coverage of the candidate anomaly region is defined by the geometric area of ​​the merged set of triangular faces.

[0135] Step 503: Mark the target adjacent pixel pairs as anomaly regions to be confirmed, and map the position coordinates of the anomaly regions to be confirmed to the surface triangular mesh of the real scene 3D model, so as to perform spatial adjacency traversal operation on the triangular facets corresponding to the anomaly regions to be confirmed, and merge all triangular facets with shared edges and vertices as candidate anomaly regions.

[0136] In this step, the "abnormal region to be confirmed" refers to the set of anomalous pixel pairs initially selected, which requires further spatial merging and region-level judgment. A triangular facet refers to a triangular mesh unit on the surface of a real-world 3D model, consisting of three vertices and three edges, used for geometric structure representation. A candidate anomalous region refers to a continuous anomalous region generated by merging triangular faces sharing common edges or vertices, characterizing the range of potential defects.

[0137] In this embodiment of the invention, the position coordinates of the anomaly region to be confirmed are converted into the corresponding vertex coordinates of the triangular facets in the surface triangular mesh of the real-world 3D model. Each position coordinate is associated with the nearest vertex of the triangular facet through a nearest neighbor matching algorithm. All triangular facets containing the vertices of the triangular facets are extracted as an initial set of facets to be processed. The first triangular facet in this set is selected as the current facet, marked as processed, and added to a temporary facet set of candidate anomaly regions. All adjacent triangular facets sharing edges or vertices with the current facet are traversed. If an adjacent triangular facet is not marked as processed and is included in the initial set of facets to be processed, the adjacent triangular facet is marked as processed and added to the temporary facet set. This traversal operation is repeated until no new triangular facets are added. All triangular facets in the temporary facet set are merged into a candidate anomaly region, the coverage of which is defined by the geometric area of ​​the merged triangular facet set.

[0138] Step 504: Determine the final abnormal region based on the comparison result of the sum of the comprehensive difference values ​​of all target adjacent pixel pairs in the candidate abnormal region and the preset region tolerance value;

[0139] In this step, the total sum of differences refers to the cumulative sum of the differences between all adjacent pixel pairs within the candidate anomaly region, used to quantify the intensity of the regional anomaly. The preset regional tolerance value is the regional threshold for determining whether a candidate anomaly region is a final anomaly. The final anomaly region refers to a candidate anomaly region that meets the regional tolerance condition, identified as a defect region with geometric structural breaks or distortions.

[0140] In this embodiment of the invention, the comprehensive difference value of all adjacent pixel pairs of the target in the candidate abnormal region is accumulated (e.g., 0.7×10=7.0). If the sum exceeds the preset region tolerance value (e.g., 5.0), it is determined to be the final abnormal region; otherwise, it is downgraded to a low-risk region.

[0141] Step 505: The location coordinates, coverage area, segmentation label combination type and total comprehensive difference value of the final abnormal area are associated and stored to generate a quality inspection result. The quality inspection result includes abnormal areas with broken or distorted geometric structures.

[0142] In this step, the segmentation label combination type refers to the label combination form of adjacent pixel pairs in the final abnormal region (such as plane class-hole class), which is used to identify the abnormal type.

[0143] In this embodiment of the invention, the coverage area is defined by the geometric area of ​​the triangular facet set after merging candidate abnormal regions. The segmentation label combination type (such as plane type - hole type) identifies the abnormal type (fracture). The sum of the comprehensive difference values ​​(such as 7.0) reflects the severity of the abnormality. The above information is associated and stored to generate a quality inspection result containing abnormal regions with geometric structural fractures or geometric structural distortions.

[0144] This invention addresses the problem of insufficient accuracy in traditional 3D model quality inspection caused by false detection of isolated pixels and rigid thresholds. By quantifying the severity of anomalies through the sum of comprehensive difference values ​​and clarifying the nature of defects (fracture or distortion) by combining label combination types, it achieves interpretable quality inspection results output.

[0145] This invention provides a specific embodiment. Step 102 involves optimizing the multispectral imaging data to generate standardized input data, specifically including the following steps:

[0146] Step 201: The data of each multispectral band in the multispectral imaging data is converted to a preset frequency domain space through a dynamic optimization processing node to obtain a target frequency domain space. Based on the frequency distribution characteristics of the target frequency domain space, noise suppression processing is performed on the multispectral imaging data to generate the first target imaging data. The computing resource allocation strategy of the dynamic optimization processing node is dynamically adjusted according to the amount of multispectral imaging data and the average scene illumination intensity.

[0147] In this step, the dynamic optimization processing node refers to a processing module that dynamically allocates computing resources based on the amount of multispectral imaging data and scene illumination, used to perform operations such as frequency domain noise suppression. The preset frequency domain space refers to a predefined frequency domain coordinate system (such as the complex space after Fourier transform) used to analyze image frequency components. The target frequency domain space refers to the frequency domain space where the multispectral data resides after transformation, containing frequency distribution information. Frequency distribution characteristics refer to the energy distribution of different frequency components in the target frequency domain space; high-frequency components correspond to noise, and low-frequency components correspond to effective texture. The first target imaging data refers to the multispectral data after frequency domain noise suppression processing, preserving low-frequency texture features.

[0148] In this embodiment of the invention, the dynamic optimization processing node dynamically allocates computing resources based on the amount of multispectral data and the average illumination intensity of the scene (e.g., prioritizing GPU resources when there is a large amount of data). The image data of each band is converted to a preset frequency domain space (e.g., frequency domain coordinate system) through a frequency domain transformation algorithm (e.g., fast Fourier transform). Based on the frequency distribution characteristics (e.g., high-frequency components correspond to noise and low-frequency components correspond to texture), high-frequency noise components (e.g., sensor noise or stripe noise) are filtered out, while low-frequency texture features (e.g., building surface details) are preserved, generating the first target imaging data after noise suppression.

[0149] Step 202: Adjust the pixel brightness values ​​of the first target imaging data according to the illumination intensity distribution of a local area in the first target imaging data to generate the second target imaging data;

[0150] In this step, the local region refers to the neighborhood defined by a preset window size (such as 5×5 or 3×3 pixels) centered on the current pixel in the multispectral imaging data. It is used to statistically analyze the illumination intensity distribution characteristics of pixels within this range, such as the mean and variance of brightness, to drive the adaptive illumination equalization operation. The window size of the local region is dynamically set according to the image resolution and scene complexity; a smaller window is used for high-resolution images to preserve details, while a larger window is used for low-resolution images to improve statistical stability. Illumination intensity distribution refers to the statistical characteristics of brightness values ​​(such as local mean and variance) in different regions of the image.

[0151] In this embodiment of the invention, for each band of the first target imaging data, the illumination intensity distribution (e.g., mean and variance of brightness) of a local region (e.g., a 5×5 pixel window) is statistically analyzed. For highlight areas (brightness values ​​higher than a preset threshold), the pixel brightness value is reduced using a compression coefficient (e.g., 0.8), and for shadow areas (brightness values ​​lower than a preset threshold), the pixel brightness value is increased using an enhancement coefficient (e.g., 1.5), generating second target imaging data with equalized illumination. Pixel brightness value refers to the optical reflection or radiation intensity value of a single pixel in a specific multispectral band. The second target imaging data refers to the multispectral image data after illumination equalization processing, eliminating interference from highlights and shadows.

[0152] Step 203: Perform spatial normalization on the second target imaging data to generate third target imaging data;

[0153] In this step, the third target imaging data refers to the spatially aligned and cropped multispectral image data with consistent pixel coordinates.

[0154] In this embodiment of the invention, the multi-band images in the second target imaging data are uniformly cropped according to a preset size (e.g., 512×512 pixels), and the images of different multispectral bands are aligned to the same spatial coordinate system through affine transformation (e.g., translation, rotation, scaling), ensuring that the pixel coordinates of the same ground object locations in all multispectral band images are completely consistent, thereby generating spatially aligned third target imaging data.

[0155] Step 204: Based on a preset numerical range, linearly scale the pixel values ​​of each multispectral band in the third target imaging data to generate standardized input data adapted to the deep learning network.

[0156] In this step, the preset numerical range refers to the range of input data required by the deep learning network (such as [0,1] or [-1,1]). The pixel value refers to the numerical value of a single pixel in the image, characterizing its optical properties (such as reflectance intensity).

[0157] In this embodiment of the invention, for each multispectral band of the third target imaging data, the global maximum and minimum values ​​of its pixel values ​​are calculated, and the pixel values ​​are linearly mapped to a preset value range (such as the [0,1] interval) according to the formula (scaled value = (original value - minimum value) / (maximum value - minimum value)) to generate standardized input data.

[0158] This invention eliminates sensor noise through frequency domain noise suppression, resolves complex lighting interference through adaptive illumination equalization, ensures multi-band alignment through spatial standardization, and adapts numerical normalization to deep learning input, generating highly consistent, low-noise standardized input data. This improves the accuracy and robustness of subsequent 3D modeling and quality checks, and is particularly suitable for scenes with interlaced shadows and high dynamic range.

[0159] Figure 2 This invention provides a schematic diagram of a real-world 3D model quality inspection system that integrates deep learning, as shown in the embodiment of the invention. Figure 2 As shown, the system includes:

[0160] The acquisition module 21 is used to acquire multispectral imaging data of the target scene and construct a real-scene 3D model based on the multispectral imaging data.

[0161] Optimization module 22 is used to optimize the multispectral imaging data and generate standardized input data;

[0162] Extraction module 23 is used to extract the correlation features between multispectral bands based on the standardized input data using a deep learning-driven semantic segmentation network, and generate pixel-level segmentation results by combining the triangular mesh connection relationship of the real-scene 3D model.

[0163] Verification module 24 is used to verify the pixel-level segmentation results based on preset topological continuity rules and generate a multispectral feature association dataset.

[0164] The inspection module 25 is used to perform quality inspection on the real-scene 3D model based on the comparison results of the geometric feature segmentation deviation between different multispectral bands in the multispectral feature association dataset and the preset tolerance range, and to obtain the quality inspection results, which include abnormal areas of geometric structure breakage or geometric structure distortion.

[0165] Figure 2 The aforementioned quality inspection system for real-world 3D models integrating deep learning can perform... Figure 1 The implementation principle and technical effects of the deep learning-integrated real-scene 3D model quality inspection method described in the illustrated embodiment will not be repeated here. The specific methods of operation of each module and unit in the deep learning-integrated real-scene 3D model quality inspection system described above have been detailed in the embodiments related to this method, and will not be elaborated upon here.

[0166] In one possible design, Figure 2 The illustrated embodiment of a real-world 3D model quality inspection system integrating deep learning can be implemented as a computing device, such as... Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0167] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component 32.

[0168] The processing component 32 is used for: acquiring multispectral imaging data of the target scene; constructing a real-scene 3D model based on the multispectral imaging data; optimizing the multispectral imaging data to generate standardized input data; extracting correlation features between multispectral bands using a deep learning-driven semantic segmentation network based on the standardized input data, and generating pixel-level segmentation results by combining the triangular mesh connection relationship of the real-scene 3D model; verifying the pixel-level segmentation results based on preset topological continuity rules to generate a multispectral feature association dataset; and performing a quality check on the real-scene 3D model based on the comparison results of geometric feature segmentation deviations between different multispectral bands in the multispectral feature association dataset with preset tolerance ranges, obtaining a quality check result, wherein the quality check result includes abnormal regions with geometric structural breaks or geometric structural distortions.

[0169] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0170] Storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0171] Of course, computing devices may also include other components, such as input / output interfaces, display components, communication components, etc.

[0172] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc.

[0173] The communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0174] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform. In this case, the computing device can refer to a cloud server, and the aforementioned processing components, storage components, etc., can be basic server resources rented or purchased from the cloud computing platform.

[0175] This invention also provides a computer storage medium storing a computer program, which, when executed by a computer, can perform the above-described functions. Figure 1 The illustrated embodiment presents a method for quality inspection of real-world 3D models that incorporates deep learning.

[0176] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0177] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for quality inspection of real-world 3D models integrating deep learning, characterized in that, include: Acquire multispectral imaging data of the target scene, and construct a real-world 3D model based on the multispectral imaging data; The multispectral imaging data is optimized to generate standardized input data; Based on the standardized input data, a deep learning-driven semantic segmentation network is used to extract the correlation features between multispectral bands, and combined with the triangular mesh connection relationship of the real-scene 3D model, pixel-level segmentation results are generated. Based on preset topological continuity rules, the pixel-level segmentation results are verified to generate a multispectral feature association dataset. Based on the comparison results of the geometric feature segmentation deviation between different multispectral bands in the multispectral feature association dataset and the preset tolerance range, the real-scene 3D model is subjected to quality inspection to obtain quality inspection results. The quality inspection results include abnormal areas with geometric structure fractures or geometric structure distortions.

2. The method according to claim 1, characterized in that, Based on the standardized input data, a deep learning-driven semantic segmentation network is used to extract the correlation features between multispectral bands. Combined with the triangular mesh connection relationship of the real-world 3D model, pixel-level segmentation results are generated, including: The standardized input data is fed into a deep learning-driven geometrically constrained semantic segmentation network to extract the correlation features between multispectral bands; Obtain the target-level feature map output by the geometrically constrained semantic segmentation network, and perform upsampling and feature concatenation on the target-level feature map to generate a fused feature map; Based on the triangular mesh connection relationship of the real scene 3D model, the weighted relationship between the feature similarity and segmentation prediction difference of adjacent pixel pairs in the fused feature map is calculated to generate a constraint weight matrix. The segmentation prediction probability distribution of the fused feature map is adjusted according to the constraint weight matrix so that the segmentation prediction values ​​of adjacent pixels satisfy the geometric continuity constraint, resulting in the adjusted fused feature map. A pixel-by-pixel classification operation is performed on the adjusted fused feature map to generate pixel-level segmentation results.

3. The method according to claim 2, characterized in that, Based on the triangular mesh connection relationship of the real-scene 3D model, the weighted relationship between the feature similarity and segmentation prediction difference of adjacent pixel pairs in the fused feature map is calculated to generate a constraint weight matrix, including: Based on the connection relationship of the surface triangular mesh of the real-world 3D model, the adjacent pixel pairs corresponding to the edges of each triangular facet in the fused feature map are determined. Extract the feature vector of each pixel in the multispectral channels of adjacent pixel pairs in the fused feature map to calculate the feature similarity; Extract the segmentation prediction probability distribution of each pixel in the adjacent pixel pairs in the fused feature map to calculate the segmentation prediction difference. Based on the preset weighting coefficients, the feature similarity and the segmentation prediction difference are normalized and linearly superimposed to generate the weighted constraint value corresponding to each pair of adjacent pixels. The weighted constraint values ​​are mapped to the fused feature map to construct a constraint weight matrix with the same resolution as the fused feature map.

4. The method according to claim 2, characterized in that, The segmentation prediction probability distribution of the fused feature map is adjusted according to the constraint weight matrix to ensure that the segmentation prediction values ​​of adjacent pixels satisfy the geometric continuity constraint, resulting in an adjusted fused feature map, including: The adjustment coefficient for each pixel is calculated based on the constraint strength values ​​of adjacent pixel pairs in the constraint weight matrix. The segmentation prediction probability distribution of each pixel in the adjacent pixel pair in the fused feature map is adjusted using the adjustment coefficient to obtain the superimposed segmentation prediction probability distribution; The superimposed segmentation prediction probability distribution is normalized to generate an updated segmentation prediction probability distribution. The updated segmentation prediction probability distribution replaces the segmentation prediction probability distribution of the corresponding pixel in the fused feature map to generate an adjusted fused feature map, so that the segmentation prediction values ​​of adjacent pixels satisfy the geometric continuity constraint.

5. The method according to claim 1, characterized in that, Based on preset topological continuity rules, the pixel-level segmentation results are verified to generate a multispectral feature association dataset, including: Obtain the geometric structure type label of each pixel in the adjacent pixel pair from the pixel-level segmentation result, and generate a segmentation label combination by combining the geometric structure type label; According to preset topological continuity rules, it is determined whether the combination of segmentation labels meets the geometric continuity requirements. The preset topological continuity rules include allowing planar labels to be adjacent to edge labels and prohibiting planar labels from being adjacent to hole labels. For segmentation label combinations that are determined to meet the geometric continuity requirements, assign valid identifiers and generate target segmentation label combinations; For segmentation label combinations that are determined to be inconsistent with geometric continuity requirements, an abnormal identifier is assigned, a non-target segmentation label combination is generated, and the reflection intensity difference of adjacent pixel pairs corresponding to the non-target segmentation label combination is calculated. Combined with a preset channel weight coefficient, a comprehensive difference value is generated. The target segmentation label combination, the non-target segmentation label combination, the comprehensive difference value, the position coordinates of the adjacent pixel pairs corresponding to the segmentation label combination, and the preset channel weight coefficients are associated and stored to generate a multispectral feature association dataset containing verification identifiers, including valid identifiers and abnormal identifiers.

6. The method according to claim 1, characterized in that, Based on the comparison results of the geometric feature segmentation deviation between different multispectral bands in the multispectral feature association dataset and the preset tolerance range, the real-scene 3D model is subjected to quality inspection to obtain quality inspection results. The quality inspection results include abnormal areas with geometric structural breaks or distortions, including: The layer difference threshold is defined based on the preset tolerance range and the resolution level and scene type of the real-world 3D model. The comprehensive difference value in the multispectral feature association dataset is compared with the hierarchical difference threshold to filter out target adjacent pixel pairs; The target adjacent pixel pairs are marked as anomaly regions to be confirmed, and the position coordinates of the anomaly regions to be confirmed are mapped to the surface triangular mesh of the real scene 3D model, so as to perform spatial adjacency traversal operation on the triangular facets corresponding to the anomaly regions to be confirmed, and merge all triangular facets with shared edges and vertices as candidate anomaly regions. The final abnormal region is determined by comparing the sum of the comprehensive difference values ​​of all adjacent pixel pairs within the candidate abnormal region with the preset region tolerance value. The location coordinates, coverage area, segmentation label combination type, and total difference value of the final abnormal region are associated and stored to generate a quality inspection result. The quality inspection result includes abnormal regions with geometric structural breaks or geometric structural distortions.

7. The method according to claim 1, characterized in that, The multispectral imaging data is optimized to generate standardized input data, including: The data of each multispectral band in the multispectral imaging data is converted to a preset frequency domain space by a dynamic optimization processing node to obtain a target frequency domain space. Based on the frequency distribution characteristics of the target frequency domain space, noise suppression processing is performed on the multispectral imaging data to generate the first target imaging data. The computing resource allocation strategy of the dynamic optimization processing node is dynamically adjusted according to the amount of multispectral imaging data and the average illumination intensity of the scene. Based on the illumination intensity distribution of a local area in the first target imaging data, the pixel brightness values ​​of the first target imaging data are adjusted to generate the second target imaging data. Spatial normalization is performed on the second target imaging data to generate the third target imaging data; Based on a preset numerical range, the pixel values ​​of each multispectral band in the imaging data of the third target are linearly scaled to generate standardized input data adapted to the deep learning network.

8. A quality inspection system for real-world 3D models integrating deep learning, characterized in that, include: The acquisition module is used to acquire multispectral imaging data of the target scene and construct a real-scene 3D model based on the multispectral imaging data. The optimization module is used to optimize the multispectral imaging data and generate standardized input data. The extraction module is used to extract the correlation features between multispectral bands based on the standardized input data using a deep learning-driven semantic segmentation network, and combine the triangular mesh connection relationship of the real-scene 3D model to generate pixel-level segmentation results. The verification module is used to verify the pixel-level segmentation results based on preset topological continuity rules and generate a multispectral feature association dataset. The inspection module is used to perform quality checks on the real-world 3D model based on the comparison results of the geometric feature segmentation deviation between different multispectral bands in the multispectral feature association dataset with a preset tolerance range, and to obtain quality inspection results. The quality inspection results include abnormal areas with geometric structure fractures or geometric structure distortions.

9. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the real-scene 3D model quality inspection method integrating deep learning as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The system contains a computer program that, when executed by a computer, implements a method for quality inspection of real-world 3D models that integrates deep learning, as described in any one of claims 1 to 7.

Images