Real scene three-dimensional model quality inspection method and system fused with deep learning

Abstract

The invention provides a live-action three-dimensional model quality inspection method and system fused with deep learning, and the method comprises the steps: building a live-action three-dimensional model through obtaining multispectral imaging data of a target scene; multispectral imaging data is optimized, standardized input data is generated, on the basis of the standardized input data, correlation characteristics among multispectral bands are extracted through a semantic segmentation network driven by deep learning, a pixel-level segmentation result is generated in combination with geometric structure characteristics of a live-action three-dimensional model, verification is carried out based on a preset topological continuity rule, and the segmentation result is obtained. Generating a multispectral feature associated data set, and performing quality inspection on the live-action three-dimensional model according to a comparison result of geometric feature segmentation deviations among different wavebands and a preset tolerance range to obtain a quality inspection result; according to the method, the refined quality inspection of the live-action three-dimensional model is realized, the abnormal region with the fractured or distorted geometric structure is effectively identified and positioned, and the geometric consistency and the overall quality of the live-action three-dimensional model under the complex illumination condition are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional model detection, and particularly to a method and system for inspecting the quality of real-scene three-dimensional models integrating deep learning. Background Art

[0002] The geometric consistency inspection of real-scene three-dimensional models in complex lighting environments, especially in scenes with interlaced shadows and high dynamic ranges, is crucial for ensuring the accuracy and reliability of the models. Such application scenarios usually involve fields such as urban planning, virtual reality, and cultural heritage protection, where high-quality three-dimensional models are the foundation. Due to the changes in natural light and the influence of shadows caused by buildings or other objects, the acquired image data often has high contrast and complex lighting conditions, which poses challenges to three-dimensional reconstruction algorithms. In order to ensure that the generated three-dimensional models are not only visually realistic but also geometrically accurate, a technology capable of effectively processing data under these complex lighting conditions is needed.

[0003] Currently, the mainstream solution for this demand is a method that combines a deep learning segmentation model driven by single-spectral image data with point cloud registration verification. This solution extracts geometric features from single-spectral image data through a pre-trained semantic segmentation network. After generating a preliminary segmentation result, point cloud registration technology is used to compare the geometric consistency between the three-dimensional model and the segmentation result, and the deviation areas are marked as anomalies. There are some defects in the above technical solutions. For example, relying on single-spectral data, it is sensitive to lighting interferences such as shadows and highlights, resulting in low feature extraction ability of the segmentation model in dark areas or overexposed areas and an increase in the false detection rate; point cloud registration verification cannot effectively distinguish real structural breaks from texture differences caused by lighting, especially in the multi-scale curved surface junctions, where missed detections are likely to occur. Summary of the Invention

[0004] The present invention provides a method and system for inspecting the quality of real-scene three-dimensional models integrating deep learning to solve the problems in the prior art such as low feature extraction ability in dark areas or overexposed areas, increased false detection rate; inability to effectively distinguish real structural breaks from texture differences caused by lighting, especially in the multi-scale curved surface junctions, where missed detections are likely to occur.

[0005] In a first aspect, the present invention provides a method for inspecting the quality of real-scene three-dimensional models integrating deep learning, including: Obtaining multi-spectral imaging data of a target scene, and constructing a real-scene three-dimensional model based on the multi-spectral imaging data; Performing optimization processing on the multi-spectral imaging data to generate standardized input data; Based on the standardized input data, using a semantic segmentation network driven by deep learning to extract the correlation features between multi-spectral bands, and combining the triangular mesh connection relationship of the real-scene three-dimensional model to generate a pixel-level segmentation result; Based on the preset topological continuity rules, verify the pixel-level segmentation result to generate a multi-spectral feature association dataset; According to the comparison result between the geometric feature segmentation deviation between different multi-spectral bands in the multi-spectral feature association dataset and the preset tolerance range, perform quality inspection on the real-scene three-dimensional model to obtain a quality inspection result, where the quality inspection result includes abnormal areas of geometric structure fracture or geometric structure distortion.

[0006] Optionally, based on the standardized input data, use a deep learning-driven semantic segmentation network to extract the association features between multi-spectral bands, and combine with the triangular mesh connection relationship of the real-scene three-dimensional model to generate a pixel-level segmentation result, including: Input the standardized input data into a deep learning-driven geometric constraint semantic segmentation network to extract the association features between multi-spectral bands; Obtain the target hierarchical feature map output by the geometric constraint semantic segmentation network, perform an upsampling operation and a feature splicing process on the target hierarchical feature map to generate a fusion feature map; Based on the triangular mesh connection relationship of the real-scene three-dimensional model, calculate the weighted relationship between the feature similarity and the segmentation prediction difference of adjacent pixel pairs in the fusion feature map to generate a constraint weight matrix; Adjust the segmentation prediction probability distribution of the fusion feature map according to the constraint weight matrix to make the segmentation prediction values of adjacent pixels satisfy the geometric continuity constraint, and obtain the adjusted fusion feature map; Perform a per-pixel classification operation on the adjusted fusion feature map to generate a pixel-level segmentation result.

[0007] Optionally, based on the triangular mesh connection relationship of the real-scene three-dimensional model, calculate the weighted relationship between the feature similarity and the segmentation prediction difference of adjacent pixel pairs in the fusion feature map to generate a constraint weight matrix, including: Based on the connection relationship of the surface triangular mesh of the real-scene three-dimensional model, determine the adjacent pixel pairs corresponding to each triangular patch edge in the fusion feature map; Extract the feature vectors of each pixel in the multi-spectral channels of the adjacent pixel pairs in the fusion feature map to calculate the feature similarity; Extract the segmentation prediction probability distribution of each pixel in the adjacent pixel pairs in the fusion feature map to calculate the segmentation prediction difference; According to the preset weighting coefficient, perform normalized linear superposition on the feature similarity and the segmentation prediction difference to generate a weighted constraint value corresponding to each adjacent pixel pair; Map the weighted constraint value to the fusion feature map to construct a constraint weight matrix consistent with the resolution of the fusion feature map.

[0008] Optionally, adjust the segmentation prediction probability distribution of the fused feature map according to the constraint weight matrix to make the segmentation prediction values of adjacent pixels satisfy the geometric continuity constraint, and obtain an adjusted fused feature map, including: Calculate the adjustment coefficient of each pixel according to the constraint strength value of the adjacent pixel pair in the constraint weight matrix; Use the adjustment coefficient to adjust the segmentation prediction probability distribution of each pixel in the adjacent pixel pair in the fused feature map to obtain a superimposed segmentation prediction probability distribution; Normalize the superimposed segmentation prediction probability distribution to generate an updated segmentation prediction probability distribution; Replace the segmentation prediction probability distribution of the corresponding pixel in the fused feature map with the updated segmentation prediction probability distribution to generate an adjusted fused feature map, so that the segmentation prediction values of adjacent pixels satisfy the geometric continuity constraint.

[0009] In a second aspect, the present invention provides a real-scene three-dimensional model quality inspection system integrating deep learning, including: An acquisition module, configured to acquire multi-spectral imaging data of a target scene, and construct a real-scene three-dimensional model based on the multi-spectral imaging data; An optimization module, configured to perform optimization processing on the multi-spectral imaging data to generate standardized input data; An extraction module, configured to extract the correlation features between multi-spectral bands based on the standardized input data by using a deep learning-driven semantic segmentation network, and combine the triangular mesh connection relationship of the real-scene three-dimensional model to generate a pixel-level segmentation result; A verification module, configured to verify the pixel-level segmentation result based on a preset topological continuity rule to generate a multi-spectral feature correlation data set; An inspection module, configured to perform quality inspection on the real-scene three-dimensional model according to the comparison result between the geometric feature segmentation deviation between different multi-spectral bands in the multi-spectral feature correlation data set and a preset tolerance range, and obtain a quality inspection result, where the quality inspection result includes abnormal regions with geometric structure fractures or geometric structure distortions.

[0010] In a third aspect, the present invention provides a computing device, including a processor and a memory, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the method for inspecting the quality of a real-scene three-dimensional model integrating deep learning according to any one of the first aspects.

[0011] In a fourth aspect, the present invention provides a computer storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method for inspecting the quality of a real-scene three-dimensional model integrating deep learning according to any one of the first aspects is implemented.

[0012] In the present invention, multispectral imaging data of a target scene is acquired, and based on the multispectral imaging data, a real-scene three-dimensional model is constructed; the multispectral imaging data is optimized to generate standardized input data; based on the standardized input data, a deep learning-driven semantic segmentation network is used to extract the correlation features between multispectral bands, and combined with the triangular mesh connection relationship of the real-scene three-dimensional model, a pixel-level segmentation result is generated; based on a preset topological continuity rule, the pixel-level segmentation result is verified to generate a multispectral feature correlation dataset; according to the comparison result between the geometric feature segmentation deviation between different multispectral bands in the multispectral feature correlation dataset and a preset tolerance range, a quality inspection of the real-scene three-dimensional model is performed to obtain a quality inspection result, and the quality inspection result includes abnormal regions with geometric structure breaks or geometric structure distortions. The technical solution provided by the present invention captures the complete reflection characteristics of the scene in a complex lighting environment through multispectral imaging data (including visible and non-visible light bands), makes up for the texture loss problem of single-spectrum data in shadow and highlight areas, provides high-fidelity input for real-scene three-dimensional model construction, and enhances the geometric detail restoration ability; through frequency-domain noise suppression and adaptive illumination equalization operations, sensor noise and illumination unevenness interference are eliminated, and standardized data with consistent illumination and controllable noise is generated, solving the problem of unstable model feature extraction caused by data quality fluctuations in traditional methods; a deep learning network is used to extract the correlation features between multispectral bands, and combined with the geometric structure information of the three-dimensional model, high-precision segmentation of pixel-level geometric categories (such as planes, edges, holes) is achieved, breaking through the missegmentation bottleneck of traditional single-spectrum models in complex geometric regions; the segmentation result is verified by a preset geometric topology rule (such as planes and holes are not allowed to be adjacent), and combined with the quantitative analysis of the feature differences between multispectral bands, potential abnormal regions are located, solving the problem of false detection / missing detection caused by pure data-driven models ignoring physical structure rules; through the dynamic matching of the difference value in the multispectral feature correlation dataset and the tolerance range, real geometric anomalies and illumination interference are distinguished, and a structured quality inspection result is output, realizing the objectivity and interpretability of the quality inspection result in a complex lighting scene.Input the standardized input data into the semantic segmentation network driven by a deep learning network to extract multi-spectral correlation features and generate target-level feature maps; fuse multi-scale information through upsampling and shallow feature splicing to construct a fused feature map; based on the triangular mesh connection relationship of the real scene 3D model, quantify the feature similarity and segmentation difference degree of adjacent pixel pairs to generate a constraint weight matrix; use this matrix to dynamically adjust the segmentation prediction probability distribution to force the predicted values of adjacent pixels to satisfy geometric continuity, and finally output a high-consistency pixel-level segmentation result through per-pixel classification; through the dynamic embedding of multi-spectral feature fusion and geometric topology constraints, the detection robustness of the segmentation model for geometric structure fractures and distortion regions in complex illumination scenarios is improved, effectively solving the problem of high false detection rate caused by the limitations of single-spectral data and the lack of rule constraints in traditional methods, and at the same time ensuring that the pixel-level segmentation result is strictly consistent with the surface topology rules of the real scene 3D model.

[0013] These aspects or other aspects of the present invention will be more clearly understood in the following description of the embodiments. Brief Description of the Drawings

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or in the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0015] Figure 1 It is a flowchart of a method for inspecting the quality of a real scene 3D model by fusing deep learning provided by an embodiment of the present invention; Figure 2 It is a schematic structural diagram of a system for inspecting the quality of a real scene 3D model by fusing deep learning provided by an embodiment of the present invention; Figure 3 It is a schematic structural diagram of a computing device provided by an embodiment of the present invention. Detailed Embodiments

[0016] In order to enable those skilled in the art of this technology to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention.

[0017] In some of the processes described in the specification, claims, and above-mentioned drawings of the present invention, a plurality of operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear herein or may be executed in parallel. The operation numbers such as 101, 102, etc. are only used to distinguish different operations, and the numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent a sequence, and do not limit that "first" and "second" are different types.

[0018] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0019] Figure 1 The present invention provides a flowchart of a method for inspecting the quality of a real-scene three-dimensional model integrating deep learning. As Figure 1 shown, the method includes: Aiming at the problems of insufficient segmentation accuracy caused by the sensitivity of the single-spectral image data-driven method to shadows and highlights and the single point cloud registration verification rule in complex lighting environments, the present invention proposes a collaborative verification mechanism that integrates multi-spectral data and geometric topology rules. By introducing multi-spectral imaging data (such as visible light, near-infrared, and thermal-infrared bands), using the complementarity of different bands to lighting interference (such as near-infrared penetrating shadows and thermal-infrared weakening highlights), and combining frequency-domain noise suppression and dynamic lighting equalization preprocessing, standardized input data resistant to interference is generated; further, a geometric constraint semantic segmentation network is designed to embed the triangular mesh topology relationship of the three-dimensional model into the deep learning feature extraction process. Through the fusion of the associated features between multi-spectral bands and geometric structure features, the pixel-level segmentation accuracy in complex lighting areas is improved; finally, a multi-spectral feature association dataset is constructed, and based on the double verification of the geometric feature segmentation deviation and topological continuity rules between bands, the contradiction of abnormal areas is quantified (such as the significant reflection difference of the crack area in the visible light and near-infrared bands and the violation of the adjacent rule of plane-holes), so as to accurately distinguish real geometric defects from lighting artifacts, solve the pain points of high false detection rate and multi-scale missed detection of traditional methods, and realize highly robust inspection of the geometric consistency of three-dimensional models in complex lighting scenarios. Based on this, the present invention provides a method for inspecting the quality of a real-scene three-dimensional model integrating deep learning. As Figure 1 shown, it includes: Step 101: Obtain the hyperspectral imaging data of the target scene, and construct a real-scene three-dimensional model based on the hyperspectral imaging data; In this step, the target scene requires a real physical environment for 3D modeling and quality inspection, such as building facades, terrain surfaces, etc. Hyperspectral imaging data refers to a set of images that includes visible light and at least one non-visible light band (such as infrared). Each hyperspectral band records the reflection or radiation characteristics of the scene at different wavelengths. The real-scene three-dimensional model refers to a three-dimensional digital model reconstructed based on hyperspectral data, which represents the geometric structure and surface texture of the target scene.

[0020] In the embodiments of the present invention, visible light and non-visible light band (such as near-infrared, thermal infrared) data of the target scene are collected by a hyperspectral imaging device (such as a hyperspectral camera or lidar) to generate hyperspectral imaging data containing multi-band texture information; based on this data, a structure-from-motion algorithm or a laser point cloud registration technique is used to calculate the three-dimensional point cloud coordinates of the scene, and a real-scene three-dimensional model with texture mapping is generated through a surface reconstruction algorithm (such as Poisson reconstruction). This model represents the geometric structure and surface attributes of the target scene and provides a spatial reference for subsequent quality inspection.

[0021] Step 102: Optimize the hyperspectral imaging data to generate standardized input data; In this step, the standardized input data refers to the hyperspectral image after noise suppression, illumination equalization, and normalization processing, and its size and numerical range are adapted to the input requirements of the deep learning model.

[0022] In the embodiments of the present invention, frequency-domain noise suppression (such as filtering high-frequency noise by fast Fourier transform) and adaptive illumination equalization (such as local histogram equalization) are performed on the hyperspectral imaging data to eliminate shadow and highlight interference and obtain the second target imaging data; subsequently, the multi-band images in the second target imaging data are cropped to a preset size and normalized to a unified numerical range (such as [0,1]) to generate standardized input data adapted to the deep learning network. This process ensures processing efficiency through a dynamic resource allocation strategy (such as edge computing nodes adjusting computing power according to the data volume).

[0023] Step 103: Based on the standardized input data, use a deep learning-driven semantic segmentation network to extract the correlation features between hyperspectral bands, and combine the triangular mesh connection relationship of the real-scene three-dimensional model to generate a pixel-level segmentation result; In this step, the associated feature refers to the joint representation between multi-spectral bands extracted by a semantic segmentation network driven by a deep learning network, such as the correlation between near-infrared and visible light reflection characteristics. The pixel-level segmentation result refers to the pixel classification result output by the deep learning network, and each pixel is labeled as a geometric structure type such as a plane class, an edge class, or a hole class. The triangular mesh connection relationship refers to the edge connection topological structure between triangular patches on the surface of the real scene three-dimensional model, including the three edges of each triangular patch and the vertex information connected to them.

[0024] In an embodiment of the present invention, the standardized input data is input into a semantic segmentation network driven by deep learning. The encoder extracts the associated features between multi-spectral bands (such as the association of reflection characteristics between near-infrared and visible light), and the decoder combines the triangular mesh connection relationship (such as edge adjacency information) of the real scene three-dimensional model to output the pixel-level segmentation result, where the label of each pixel identifies its geometric structure type (such as plane class, edge class, hole class).

[0025] Step 104: Based on a preset topological continuity rule, verify the pixel-level segmentation result to generate a multi-spectral feature association dataset; In this step, the preset topological continuity rule refers to the predefined geometric structure adjacent relationship constraint. For example, it is allowed for the plane class to be adjacent to the edge class, and it is prohibited for the plane class to be adjacent to the hole class.

[0026] In an embodiment of the present invention, based on obtaining the geometric structure type labels of each pixel in the adjacent pixel pairs from the pixel-level segmentation result, a segmentation label combination is generated, and it is verified whether the segmentation label combination conforms to the preset topological continuity rule (such as it is prohibited for the plane class to be adjacent to the hole class). For the segmentation label combination that violates the rule, calculate the reflection intensity difference of its corresponding adjacent pixel pair in each multi-spectral band (such as the sum of squared differences in the near-infrared band), and generate a comprehensive difference value by weighting. Associate and store all the segmentation label combinations, the position coordinates of their corresponding adjacent pixel pairs, the comprehensive difference value, and the preset channel weight coefficient as the multi-spectral feature association dataset.

[0027] Step 105: According to the comparison result between the geometric feature segmentation deviation between different multi-spectral bands in the multi-spectral feature association dataset and the preset tolerance range, perform a quality inspection on the real scene three-dimensional model to obtain a quality inspection result, where the quality inspection result includes abnormal areas with geometric structure breaks or geometric structure distortions; In this step, the geometric feature segmentation deviation refers to the inconsistency of segmentation labels at the same position in different bands, reflecting geometric structure contradictions. The preset tolerance range refers to a set of abnormal determination thresholds preset according to the scene type, which is used to dynamically adapt the abnormal determination criteria. The comparison result refers to the comparison result (exceeding or not exceeding) of the total sum of the comprehensive difference values and the regional tolerance value, which is used to trigger abnormal marking. The quality inspection result refers to a detection report containing the position, type, and severity of the geometric structure fracture or distortion area.

[0028] In the embodiment of the present invention, a hierarchical difference threshold is defined according to the resolution level and scene type of the real scene three-dimensional model. The comprehensive difference value in the multi-spectral feature dataset is compared with this threshold to screen target adjacent pixel pairs, mark these target pixel pairs as areas to be confirmed as abnormal, and map their coordinates to the surface of the three-dimensional model for spatial adjacency traversal. Triangular patches sharing edges and vertices are merged to form candidate abnormal areas; the final abnormal areas, including geometric structure fractures or distortions, are determined by comparing the total sum of the comprehensive difference values of all adjacent pixel pairs within the candidate areas with the preset tolerance values. Finally, the position, range, label type, and total sum of the difference values of the final abnormal areas are associated and stored to generate a quality inspection result. This process effectively identifies and records the abnormal areas in the three-dimensional model.

[0029] For example, taking the detection of cracks on the ancient building wall as an example, first, visible light (texture) and near-infrared (material) data of the ancient building wall are collected by a multi-spectral camera to construct a real scene three-dimensional model; high-frequency noise in the infrared image of the near multi-spectral imaging data is filtered out, and the brightness of the shadow area of the visible light image is equalized to generate standardized input data; a semantic segmentation network driven by deep learning is used to identify the wall (plane class) and cracks (hole class), and adjacent abnormalities between the plane and the hole are verified; through calculation, it is found that the difference in the reflection intensity between the crack and the wall in the near-infrared band is significant (such as a difference of 0.8), and the comprehensive difference value is 0.72; the quality of the crack and the wall is judged. If the comprehensive difference value of 0.72 > the tolerance threshold of 0.6, it is marked as a crack abnormality, and a quality inspection result including the abnormal area with geometric structure fracture or geometric structure distortion and the severity of the abnormality is output.

[0030] The embodiment of the present invention solves the problems of high false detection rate and missed detection of topological fractures caused by light interference in the quality inspection of traditional real scene three-dimensional models; the standardized input data improves the robustness of the model, and the multi-spectral feature correlation verification enhances the credibility of abnormal detection, realizing high-precision and automated geometric quality inspection in complex lighting scenarios.

[0031] The present invention provides a specific embodiment, step 103, based on the standardized input data, using a semantic segmentation network driven by deep learning to extract the correlation features between multi-spectral bands, and combining the triangular mesh connection relationship of the real scene three-dimensional model to generate a pixel-level segmentation result, specifically including the following steps: Step 301: Input the standardized input data into a deep learning-driven geometric constraint semantic segmentation network to extract the correlation features between multi-spectral bands; In this step, the correlation features refer to the features generated after fusing multi-spectral band information through a geometric constraint semantic segmentation network driven by a deep learning network, including the joint representation of visible and non-visible light bands.

[0032] In the embodiment of the present invention, the geometric constraint semantic segmentation network performs cross-channel feature fusion on the visible and non-visible light bands (such as near-infrared and thermal infrared) in the standardized input data through a multi-layer stacked feature extraction unit, establishes the correlation relationship between different bands by sharing cross-channel weight parameters, and finally outputs the correlation features between multi-spectral bands.

[0033] Step 302: Obtain the target hierarchical feature map output by the geometric constraint semantic segmentation network, perform an upsampling operation and a feature stitching process on the target hierarchical feature map to generate a fused feature map; In this step, the target hierarchical feature map refers to the low-resolution feature map output by the deep layer of the semantic segmentation network, which contains abstract correlation features. The fused feature map refers to the feature map containing multi-scale geometric structure information generated through upsampling and feature stitching operations.

[0034] In the embodiment of the present invention, the target hierarchical feature map is a low-resolution feature map output by the deep layer of the network (such as with a resolution of 1 / 16 of the original input), which is gradually upsampled to a higher resolution (such as 1 / 8, 1 / 4, 1 / 1) through a bilinear interpolation algorithm, and the feature map of the current level is spatially aligned with the intermediate feature map of the same resolution in the shallow layer of the network for channel dimension stitching after each upsampling, and finally a fused feature map integrating multi-scale geometric structure information is generated.

[0035] Step 303: Based on the triangular mesh connection relationship of the real scene three-dimensional model, calculate the weighted relationship between the feature similarity and the segmentation prediction difference degree of adjacent pixels in the fused feature map to generate a constraint weight matrix; In this step, the feature similarity refers to the similarity measure of adjacent pixels in the multi-spectral channel, which is calculated through a cosine similarity algorithm. The segmentation prediction difference degree refers to the measure of the inconsistency of adjacent pixels in the segmentation prediction labels, which is calculated based on the category difference value (such as cross entropy). The weighted relationship refers to the calculation relationship of the linear combination of the feature similarity and the segmentation prediction difference degree according to a preset ratio. The constraint weight matrix refers to the matrix generated according to the weighted relationship, and the matrix element values reflect the intensity of the geometric continuity constraint between adjacent pixels.

[0036] In an embodiment of the present invention, based on the triangular mesh connection relationship of the real - scene three - dimensional model, adjacent pixel pairs corresponding to each edge of the triangular patches in the fused feature map are determined; the feature vectors of each pixel on the multi - spectral channels are extracted, and the feature similarity is calculated; the segmentation prediction probability distribution vector of each pixel is extracted, and the segmentation prediction difference degree is calculated; according to a preset weighting coefficient, the feature similarity and the segmentation prediction difference degree are superimposed to generate a weighted constraint value corresponding to each adjacent pixel pair; the weighted constraint value is mapped to the fused feature map to construct a constraint weight matrix, and each element value in the matrix reflects the geometric continuity strength that should be satisfied between adjacent pixels.

[0037] Step 304: Adjust the segmentation prediction probability distribution of the fused feature map according to the constraint weight matrix, so that the segmentation prediction values of adjacent pixels satisfy the geometric continuity constraint, and obtain an adjusted fused feature map. In this step, the segmentation prediction value refers to the class probability value of a single pixel in the segmentation prediction result (for example, [0.2, 0.8] means the probability of belonging to the second class is 80%). The geometric continuity constraint refers to the rule that forces the segmentation results of adjacent pixels to satisfy topological continuity, which is realized through the constraint weight matrix. The adjusted fused feature map refers to the feature map after applying the geometric continuity constraint, and its segmentation prediction result has eliminated isolated outliers.

[0038] In an embodiment of the present invention, the constraint weight matrix is adjusted with the segmentation prediction probability distribution of the fused feature map to force the prediction probability values of adjacent pixels to approach consistency (for example, when the weight in the edge region is higher, the probability that adjacent pixels are predicted as the same class increases), thereby eliminating isolated outliers in the segmentation result, making the segmentation prediction values of adjacent pixels satisfy the geometric continuity constraint, and obtaining an adjusted fused feature map.

[0039] Step 305: Perform a per - pixel classification operation on the adjusted fused feature map to generate a pixel - level segmentation result.

[0040] In an embodiment of the present invention, the maximum - value point function is used to perform a maximum - value operation on the multi - class probability values of each pixel in the adjusted fused feature map to generate a final pixel - level segmentation result, and the label value of each pixel corresponds to the geometric structure type it belongs to (such as plane, edge, hole).

[0041] The embodiment of the present invention solves the problems of high mis - detection rate of geometric structures and missed detection of topological breaks caused by complex light interference in traditional three - dimensional model quality inspection; embeds physical geometric rules into the deep - learning inference process through the constraint weight matrix to improve the detection robustness in complex scenes (such as shadow - interlaced regions); the feature stitching operation retains multi - scale geometric details and avoids the loss of small - scale structures; the per - pixel classification ensures high - precision spatial positioning of the detection results.

[0042] The present invention provides a specific embodiment. In step 303, based on the triangular mesh connection relationship of the real-scene three-dimensional model, the weighted relationship between the feature similarity and the segmentation prediction difference degree of adjacent pixels in the fused feature map is calculated to generate a constraint weight matrix, which specifically includes the following steps: Step 311: Based on the connection relationship of the surface triangular meshes of the real-scene three-dimensional model, determine the adjacent pixel pairs corresponding to each edge of the triangular patches in the fused feature map; In this step, the edge of the triangular patch refers to the boundary line segment of a single triangular patch in the real-scene three-dimensional model, which is formed by connecting two vertices and is used to define the geometric correlation of adjacent pixel pairs. The adjacent pixel pair refers to two pixels that share the same edge of the triangular patch in the fused feature map, which is a spatial adjacent relationship defined based on the geometric structure of the real-scene three-dimensional model.

[0043] In the embodiment of the present invention, first, the surface triangular mesh structure of the real-scene three-dimensional model is parsed, and the edge connection information of all triangular patches is extracted; according to the spatial position of the edge of the triangular patch in the real-scene three-dimensional model, it is mapped to the two-dimensional pixel coordinate system of the fused feature map, and two pixels sharing the same edge are determined as adjacent pixel pairs. This process ensures that the adjacent pixel pairs are strictly correlated with the geometric topology of the three-dimensional model.

[0044] Step 312: Extract the feature vectors of each pixel in the adjacent pixel pairs in the fused feature map on the multi-spectral channels to calculate the feature similarity; In this step, the multi-spectral channel refers to a data channel that includes visible light and at least one non-visible light band (such as thermal infrared), which characterizes the reflection or radiation characteristics of different bands of the scene. The feature vector refers to a vector formed by arranging the feature values of a single pixel on the multi-spectral channels in the channel order, which is used to quantify the multi-spectral characteristics of the pixel. The feature similarity refers to the measure of the feature similarity of adjacent pixel pairs on the multi-spectral channels, which is obtained by calculating the reciprocal of the sum of the squared channel differences. The larger the value, the more consistent the spectral features.

[0045] In the embodiment of the present invention, the feature values of each pixel on the multi-spectral channels (such as visible light, near-infrared, thermal infrared) in the fused feature map are vectorized to generate feature vectors; for each adjacent pixel pair, the reciprocal of the sum of the squared differences of each channel between the feature vectors of the two pixels is calculated as the feature similarity, where the smaller the sum of the squared differences, the more similar the spectral features, and the larger the reciprocal calculation result.

[0046] Step 313: Extract the segmentation prediction probability distribution of each pixel in the adjacent pixel pairs in the fused feature map to calculate the segmentation prediction difference degree; In this step, the segmentation prediction probability distribution refers to the set of probability values of each pixel in adjacent pixel pairs in the fused feature map belonging to different geometric structure categories in the segmentation prediction result, reflecting the prediction confidence of the deep learning model for pixel categories. The segmentation prediction difference degree refers to the measure of the inconsistency in the segmentation prediction results of adjacent pixel pairs, calculated by the sum of the absolute values of category probability differences. The larger the value, the more significant the prediction contradiction.

[0047] In the embodiment of the present invention, the segmentation prediction probability distribution of each pixel in the fused feature map is obtained by traversing each pixel position in the fused feature map, reading its corresponding multi-dimensional feature vector, and converting the vector into probability values of each geometric category through a classifier to generate the segmentation prediction probability distribution of each pixel. For example, the probability distribution of pixel A is 0.7 for the plane category and 0.3 for the hole category, indicating that the probability of this pixel belonging to the plane area is 70% and the probability of belonging to the hole area is 30%. For each adjacent pixel pair, the sum of the absolute value differences of the probability distribution vectors of the two pixels in each category is calculated as the segmentation prediction difference degree, where the larger the sum of the differences, the more inconsistent the segmentation prediction results.

[0048] Step 314: According to the preset weighting coefficient, perform normalized linear superposition on the feature similarity and the segmentation prediction difference degree to generate a weighted constraint value corresponding to each adjacent pixel pair; In this step, the preset weighting coefficient refers to the weight ratio parameter of the feature similarity and the segmentation prediction difference degree set in advance, used to control the contribution ratio of the two to the final constraint strength. The weighted constraint value refers to the value generated by normalized linear superposition, reflecting the geometric continuity constraint strength that should be satisfied between adjacent pixel pairs. The larger the value, the stronger the constraint.

[0049] In the embodiment of the present invention, the feature similarity and the segmentation prediction difference degree are respectively normalized so that their numerical ranges are in the interval [0, 1]. The normalized feature similarity is multiplied by the preset feature similarity weight coefficient, and the normalized segmentation prediction difference degree is multiplied by the segmentation prediction difference degree weight coefficient, and the two are added to generate the weighted constraint value.

[0050] Step 315: Map the weighted constraint value to the fused feature map to construct a constraint weight matrix with the same resolution as the fused feature map; In the embodiments of the present invention, based on the row and column positions of adjacent pixel pairs corresponding to the edges of triangular patches in the fused feature map, the row and column indices of each element in the initial constraint weight matrix are determined, where the row and column coordinates of the two pixels in each adjacent pixel pair are used as the row index and column index of the matrix element respectively; the weighted constraint values of each adjacent pixel pair are simultaneously filled into the positions corresponding to the row index and column index in the initial constraint weight matrix and their symmetric positions, so that the element values of row index i, column index j, row index e, and column index f in the matrix are the same; for pixel pairs in the fused feature map that are not associated with the edges of triangular patches, a preset default constraint value is filled in the corresponding positions of the initial constraint weight matrix; after traversing all adjacent pixel pairs to complete the filling operation of the initial constraint weight matrix, a constraint weight matrix with the same resolution as the fused feature map is generated, where the value of each matrix element reflects the geometric continuity constraint strength that should be satisfied between the corresponding pixel pairs.

[0051] The embodiments of the present invention solve the problems of high false detection rate caused by the separation of rules and data and difficult geometric consistency verification under complex lighting in traditional three-dimensional model quality inspection; through the constraint weight matrix, the geometric structure features of the three-dimensional model and the characteristics of multi-spectral data are deeply fused to improve the detection accuracy and robustness of fracture or distortion regions.

[0052] For example, in the quality inspection of the 3D model of ancient building roof tiles, first, based on the edge relationship of the triangular mesh on the tile surface, traverse the edges of all triangular patches, and define two pixels sharing the same edge as an adjacent pixel pair to ensure coverage of all tile joint areas. Second, for each adjacent pixel pair, extract its feature vectors in the visible light (such as red, green, and blue bands) and the near-infrared band, and calculate the sum of the squared differences in reflectance of the adjacent pixels in each band. If the sum of the squared differences is small (for example, the difference is less than the threshold of 5 for a complete tile due to consistent material), it is determined to have a high feature similarity; if the sum of the squared differences increases significantly (for example, the difference exceeds the threshold of 20 for a missing tile due to the exposed base layer), the feature similarity decreases. Next, extract the segmentation prediction probability distribution vectors of each pixel in the adjacent pixel pair (such as the "hole class" probability and the "plane class" probability), and calculate the sum of the absolute values of the probability differences between the two. If the sum of the absolute values of the differences exceeds the threshold of 0.7 (for example, the probability of predicting a missing area as the "hole class" is 0.9, and the probability of predicting the adjacent complete area as the "plane class" is 0.85, then the sum of the absolute values of the differences is 0.15 + 0.15 = 0.3, not exceeding the standard; while if the difference at the missing edge increases significantly to 0.8), it is determined to have a high segmentation prediction difference degree. Subsequently, according to the preset weighting coefficients (feature similarity weight 0.6, segmentation prediction difference degree weight 0.4), normalize the feature similarity and the segmentation prediction difference degree to the [0, 1] interval respectively and then linearly superimpose them to generate a weighted constraint value: for example, if the feature similarity of an adjacent pixel pair is 0.8 and the segmentation prediction difference degree is 0.3, the weighted constraint value is (0.8×0.6) + (1 - 0.3×0.4) = 0.72 (high constraint value); while if the feature similarity is 0.3 and the segmentation prediction difference degree is 0.8, the weighted constraint value is (0.3×0.6) + (1 - 0.8×0.4) = 0.18 + 0.24 = 0.42 (low constraint value). Finally, map the weighted constraint values of all adjacent pixel pairs to a constraint weight matrix with the same resolution as the fused feature map, where the high constraint value area (such as the joint of a complete tile) corresponds to the high weight value in the matrix, forcing the segmentation result to maintain continuity; the low constraint value area (such as the missing edge) corresponds to the low weight value, allowing the segmentation boundary to mutate.

[0053] The present invention provides a specific embodiment. Step 304: Adjust the segmentation prediction probability distribution of the fused feature map according to the constraint weight matrix to make the segmentation prediction values of adjacent pixels satisfy the geometric continuity constraint, and obtain the adjusted fused feature map, which specifically includes the following steps: Step 321: Calculate the adjustment coefficient of each pixel according to the constraint intensity value of the adjacent pixel pair in the constraint weight matrix; [[ID=⑥]] In this step, the constraint strength value refers to the geometric continuity constraint strength of adjacent pixel pairs in the constraint weight matrix. The larger the value, the more consistent the predicted results of adjacent pixels need to be. The adjustment coefficient refers to the adjustment amplitude calculated based on the constraint strength and probability difference, and is used to force the predicted values of adjacent pixels to converge.

[0054] In the embodiment of the present invention, for each pixel and all its adjacent pixels (determined based on the triangular mesh connection relationship), the constraint strength value of the corresponding adjacent pixel pair is obtained from the constraint weight matrix; the absolute value of the probability difference between this pixel and each adjacent pixel in the same geometric category is calculated (for example, the probability difference in the plane category = |0.7 - 0.2| = 0.5), and the constraint strength value is multiplied by the absolute value of the probability difference to generate an adjustment coefficient for each adjacent pixel and each category. For example, if the constraint strength is 0.9 and the probability difference is 0.5, then the adjustment coefficient = 0.9×0.5 = 0.45.

[0055] Step 322: Use the adjustment coefficient to adjust the segmentation prediction probability distribution of each pixel in the adjacent pixel pairs in the fused feature map to obtain the superimposed segmentation prediction probability distribution; In this step, the superimposed segmentation prediction probability distribution refers to the unnormalized probability distribution after the adjustment coefficient acts on the original segmentation prediction probability distribution, and the sum of probabilities may not be 1.

[0056] In the embodiment of the present invention, the adjustment operation realizes the convergence of the predicted results of adjacent pixels by directly increasing or decreasing the probability values. In specific implementation, for each geometric category of each pixel, if the segmentation prediction probability distribution of this category of the adjacent pixel is higher than that of the current pixel, the adjustment coefficient is subtracted from the segmentation prediction probability distribution of the current pixel, and vice versa. For example, the segmentation prediction probability distribution of the plane category of pixel A is 0.7, and that of its adjacent pixel B is 0.2 (probability difference 0.5), and the adjustment coefficient is 0.45. Then, after adjustment, the segmentation prediction probability distribution of the plane category of pixel A = 0.7 - 0.45 = 0.25, and the segmentation prediction probability distribution of the plane category of pixel B = 0.2 + 0.45 = 0.65.

[0057] Step 323: Perform normalization processing on the superimposed segmentation prediction probability distribution to generate an updated segmentation prediction probability distribution; In this step, the updated segmentation prediction probability distribution refers to the legal probability distribution after normalization, with the sum of probabilities being 1 and the adjustment effect being retained.

[0058] In the embodiments of the present invention, the normalization process ensures that the sum of the adjusted probability values is 1. In a specific implementation, the cumulative segmentation prediction probability distributions of each pixel are accumulated by category to calculate the total value (for example, plane class 0.25 + hole class 0.75 = 1.0), and then the cumulative segmentation prediction probability distributions of each category are divided by the total value. For example, the cumulative segmentation prediction probability distribution of pixel A is plane class 0.25 and hole class 0.75, and after normalization, it remains 0.25 and 0.75; if the cumulative segmentation prediction probability distribution is plane class 0.5 and hole class 0.6 (total 1.1), then the updated segmentation prediction probability distribution after normalization is 0.5 / 1.1 ≈ 0.45, 0.6 / 1.1 ≈ 0.55.

[0059] Step 324: Replace the segmentation prediction probability distribution of the corresponding pixel in the fusion feature map with the updated segmentation prediction probability distribution to generate an adjusted fusion feature map, so that the segmentation prediction values of adjacent pixels satisfy the geometric continuity constraint; In this step, the adjusted fusion feature map refers to the feature map after the updated probability distribution replaces the original segmentation prediction probability distribution, and is used for subsequent per-pixel classification to generate the final pixel-level segmentation result.

[0060] In the embodiments of the present invention, in a specific implementation, the normalized and updated segmentation prediction probability distributions of each pixel are re-encoded into a multi-dimensional feature vector in the category order, and the original feature vector at the corresponding position in the fusion feature map is replaced. For example, if the original feature vector is [0.7, 0.3] and the adjusted one is [0.25, 0.75], then the original data is directly overwritten to form a new adjusted feature map. In the adjusted fusion feature map, wall pixels are more inclined to the crack class, and crack region pixels are more inclined to the plane class. However, due to the high constraint strength, the prediction difference between the two is significantly reduced, meeting the geometric continuity requirements of the wall-crack transition region.

[0061] The embodiments of the present invention solve the problem of false detection of fractures or distortions caused by traditional models ignoring topological rules by dynamically adjusting the prediction probability distributions of adjacent pixels to force the pixel-level segmentation results to satisfy the geometric continuity constraint; the adjustment coefficient is adaptively generated based on the constraint strength and probability difference driven by multi-spectral, avoiding manual intervention; the normalization process ensures the legality of probabilities, and finally improves the segmentation accuracy and structural rationality in complex scenarios.

[0062] The present invention provides a specific embodiment. In step 104, based on a preset topological continuity rule, the pixel-level segmentation result is verified to generate a multi-spectral feature association data set, which specifically includes the following steps: Step 401: Obtain the geometric structure type labels of each pixel in the adjacent pixel pairs from the pixel-level segmentation result, and generate a segmentation label combination in combination with the geometric structure type labels; In this step, the geometric structure type label refers to the identification of the geometric category to which each pixel in the pixel-level segmentation result belongs, including plane class labels (continuous surfaces), edge class labels (structural boundaries), and hole class labels (broken or missing regions). The segmentation label combination refers to the pairing form of the geometric structure type labels of two pixels in an adjacent pixel pair, such as plane class - hole class, edge class - plane class.

[0063] In the embodiment of the present invention, first, the geometric structure type label of each pixel is extracted from the pixel-level segmentation result through morphological operations or image segmentation algorithms; all adjacent pixel pairs will be traversed, and the geometric structure type labels of the pixels therein will be recorded and combined in order (such as plane class - hole class) to form a segmentation label combination for subsequent topological rule verification.

[0064] Step 402: According to the preset topological continuity rules, determine whether the segmentation label combination meets the geometric continuity requirements. The preset topological continuity rules include allowing plane class labels to be adjacent to edge class labels and prohibiting plane class labels from being adjacent to hole class labels; In this step, the geometric continuity requirement refers to the rule based on the geometric integrity of the surface of the real scene three-dimensional model, which is used to judge whether the segmentation label combination meets the physical structure rationality.

[0065] In the embodiment of the present invention, the compliance of the label combination is determined according to the preset topological continuity rules. Through pairwise matching, if the segmentation label combination violates the prohibition rule (prohibiting the adjacent of plane class and hole class), it is marked as non-compliant, otherwise it is compliant. For example, the label "building - vegetation hole" is determined to be non-compliant because it violates the rule of prohibiting the adjacent of plane and hole, while "building - road edge" complies with the rule.

[0066] Step 403: Assign a valid identifier to the segmentation label combination determined to meet the geometric continuity requirements to generate a target segmentation label combination; In this step, the valid identifier refers to the unique identifier that marks the segmentation label combination that meets the geometric continuity requirements and is used to identify the normal area.

[0067] In the embodiment of the present invention, the segmentation label combination determined to be compliant is assigned a valid identifier (such as the value 1) to form a set of target segmentation label combinations. For example, after marking the compliant segmentation label combinations, only the valid combinations such as "building - road edge" are retained, while the abnormal combinations such as "building - vegetation hole" are excluded.

[0068] Step 404: Assign an abnormal identifier to the segmentation label combination determined not to meet the geometric continuity requirements to generate a non-target segmentation label combination, and calculate the reflection intensity difference of the adjacent pixel pair corresponding to the non-target segmentation label combination, and combine the preset channel weight coefficient to generate a comprehensive difference value; In this step, the anomaly identifier is the unique identifier that marks the segmentation label combination violating the geometric continuity requirement, and is used to identify the anomaly area to be verified. The reflection intensity difference refers to the absolute difference in the reflection intensity values of adjacent pixel pairs in each channel of the multispectral data, and is used to quantify the contradiction of material properties. The preset channel weight coefficient refers to the contribution ratio parameter of each spectral channel preset in the difference calculation (such as the near-infrared weight of 0.7 and the visible light weight of 0.3). The comprehensive difference value is the value generated by weighting the reflection intensity differences of each channel, and reflects the contradiction intensity of the anomaly area in the multispectral dimension.

[0069] In the embodiment of the present invention, for the segmentation label combination determined as "not conforming", an anomaly identifier is assigned to generate a non-target segmentation label combination; the reflection intensity values of each channel in the multispectral imaging data corresponding to the non-target segmentation label combination are obtained for the adjacent pixel pairs; for each channel, the absolute difference in the reflection intensity values of the two pixels in the adjacent pixel pair is calculated to generate the reflection intensity difference of each channel; the reflection intensity difference of each channel is multiplied by the preset weight coefficient (such as the near-infrared weight of 0.7 and the visible light weight of 0.3) to generate the weighted difference value of each channel; the weighted difference values of all channels are summed to generate a comprehensive difference value to quantify the spectral contradiction of the anomaly area.

[0070] Step 405: Correlate and store the target segmentation label combination, the non-target segmentation label combination, the comprehensive difference value, the position coordinates of the adjacent pixel pairs corresponding to the segmentation label combination, and the preset channel weight coefficient to generate a multispectral feature correlation dataset containing a verification identifier, where the verification identifier includes a valid identifier and an anomaly identifier.

[0071] In this step, the verification identifier refers to the code that uniquely identifies the type of the segmentation label combination, including a valid identifier and an anomaly identifier.

[0072] In the embodiment of the present invention, all data is stored as a structured dataset according to the correlation relationship. For example, each pixel pair record includes building-road edge (valid identifier), building-vegetation hole (anomaly identifier), comprehensive difference value of 0.13, pixel position coordinates (100, 50) and (101, 50), and the preset channel weight coefficient, and finally is integrated into a multispectral feature correlation dataset for subsequent model optimization.

[0073] The embodiment of the present invention solves the problems of high false detection rate and poor adaptability to complex scenarios caused by the dependence on a single data source in the traditional three-dimensional model quality inspection; quantifies the spectral contradiction of the anomaly area through the comprehensive difference value, combines valid and anomaly identifiers to achieve full-scale data analysis, improves the detection accuracy and efficiency, and at the same time supports the traceability verification of the anomaly area.

[0074] The present invention provides a specific embodiment. In step 105, according to the comparison result between the geometric feature segmentation deviation between different multispectral bands in the multispectral feature association dataset and the preset tolerance range, the quality of the real-scene three-dimensional model is inspected to obtain a quality inspection result. The quality inspection result includes abnormal areas of geometric structure fracture or geometric structure distortion, and specifically includes the following steps: Step 501: Define a hierarchical difference threshold corresponding to the resolution level and scene type of the real-scene three-dimensional model based on the preset tolerance range; In this step, the resolution level refers to the accuracy level of the real-scene three-dimensional model, such as millimeter level (0.1 mm / pixel) or centimeter level (1 cm / pixel), which affects the setting of the hierarchical difference threshold. The scene type refers to the application environment classification of the real-scene three-dimensional model, including indoor (stable lighting) and outdoor (complex lighting), and is used to adjust the threshold sensitivity. The hierarchical difference threshold refers to the difference determination threshold dynamically set according to the resolution level and scene type of the real-scene three-dimensional model. For example, the threshold for a high-resolution outdoor scene is 0.6.

[0075] In the embodiment of the present invention, the preset tolerance range includes the difference threshold matching rules for different model resolutions (such as millimeter level, centimeter level) and scene types (such as indoor, outdoor). Among them, a high-resolution model corresponds to a lower difference threshold (such as 0.5), a low-resolution model corresponds to a higher difference threshold (such as 0.8), and the outdoor scene needs to additionally increase the threshold due to complex lighting (such as threshold × 1.2).

[0076] Step 502: Compare the comprehensive difference value in the multispectral feature association dataset with the hierarchical difference threshold to screen out target adjacent pixel pairs; In this step, the target adjacent pixel pairs refer to the adjacent pixel pairs whose comprehensive difference value exceeds the hierarchical difference threshold, and are used as candidates for the area to be confirmed as abnormal.

[0077] In the embodiment of the present invention, the position coordinates of the area to be confirmed as abnormal are converted into the vertex coordinates of the corresponding triangular patches in the surface triangular mesh. Each position coordinate is associated with the nearest triangular patch vertex through the nearest neighbor matching algorithm; based on the coordinates of this vertex, all triangular patches containing this vertex are extracted as the initial set of patches to be processed; the first triangular patch is selected as the current patch, marked as the processed state, and added to the temporary patch set of the candidate abnormal area; traverse all adjacent triangular patches sharing an edge or vertex with the current patch. If the adjacent triangular patch is not marked as the processed state and is included in the initial set of patches to be processed, it is marked as the processed state and added to the temporary patch set. Repeat this traversal operation until no new triangular patches are added; all triangular patches in the temporary patch set are merged into the candidate abnormal area, and the coverage range of the candidate abnormal area is defined by the geometric area of the merged triangular patch set.

[0078] Step 503: Mark the target adjacent pixel pairs as regions with anomalies to be confirmed, and map the position coordinates of the regions with anomalies to be confirmed to the surface triangular meshes of the real-scene three-dimensional model, so as to perform a spatial adjacency traversal operation on the triangular patches corresponding to the regions with anomalies to be confirmed, and merge all triangular patches sharing edges and vertices into candidate anomaly regions; In this step, the regions with anomalies to be confirmed refer to the set of initially screened anomaly pixel pairs, which need to be further spatially merged and region-level determined. The triangular patches refer to the triangular mesh units composed of three vertices and three edges on the surface of the real-scene three-dimensional model, which are used for geometric structure representation. The candidate anomaly regions refer to the continuous anomaly regions generated by merging triangular patches sharing edges or vertices, which represent the potential defect ranges.

[0079] In the embodiment of the present invention, convert the position coordinates of the regions with anomalies to be confirmed into the corresponding triangular patch vertex coordinates in the surface triangular meshes of the real-scene three-dimensional model, where each position coordinate is associated with the nearest triangular patch vertex through the nearest neighbor matching algorithm, and extract all triangular patches containing the triangular patch vertices as the initial set of patches to be processed; select the first triangular patch from this set as the current patch, mark it as the processed state, and add it to the temporary patch set of the candidate anomaly regions; traverse all adjacent triangular patches sharing edges or vertices with the current patch, if the adjacent triangular patch is not marked as the processed state and is included in the initial set of patches to be processed, then mark the adjacent triangular patch as the processed state and add it to the temporary patch set, repeat this traversal operation until no new triangular patches are added; merge all triangular patches in the temporary patch set into the candidate anomaly regions, and its coverage range is defined by the geometric area of the merged triangular patch set.

[0080] Step 504: Determine the final anomaly region according to the comparison result between the total sum of the comprehensive difference values of all target adjacent pixel pairs in the candidate anomaly region and the preset region tolerance value; In this step, the total sum of the comprehensive difference values refers to the accumulated result of the comprehensive difference values of all target adjacent pixel pairs in the candidate anomaly region, which is used to quantify the regional anomaly intensity. The preset region tolerance value refers to the region-level threshold for determining whether the candidate anomaly region is the final anomaly. The final anomaly region refers to the candidate anomaly region that meets the region tolerance condition, and is determined as the defect region with geometric structure fracture or distortion.

[0081] In the embodiment of the present invention, accumulate the comprehensive difference values of all target adjacent pixel pairs in the candidate anomaly region (such as 0.7×10 = 7.0), if the total sum exceeds the preset region tolerance value (such as 5.0), then it is determined as the final anomaly region, otherwise it is downgraded to a low-risk region.

[0082] Step 505: Correlate and store the position coordinates, coverage range, segmentation label combination type, and the total comprehensive difference value of the final abnormal area to generate a quality inspection result, where the quality inspection result includes abnormal areas with geometric structure fractures or geometric structure distortions. In this step, the segmentation label combination type refers to the label combination form of adjacent pixel pairs in the final abnormal area (such as plane class - hole class), which is used to identify the type of abnormality.

[0083] In the embodiment of the present invention, the coverage range is defined by the geometric area of the triangular patch set after merging the candidate abnormal areas. The segmentation label combination type (such as plane class - hole class) identifies the type of abnormality (fracture), and the total comprehensive difference value (such as 7.0) reflects the severity of the abnormality. The above information is correlated and stored to generate a quality inspection result including abnormal areas with geometric structure fractures or geometric structure distortions.

[0084] The embodiment of the present invention solves the problem of insufficient accuracy caused by misdetection of isolated pixels and rigid thresholds in traditional three - dimensional model quality inspection; quantifies the severity of abnormalities through the total comprehensive difference value, and clarifies the nature of defects (fracture or distortion) in combination with the label combination type, realizing the output of an interpretable quality inspection result.

[0085] The present invention provides a specific embodiment. In step 102, the multi - spectral imaging data is optimized to generate standardized input data, which specifically includes the following steps: Step 201: Convert the data of each multi - spectral band in the multi - spectral imaging data to a preset frequency domain space through a dynamic optimization processing node to obtain a target frequency domain space. Based on the frequency distribution characteristics of the target frequency domain space, perform noise suppression processing on the multi - spectral imaging data to generate first - target imaging data, where the calculation resource allocation strategy of the dynamic optimization processing node is dynamically adjusted according to the data volume of the multi - spectral imaging data and the average scene illumination intensity. In this step, the dynamic optimization processing node refers to a processing module that dynamically allocates computing resources according to the data volume of the multi - spectral imaging data and the scene illumination, and is used to perform operations such as frequency - domain noise suppression. The preset frequency domain space refers to a pre - defined frequency domain coordinate system (such as the complex space after Fourier transform), which is used to analyze the frequency components of the image. The target frequency domain space refers to the frequency domain space where the multi - spectral data is located after conversion, which contains frequency distribution information. The frequency distribution characteristics refer to the energy distribution of different frequency components in the target frequency domain space, where high - frequency components correspond to noise and low - frequency components correspond to effective textures. The first - target imaging data refers to the multi - spectral data after frequency - domain noise suppression processing, which retains low - frequency texture features.

[0086] In an embodiment of the present invention, the dynamic optimization processing node dynamically allocates computing resources according to the data volume of the multispectral data and the average scene illumination intensity (for example, GPU resources are preferentially allocated when the data volume is high). The image data of each band is converted to a preset frequency domain space (such as a frequency domain coordinate system) through a frequency domain conversion algorithm (such as the fast Fourier transform). Based on the frequency distribution characteristics (for example, high-frequency components correspond to noise and low-frequency components correspond to textures), high-frequency noise components (such as sensor noise or stripe noise) are filtered out, and low-frequency texture features (such as building surface details) are retained to generate first target imaging data after noise suppression.

[0087] Step 202: Adjust the pixel brightness values of the first target imaging data according to the illumination intensity distribution in the local area of the first target imaging data to generate second target imaging data; In this step, the local area refers to the neighborhood range defined by a preset window size (such as 5×5 or 3×3 pixels) centered on the current pixel in the multispectral imaging data, which is used to statistically calculate illumination intensity distribution characteristics such as the brightness mean and variance of the pixels within this range to drive the adaptive illumination equalization operation. The window size of the local area is dynamically set according to the image resolution and scene complexity. A smaller window is used for high-resolution images to retain details, and a larger window is used for low-resolution images to improve statistical stability. The illumination intensity distribution refers to the statistical characteristics of the brightness values in different regions of the image (such as local mean and variance).

[0088] In an embodiment of the present invention, for the image of each band in the first target imaging data, the illumination intensity distribution (such as brightness mean and variance) of the local area (such as a 5×5 pixel window) is statistically calculated. For the high-light area (brightness value higher than the preset threshold), the pixel brightness value is reduced by a compression coefficient (such as 0.8), and for the shadow area (brightness value lower than the preset threshold), the pixel brightness value is increased by an enhancement coefficient (such as 1.5) to generate second target imaging data with balanced illumination. The pixel brightness value refers to the optical reflection or radiation intensity value of a single pixel in a specific multispectral band. The second target imaging data refers to the multispectral image data after illumination equalization processing, eliminating the interference of high light and shadow.

[0089] Step 203: Perform a spatial normalization operation on the second target imaging data to generate third target imaging data; In this step, the third target imaging data refers to the multispectral image data after spatial alignment and cropping, with consistent pixel coordinates.

[0090] In the embodiments of the present invention, the multi-band images in the second target imaging data are uniformly cropped according to a preset size (such as 512×512 pixels), and the images of different multispectral bands are aligned to the same spatial coordinate system through affine transformation (such as translation, rotation, and scaling) to ensure that the pixel coordinates of the same ground object position in all multispectral band images are exactly the same, generating spatially aligned third target imaging data.

[0091] Step 204: According to a preset numerical range, linearly scale the pixel values of each multispectral band in the third target imaging data to generate standardized input data adapted to the deep learning network.

[0092] In this step, the preset numerical range refers to the input data range required by the deep learning network (such as [0,1] or [-1,1]). The pixel value refers to the value of a single pixel in the image, representing its optical characteristics (such as reflection intensity).

[0093] In the embodiments of the present invention, for each multispectral band of the third target imaging data, calculate the global maximum and minimum values of its pixel values, and linearly map the pixel values to the preset numerical range (such as the [0,1] interval) according to the formula (scaled value = (original value - minimum value) / (maximum value - minimum value)) to generate standardized input data.

[0094] The embodiments of the present invention eliminate sensor noise through frequency domain noise suppression, solve complex lighting interference through adaptive lighting equalization, ensure multi-band alignment through spatial standardization, and adapt deep learning input through numerical normalization, generating standardized input data with high consistency and low noise, improving the accuracy and robustness of subsequent 3D modeling and quality inspection, and being particularly applicable to scenes with interlaced shadows and high dynamic range.

[0095] Figure 2 The following is a schematic structural diagram of a real-scene 3D model quality inspection system integrating deep learning provided by the embodiments of the present invention, as Figure 2 shown, the system includes: An acquisition module 21, configured to acquire multi-spectral imaging data of a target scene, and construct a real-scene 3D model based on the multi-spectral imaging data; An optimization module 22, configured to perform optimization processing on the multi-spectral imaging data to generate standardized input data; An extraction module 23, configured to extract the correlation features between multi-spectral bands based on the standardized input data by using a deep learning-driven semantic segmentation network, and generate a pixel-level segmentation result in combination with the triangular mesh connection relationship of the real-scene 3D model; A verification module 24, configured to verify the pixel-level segmentation result based on preset topological continuity rules to generate a multi-spectral feature correlation data set; A checking module 25 is configured to perform a quality check on the real-scene three-dimensional model according to a comparison result between a geometric feature segmentation deviation between different multispectral bands in a multispectral feature association dataset and a preset tolerance range, so as to obtain a quality check result, where the quality check result includes abnormal areas with geometric structure breaks or geometric structure distortions.

[0096] Figure 2 The described real-scene three-dimensional model quality checking system integrating deep learning can execute Figure 1 The described real-scene three-dimensional model quality checking method in the illustrated embodiment, its implementation principle and technical effects will not be elaborated. For the real-scene three-dimensional model quality checking system in the above embodiment, the specific manners in which each module and unit perform operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0097] In a possible design, Figure 2 The real-scene three-dimensional model quality checking system in the illustrated embodiment can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32; The storage component 31 stores one or more computer instructions, where the one or more computer instructions are for the processing component 32 to call and execute.

[0098] The processing component 32 is configured to: obtain multispectral imaging data of a target scene, and construct a real-scene three-dimensional model based on the multispectral imaging data; perform optimization processing on the multispectral imaging data to generate standardized input data; extract association features between multispectral bands based on the standardized input data by using a deep learning-driven semantic segmentation network, and combine the triangular mesh connection relationship of the real-scene three-dimensional model to generate a pixel-level segmentation result; perform verification on the pixel-level segmentation result based on a preset topological continuity rule to generate a multispectral feature association dataset; perform a quality check on the real-scene three-dimensional model according to a comparison result between a geometric feature segmentation deviation between different multispectral bands in the multispectral feature association dataset and a preset tolerance range, so as to obtain a quality check result, where the quality check result includes abnormal areas with geometric structure breaks or geometric structure distortions.

[0099] Among them, the processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-mentioned method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components for executing the above-mentioned method.

[0100] The storage component 31 is configured to store various types of data to support the operation of the terminal. The storage component may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0101] Of course, the computing device may also necessarily include other components, such as input / output interfaces, display components, communication components, etc.

[0102] The input / output interface provides an interface between the processing component and the peripheral interface module, and the above-mentioned peripheral interface module may be an output device, an input device, etc.

[0103] The communication component is configured to facilitate communication between the computing device and other devices in a wired or wireless manner, etc.

[0104] Among them, the computing device may be a physical device or an elastic computing host provided by a cloud computing platform, etc. At this time, the computing device may refer to a cloud server, and the above-mentioned processing component, storage component, etc. may be basic server resources leased or purchased from the cloud computing platform.

[0105] The embodiment of the present invention also provides a computer storage medium storing a computer program, and when the computer program is executed by a computer, it can implement the above-mentioned Figure 1 A method for inspecting the quality of a real-scene three-dimensional model integrating deep learning shown in the embodiment.

[0106] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0108] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments or equivalently replace some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for quality inspection of real - scene 3D models integrating deep learning, characterized in that, Including: Obtain the multi-spectral imaging data of the target scene, and construct a real-scene three-dimensional model based on the multi-spectral imaging data; Perform optimization processing on the multi-spectral imaging data to generate standardized input data; Based on the standardized input data, use a deep learning-driven semantic segmentation network to extract the correlation features between multi-spectral bands, and combine the triangular mesh connection relationship of the real-scene three-dimensional model to generate a pixel-level segmentation result; Based on the preset topological continuity rules, verify the pixel-level segmentation result to generate a multi-spectral feature correlation data set; According to the comparison result between the geometric feature segmentation deviation between different multi-spectral bands in the multi-spectral feature correlation data set and the preset tolerance range, perform a quality inspection on the real-scene three-dimensional model to obtain a quality inspection result, and the quality inspection result includes abnormal areas with geometric structure fractures or geometric structure distortions.

2. The method according to claim 1, characterized in that, Based on the standardized input data, use a deep learning-driven semantic segmentation network to extract the correlation features between multi-spectral bands, and combine the triangular mesh connection relationship of the real-scene three-dimensional model to generate a pixel-level segmentation result, including: Input the standardized input data into a deep learning-driven geometric constraint semantic segmentation network to extract the correlation features between multi-spectral bands; Obtain the target-level feature map output by the geometric constraint semantic segmentation network, perform an upsampling operation and a feature stitching process on the target-level feature map to generate a fused feature map; Based on the triangular mesh connection relationship of the real-scene three-dimensional model, calculate the weighted relationship between the feature similarity and the segmentation prediction difference of adjacent pixel pairs in the fused feature map to generate a constraint weight matrix; Adjust the segmentation prediction probability distribution of the fused feature map according to the constraint weight matrix to make the segmentation prediction values of adjacent pixels satisfy the geometric continuity constraint, and obtain the adjusted fused feature map; Perform a per-pixel classification operation on the adjusted fused feature map to generate a pixel-level segmentation result.

3. The method according to claim 2, wherein Based on the triangular mesh connection relationship of the real-scene three-dimensional model, calculate the weighted relationship between the feature similarity and the segmentation prediction difference of adjacent pixel pairs in the fused feature map to generate a constraint weight matrix, including: Based on the connection relationship of the surface triangular mesh of the real-scene three-dimensional model, determine the adjacent pixel pairs corresponding to each triangular patch edge in the fused feature map; Extract the feature vectors of each pixel in the multi-spectral channels of the adjacent pixel pairs in the fused feature map to calculate the feature similarity; Extract the segmentation prediction probability distribution of each pixel in the adjacent pixel pairs in the fused feature map to calculate the segmentation prediction difference; According to the preset weighting coefficient, perform a normalized linear superposition of the feature similarity and the segmentation prediction difference to generate a weighted constraint value corresponding to each adjacent pixel pair; Map the weighted constraint value to the fused feature map to construct a constraint weight matrix consistent with the resolution of the fused feature map.

4. The method according to claim 2, wherein Adjust the segmentation prediction probability distribution of the fused feature map according to the constraint weight matrix to make the segmentation prediction values of adjacent pixels satisfy the geometric continuity constraint, and obtain the adjusted fused feature map, including: Calculate the adjustment coefficient for each pixel according to the constraint strength value of adjacent pixel pairs in the constraint weight matrix; Use the adjustment coefficient to adjust the segmentation prediction probability distribution of each pixel in the adjacent pixel pairs in the fusion feature map to obtain the superimposed segmentation prediction probability distribution; Perform normalization processing on the superimposed segmentation prediction probability distribution to generate an updated segmentation prediction probability distribution; Replace the segmentation prediction probability distribution of the corresponding pixels in the fusion feature map with the updated segmentation prediction probability distribution to generate an adjusted fusion feature map, so that the segmentation prediction values of adjacent pixels satisfy the geometric continuity constraint.

5. The method according to claim 1, characterized in that Based on the preset topological continuity rule, verify the pixel-level segmentation result to generate a multi-spectral feature association dataset, including: Obtain the geometric structure type labels of each pixel in the adjacent pixel pairs from the pixel-level segmentation result, and generate a segmentation label combination in combination with the geometric structure type labels; According to the preset topological continuity rule, determine whether the segmentation label combination meets the geometric continuity requirement. The preset topological continuity rule includes allowing planar class labels to be adjacent to edge class labels and prohibiting planar class labels from being adjacent to hole class labels; Assign a valid identifier to the segmentation label combination determined to meet the geometric continuity requirement to generate a target segmentation label combination; Assign an abnormal identifier to the segmentation label combination determined not to meet the geometric continuity requirement to generate a non-target segmentation label combination, and calculate the reflection intensity difference of the adjacent pixel pairs corresponding to the non-target segmentation label combination, and generate a comprehensive difference value in combination with the preset channel weight coefficient; Associate and store the target segmentation label combination, the non-target segmentation label combination, the comprehensive difference value, the position coordinates of the adjacent pixel pairs corresponding to the segmentation label combination, and the preset channel weight coefficient to generate a multi-spectral feature association dataset including a verification identifier. The verification identifier includes a valid identifier and an abnormal identifier.

6. The method according to claim 1, wherein According to the comparison result between the geometric feature segmentation deviation between different multi-spectral bands in the multi-spectral feature association dataset and the preset tolerance range, perform quality inspection on the real-scene three-dimensional model to obtain a quality inspection result. The quality inspection result includes abnormal areas of geometric structure fracture or geometric structure distortion, including: Define a hierarchical difference threshold corresponding to the resolution level and scene type of the real-scene three-dimensional model based on the preset tolerance range; Compare the comprehensive difference value in the multi-spectral feature association dataset with the hierarchical difference threshold to screen out target adjacent pixel pairs; Mark the target adjacent pixel pairs as areas to be confirmed as abnormal, and map the position coordinates of the areas to be confirmed as abnormal to the surface triangular mesh of the real-scene three-dimensional model to perform a spatial adjacency traversal operation on the triangular patches corresponding to the areas to be confirmed as abnormal, and merge all triangular patches sharing edges and vertices into candidate abnormal areas; Determine the final abnormal area according to the comparison result between the sum of the comprehensive difference values of all target adjacent pixel pairs in the candidate abnormal area and the preset area tolerance value; Associate and store the position coordinates, coverage range, combined segmentation label type, and total comprehensive difference value of the final abnormal region to generate a quality inspection result, where the quality inspection result includes abnormal regions with geometric structure fractures or geometric structure distortions.

7. The method according to claim 1, characterized in that, Optimize the multi-spectral imaging data to generate standardized input data, including: Convert the data of each multi-spectral band in the multi-spectral imaging data to a preset frequency domain space through a dynamic optimization processing node to obtain a target frequency domain space. Based on the frequency distribution characteristics of the target frequency domain space, perform noise suppression processing on the multi-spectral imaging data to generate first target imaging data, where the computing resource allocation strategy of the dynamic optimization processing node is dynamically adjusted according to the data volume of the multi-spectral imaging data and the average scene illumination intensity; Adjust the pixel brightness values of the first target imaging data according to the illumination intensity distribution in the local region of the first target imaging data to generate second target imaging data; Perform a spatial normalization operation on the second target imaging data to generate third target imaging data; According to a preset numerical range, linearly scale the pixel values of each multi-spectral band in the third target imaging data to generate standardized input data adapted to the deep learning network.

8. A real - scene three - dimensional model quality inspection system integrating deep learning, characterized in that, Including: An acquisition module for acquiring multi-spectral imaging data of a target scene and constructing a real-scene three-dimensional model based on the multi-spectral imaging data; An optimization module for optimizing the multi-spectral imaging data to generate standardized input data; An extraction module for extracting the correlation features between multi-spectral bands based on the standardized input data, and combining the triangular mesh connection relationship of the real-scene three-dimensional model to generate a pixel-level segmentation result; A verification module for verifying the pixel-level segmentation result based on preset topological continuity rules to generate a multi-spectral feature correlation data set; An inspection module for performing a quality inspection on the real-scene three-dimensional model according to the comparison result between the geometric feature segmentation deviation between different multi-spectral bands in the multi-spectral feature correlation data set and a preset tolerance range to obtain a quality inspection result, where the quality inspection result includes abnormal regions with geometric structure fractures or geometric structure distortions.

9. A computing device, characterized in that, Including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a method for quality inspection of a real-scene three-dimensional model integrating deep learning as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, Stores a computer program, and when the computer program is executed by the computer, it implements a method for quality inspection of a real-scene three-dimensional model integrating deep learning as described in any one of claims 1 to 7.

Images