Structural anomaly detection method and device based on micro-renderable alignment, equipment and medium

Abstract

The invention discloses a structure anomaly detection method, device and equipment based on micro rendering alignment and a medium, and relates to the field of computer vision and industrial detection. The method comprises the following steps: converting a texture-free three-dimensional reference model into three-dimensional Gaussian representation capable of being micro-rendered, and carrying out pose initialization and refined alignment on a single RGB query image based on mask shape matching; rendering and generating a reference depth map consistent with the view angle of the query image under the aligned pose; and analyzing the structural difference between the RGB image and the reference depth map through a multi-modal network to realize anomaly discrimination and positioning. According to the method, the dependence of a traditional method on strict registration of textures, fixed visual angles and multi-modal data is overcome, accurate and robust structural anomaly detection can be realized under the conditions of no textures and unknown poses, and the automation degree and reliability of industrial quality inspection are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and industrial inspection, and in particular to a method, apparatus, device and medium for detecting structural anomalies based on differentiable rendering alignment. Background Technology

[0002] Detection of structural anomalies in industrial products is a crucial step in ensuring quality and safety in manufacturing. Its goal is to automatically identify components on the production line that have defects such as missing parts, damage, misassembly, or deformation. Traditional manual visual inspection methods are inefficient, inconsistent, and costly. Therefore, automated inspection technology based on computer vision has become a development trend in this field.

[0003] Currently, vision-based anomaly detection methods can be mainly categorized as follows: First, methods based on single-modal images, such as relying on RGB image feature embedding for comparison or using generative models to learn the reconstruction capabilities of normal samples, identifying anomalies through reconstruction errors. The core limitation of this type of method lies in its high dependence on the surface texture, color, and other appearance features of objects. Its detection performance significantly decreases for industrial parts without texture or with uncontrollable texture (such as solid-color plastic parts and metal structural parts). Second, methods based on multimodal fusion attempt to combine RGB, depth, and even point cloud information to improve accuracy through feature complementarity. However, these methods typically require precise spatial registration of the data across different modalities. In real-world industrial scenarios, physical misalignment caused by asynchronous acquisition devices and viewing angles introduces significant interference, leading to false detections. Third, in recent years, methods based on pose estimation or multi-view alignment have emerged to address the challenges of changing viewing angles. These methods either rely on rich texture features for matching or struggle to establish reliable 2D-3D correspondences under textureless conditions, and their high computational complexity makes them difficult to meet real-time requirements.

[0004] In summary, existing technologies have fundamental flaws when facing highly challenging industrial scenarios such as textureless reference models, unknown poses of single RGB query images, and limited training data: either they cannot distinguish between real structural anomalies and appearance differences caused by changes in viewpoint and texture due to the lack of reliable spatial geometric alignment benchmarks; or they rely on impractical multi-view acquisition or high-cost sensors to achieve alignment, leading to deployment difficulties.

[0005] Therefore, there is an urgent need for a new technical solution that can fundamentally establish an accurate and robust geometric correspondence between the query image and the standard model, and achieve texture-independent and pose-robust structural anomaly detection on this physically consistent benchmark. Summary of the Invention

[0006] The technical problem to be solved by this invention is: how to achieve accurate structural anomaly detection that is not affected by texture differences and viewpoint changes, given only a single RGB query image and a textureless 3D reference model.

[0007] In a first aspect, embodiments of the present invention provide a structural anomaly detection method based on differentiable rendering alignment, comprising: Obtain the RGB query image to be detected and the textureless 3D reference model; The three-dimensional reference model is converted into a three-dimensional Gaussian distribution representation that supports differentiable rendering, and a corresponding reference mask set is generated by rendering according to a preset set of reference poses. Extract the foreground mask from the RGB query image, and select the reference pose that is closest to the RGB query image as the initial pose by calculating the shape matching error between the foreground mask and each mask in the reference mask set. Starting from the initial pose, the rendering mask of the three-dimensional Gaussian distribution under the current pose is obtained through differentiable rendering. With the goal of minimizing the difference between the rendering mask and the foreground mask, the pose parameters are iteratively optimized to obtain the accurate pose that spatially aligns the three-dimensional reference model with the RGB query image. Under the precise pose, the three-dimensional Gaussian distribution representation is subjected to differentiable depth rendering to generate a reference depth map that is strictly aligned with the viewpoint of the RGB query image; The RGB query image and the reference depth map are input into a multimodal anomaly detection network to extract multi-scale features and calculate cross-modal feature similarity. The network is then fused to generate an anomaly score map representing structural differences.

[0008] Secondly, embodiments of the present invention also provide a structural anomaly detection device based on differentiable rendering alignment, comprising: The data acquisition module is used to acquire the RGB query image to be detected and the textureless 3D reference model; The differentiable representation construction module is used to convert the three-dimensional reference model into a three-dimensional Gaussian distribution representation that supports differentiable rendering, and to render and generate the corresponding reference mask set according to the preset reference pose set. The pose initialization module is used to extract the foreground mask from the RGB query image, and by calculating the shape matching error between the foreground mask and each mask in the reference mask set, select the reference pose that is closest to the RGB query image as the initial pose. The pose optimization module is used to obtain the rendering mask of the three-dimensional Gaussian distribution under the current pose through differentiable rendering, starting from the initial pose, and iteratively optimize the pose parameters with the goal of minimizing the difference between the rendering mask and the foreground mask, so as to obtain the accurate pose that spatially aligns the three-dimensional reference model with the RGB query image. The depth rendering module is used to perform differentiable depth rendering on the three-dimensional Gaussian distribution representation under the precise pose, and generate a reference depth map that is strictly aligned with the viewpoint of the RGB query image. An anomaly detection module is used to input the RGB query image and the reference depth map into a multimodal anomaly detection network, extract multi-scale features and calculate cross-modal feature similarity, and fuse them to generate an anomaly score map representing structural differences.

[0009] Thirdly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described structural anomaly detection method based on differentiable rendering alignment.

[0010] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described structural anomaly detection method based on differentiable rendering alignment.

[0011] The technical solution provided by this invention converts a textureless 3D reference model into a differentiable renderable geometric representation and achieves coarse-to-fine pose alignment based on mask shape matching, thereby generating a reference depth map with strictly consistent viewpoints. Finally, it uses cross-modal feature differences for anomaly detection, achieving high-precision spatial alignment under textureless, single-viewpoint, and unknown pose conditions. This fundamentally eliminates interference introduced by viewpoint changes and provides a physically consistent geometric benchmark for detection. The pure reference depth map obtained through differentiable rendering forms a naturally registered cross-modal input with the query RGB image, effectively solving the false detection problem caused by inconsistent data sources in traditional multimodal methods. The entire detection process is based entirely on geometric consistency, eliminating the dependence on surface texture, large-scale anomaly sample training, and expensive 3D acquisition equipment. While improving detection robustness and accuracy, it significantly reduces system deployment costs and data volume requirements, making it particularly suitable for high-reliability structural anomaly detection in complex industrial environments. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the structural anomaly detection method based on differentiable rendering alignment provided in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the specific process of the structural anomaly detection method based on differentiable rendering alignment provided in an embodiment of the present invention; Figure 3 An anomaly detection result diagram provided in an embodiment of the present invention; Figure 4 This is a diagram showing the pose optimization results provided in an embodiment of the present invention; Figure 5 A schematic diagram of a structural anomaly detection device based on differentiable rendering alignment provided in an embodiment of the present invention; Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0016] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0017] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0018] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0019] like Figure 1 , Figure 2 As shown, this embodiment of the invention provides a structural anomaly detection method based on differentiable rendering alignment, including steps S1-S6.

[0020] S1. Obtain the RGB query image to be detected and the textureless 3D reference model.

[0021] In this step, the acquired data includes a single RGB query image captured by a standard camera, which may contain component anomalies, and a textureless 3D reference prototype (such as a CAD mesh) created based on the object instance. Additionally, as optional input, it may include textureless reference images rendered from this 3D reference model at, for example, 20 preset standard camera poses, along with their corresponding reference pose data, such as... Figure 2 The input of the original multimodal data is shown in the figure.

[0022] S2. Convert the three-dimensional reference model into a three-dimensional Gaussian distribution representation that supports differentiable rendering, and generate a corresponding reference mask set according to a preset set of reference poses.

[0023] In a specific embodiment, converting the three-dimensional reference model into a three-dimensional Gaussian distribution representation that supports differentiable rendering specifically includes: The mesh data of the three-dimensional reference model is converted into a set of multiple three-dimensional Gaussian tuples, where each three-dimensional Gaussian tuple parameterizes its position, shape, color, and opacity in three-dimensional space.

[0024] The core objective of this step is to construct a structurally consistent, differentiable, and renderable standard geometric model, laying the foundation for subsequent pose alignment. Specifically, methods such as Mesh2Splat are used to convert the input triangular mesh model into a 3D Gaussian field representation. This representation contains a large number of 3D Gaussian distributions. ,in Represents a three-dimensional spatial location. Let covariance matrix be the variance matrix. For color information, Opacity. This representation supports fast, differentiable generation of render maps and depth maps from any viewpoint. Subsequently, using a known set of reference camera poses, a corresponding binary reference mask set is generated through differentiable rendering. The generation of the render mask follows the principles of volume rendering, and the cumulative color (or opacity) of a pixel is calculated using the following formula: in The pixel color value accumulated along the ray is N, which represents the set of all three-dimensional Gaussians that intersect with the ray. It is the color of the i-th Gaussian. This represents the cumulative transmittance along the light ray. A binary mask is obtained by determining whether the cumulative opacity of a pixel exceeds a threshold. This step eliminates texture interference at the source and establishes a physically consistent geometric basis that can be used for downstream shape matching.

[0025] S3. Extract the foreground mask from the RGB query image, and select the reference pose that is closest to the RGB query image as the initial pose by calculating the shape matching error between the foreground mask and each mask in the reference mask set.

[0026] In a specific embodiment, a foreground mask is extracted from the RGB query image, and the shape matching error between the foreground mask and each mask in the reference mask set is calculated to select the reference pose that is closest to the RGB query image as the initial pose, specifically including steps S31-S33.

[0027] S31. Extract the foreground mask from the RGB query image.

[0028] In this step, foreground segmentation is performed on the acquired RGB query image to obtain the binary shape mask of the object. Simple methods such as thresholding and background subtraction can be used, with the core objective of providing the most stable geometric contour cues and completely avoiding dependence on texture features.

[0029] S32. Calculate the chamfer distance between the foreground mask and each reference mask in the reference mask set.

[0030] In this step, to achieve initial pose estimation under textureless conditions, chamfer distance is used as a shape consistency metric. The chamfer distance is defined as: in and These represent the sets of foreground pixel coordinates in image space for the query mask and the reference mask, respectively. This distance measures the degree of matching between the two mask contours.

[0031] S33. Determine the reference pose corresponding to the minimum chamfer distance as the initial pose.

[0032] In this step, the reference mask set is traversed and its relationship with the query mask is calculated. The chamfer distance is used to select the camera pose corresponding to the reference mask that minimizes the distance as the initial pose. .like Figure 4 As shown in the pose initialization section on the left, this method solves the problem of difficult feature point matching under textureless conditions, providing a reliable starting point for subsequent fine optimization.

[0033] S4. Starting from the initial pose, obtain the rendering mask of the three-dimensional Gaussian distribution under the current pose through differentiable rendering, and iteratively optimize the pose parameters with the goal of minimizing the difference between the rendering mask and the foreground mask, to obtain the accurate pose that spatially aligns the three-dimensional reference model with the RGB query image.

[0034] In a specific embodiment, with the goal of minimizing the difference between the rendering mask and the foreground mask, the pose parameters are iteratively optimized, specifically including steps S41-S42.

[0035] S41. Define the L1 distance between the rendering mask and the foreground mask as the pose optimization loss.

[0036] In this step, in the initial pose Next, differentiable rendering is performed on the 3D Gaussian representation to obtain the rendering mask. Define pose optimization loss as This loss function depends only on the shape difference of the binary mask and does not involve variables such as texture and lighting, thus having a natural advantage in textureless scenes.

[0037] S42. Perform gradient backpropagation using the three-dimensional Gaussian distribution-represented differentiable renderer, calculate the gradient of the pose optimization loss with respect to the pose parameters, and iteratively update the pose parameters using the gradient descent method until convergence.

[0038] In this step, to achieve high-precision, differentiable pose optimization, the pose transformation is parameterized as a tangent vector in the Lie algebra space. ,in Indicates the axis of rotation. Represents the translation component. This matrix is ​​derived from the rotation matrix. Combined with the translation vector t, its expression is: The corresponding transformation matrix It can be calculated using the Rodriguez formula: In the formula It is by Constructed antisymmetric matrix, Let be the rotation angle. The complete transformation matrix T can be written as: The Jacobian matrix can be represented as: Because the 3D Gaussian renderer is differentiable, the loss Regarding pose parameters The gradient can be backpropagated using the chain rule. The system uses gradient descent (such as the Adam optimizer) for iterative updates, minimizing... Typically, convergence occurs after a certain number of iterations (e.g., 150), yielding the precise alignment pose. This process is as follows: Figure 4 As shown in the pose optimization section, it utilizes gradient information from differentiable rendering to achieve pixel-level precise alignment, fundamentally eliminating viewpoint differences.

[0039] S5. Under the precise pose, perform differentiable depth rendering on the three-dimensional Gaussian distribution representation to generate a reference depth map that is strictly aligned with the viewpoint of the RGB query image.

[0040] In this step, the precise pose is obtained. Then, with the camera fixed, differentiable depth rendering is performed on the same 3D Gaussian representation to generate a reference depth map. The depth rendering formula is: in This represents the depth value (z-coordinate) of the i-th Gaussian center on the current pixel ray. The resulting depth map... The structure is stable, noise-free, and free of texture interference, realistically reflecting the three-dimensional form of an object under ideal conditions, and is strictly aligned with the RGB query image in terms of viewpoint, such as... Figure 2 and Figure 4 As shown on the right, a high-quality, physically consistent geometric reference mode is provided for subsequent anomaly detection.

[0041] S6. Input the RGB query image and the reference depth map into the multimodal anomaly detection network, extract multi-scale features and calculate cross-modal feature similarity, and fuse them to generate an anomaly score map representing structural differences.

[0042] In a specific embodiment, the RGB query image and the reference depth map are input into a multimodal anomaly detection network to extract multi-scale features and calculate cross-modal feature similarity, and then fused to generate an anomaly score map representing structural differences, specifically including steps S61-S63.

[0043] S61. A dual-branch encoder with shared weights from a multimodal anomaly detection network is used to extract feature maps of the RGB query image and the reference depth map at multiple different resolution scales.

[0044] In this step, a strictly aligned rendered depth map and an RGB query image are input into a multimodal pyramid anomaly detection network. This network employs a dual-branch encoder with shared weights (e.g., based on ResNet18) to extract multi-scale features from both the RGB and depth modalities. The shared weights enable the network to learn a unified feature embedding across modalities, allowing structural differences to be explicitly captured.

[0045] S62. For each scale of the feature map, calculate the cosine similarity between the corresponding position vectors of the query image features and the reference depth map features to obtain the preliminary difference map for each scale.

[0046] In this step, at each feature scale, the cosine similarity between the reference depth features and the query image features is calculated: The difference diagram Each pixel value in the graph represents the degree of inconsistency between the appearance features and the geometric reference features at that location; a higher value indicates a greater likelihood of an anomaly. , The encoder extracts features from the query image and the i-th layer of the reference depth map.

[0047] S63. Upsample and fuse the preliminary difference maps at multiple scales to generate the anomaly score map.

[0048] In this step, the feature pyramid concept is employed to upsample the difference maps at different scales to the original image size, and then perform weighted fusion to obtain a high-resolution fused anomaly score map. This multi-scale fusion design enables the network to effectively detect both large-area anomalies (obvious in deep features) and small-scale, fine-grained anomalies (obvious in shallow features), such as... Figure 2 The multimodal anomaly detection section is shown. In one specific embodiment, the structural anomaly detection method based on differentiable rendering alignment further includes the following steps: S70. Based on the anomaly score map, output the anomaly detection result and the location information of the anomaly region of the RGB query image.

[0049] In a specific embodiment, the step of outputting the anomaly judgment result and the location information of the abnormal region of the RGB query image based on the anomaly score map specifically includes steps S71-S72.

[0050] S71. Perform global pooling on the anomaly score map and input it into the classifier to obtain the normal or anomaly classification results at the sample level.

[0051] In this embodiment, global average pooling is applied to the anomaly score map to obtain a global anomaly score scalar. This scalar is then input into a two-layer multilayer perceptron classification head, which outputs a binary classification result (normal / abnormal), thus completing image-level anomaly detection.

[0052] S72. Based on the anomaly score map or the fusion features extracted by the multimodal anomaly detection network, regress to generate the bounding box coordinates of the anomaly region.

[0053] In this embodiment, the anomaly score map or multi-scale fused features can be input into a four-layer multilayer perceptron localization head. This network learns to directly regress the bounding box coordinates of the anomaly region (such as the center point x, y and width and height w, h), achieving pixel-level anomaly region localization. The final anomaly detection and localization visualization results can be found in [reference needed]. Figure 3 The system demonstrates its ability to detect various structural anomalies, including rotation, displacement, breakage, missing parts, and shifts.

[0054] In summary, the present invention effectively solves the defects existing in the prior art through the above technical solutions: 1. This invention addresses the lack of reliable geometric references in traditional anomaly detection: Existing anomaly detection methods primarily rely on texture, color, or local appearance in images to identify anomalies. This invention provides a stable, controllable, and physically meaningful geometric reference mode by converting a standard 3D reference model into a differentiable 3D Gaussian representation and rendering a high-fidelity, texture-free, and illumination-free depth map with the final aligned pose. This allows the detection process to revolve around the actual structure of the object.

[0055] 2. This invention addresses the false detection problem caused by inconsistencies in cross-modal information: Traditional multimodal fusion methods often suffer from heterogeneous acquisition devices, asynchronous viewpoints, and inconsistent coordinate systems, resulting in physical mismatches in cross-modal data and a high risk of misjudgment. This invention ensures strict alignment between the standard 3D model and the input image from the same viewpoint through differentiable pose optimization. Then, a geometric depth map is rendered based on this aligned pose, ensuring that the rendered depth naturally lies in the same projected coordinate system as the query image. Subsequently, this invention utilizes a cross-modal pyramid network for feature difference analysis, thereby guaranteeing that the structural differences learned by the model are genuine defects, rather than modal contradictions or device errors. This mechanism fundamentally solves the problem of inconsistencies in cross-modal data in existing technologies.

[0056] 3. This invention overcomes the heavy reliance of existing technologies on texture features: Most anomaly detection algorithms depend on high-frequency textures, feature points, or pattern contrasts on object surfaces. The method of this invention is entirely based on rendered depth geometry as a reference, independent of texture, color, or lighting differences, and achieves anomaly detection through cross-modal structural difference learning. Since depth modalities are inherently texture-independent, this invention maintains stable performance in scenes with no texture, extremely weak texture, or uncontrollable texture, achieving truly texture-independent anomaly detection—something difficult to achieve with existing 2D methods.

[0057] 4. Eliminates reliance on multi-view, high-cost depth sensors: Many existing 3D anomaly detection methods rely on multi-view acquisition, binocular systems, or depth cameras, resulting in high deployment costs and significant hardware complexity. This invention relies on only a single image input, and through micro-pose optimization, it can align the 3D model to the actual shooting viewpoint and render a consistent depth reference modality, enabling the system to simulate 3D detection capabilities under single-view conditions.

[0058] 5. Significantly reduces reliance on large-scale training data: Traditional deep learning anomaly detection models often require a large number of normal samples for modeling, and some methods require anomaly samples for supervision, resulting in extremely high training costs. This invention, relying on a 3D model as the structural benchmark, eliminates the need for the model to learn normal structures through massive amounts of data. Instead, it directly obtains structural consistency cues through geometric alignment and cross-modal differences. This invention achieves high-quality anomaly detection performance even with scarce training data, demonstrating strong data efficiency. This has significant practical value for real-world industrial scenarios, as industrial sample collection is costly and time-consuming.

[0059] The structural anomaly detection method based on differentiable rendering alignment proposed in this invention can achieve the following substantial technical effects, thereby significantly improving the stability, accuracy, and generalization ability of structural anomaly detection: 1. Achieve precise pose alignment This invention achieves high-precision alignment between a 3D reference model and the actual shooting perspective under the constraint of a single query image through differentiable pose optimization. This process utilizes rendering gradients to directly minimize projection errors in the 2D pixel space, resulting in high-precision pose estimation. Through this mechanism, this invention successfully establishes a strict spatial correspondence between the query image and standard 3D geometry, providing a reliable benchmark for subsequent geometric consistency analysis and anomaly detection.

[0060] 2. Provides high-quality geometric references with consistent perspective. Under the optimized pose, this invention utilizes differentiable rendering technology to render a depth map of the optimized viewpoint. Since this depth map is directly rendered from a 3D Gaussian representation, its structure is stable, noise-free, and free of texture interference, realistically reflecting the 3D morphology of an object in an ideal state. The depth reference generated by this invention is physically and strictly aligned with the query image, thus providing a highly reliable geometric benchmark for cross-modal structural comparison.

[0061] 3. Precise anomaly detection effect The cross-modal pyramid detection network constructed in this invention can simultaneously receive rendered depth maps and RGB query images, and learn the general structural embedding between them through multi-scale feature fusion. The network further locates anomalous regions in the difference space through cross-modal similarity mapping, achieving detailed detection of structural anomalies such as missing, misaligned, and deformed structures. Simultaneously, this model maintains stable and good detection performance even with sparse training data, significantly outperforming traditional anomaly detection methods that rely on large-scale data.

[0062] This invention proposes a multimodal structural anomaly detection method based on single-view pose alignment, differentiable rendering, and cross-modal structural similarity learning. This method is independent of texture features, robust to changes in shooting perspective, and can complete structural alignment and anomaly localization under the condition of a single RGB image and a standard 3D reference model as input. This method is particularly suitable for practical industrial scenarios with stable structures, unknown textures, unknown poses, and limited training data. The main application areas of this invention include, but are not limited to, the following: 1. Industrial Automated Quality Inspection: This is the most direct and core application area of ​​this invention. This invention utilizes a standard 3D model to provide a stable geometric reference, and through differentiable pose alignment and cross-modal structural difference learning, it achieves automated detection of structural anomalies such as missing, misaligned, replaced, offset, and deformed industrial parts. The applicable scope includes: 1) Manufacturing industry: Used for machining and component manufacturing, including inspecting injection molded parts, die-cast parts, structural parts, metal parts, etc. for local deformation, assembly misalignment or missing key parts.

[0063] 2) Electronic product manufacturing: Applicable to the assembly consistency inspection of non-textured parts such as plastic skeletons, module support parts, and internal structural parts, which can avoid false detection and missed detection caused by changes in viewing angle.

[0064] 3) Assembly line: can be used to detect positional errors and incomplete assembly problems of tooling fixtures, structural supports, and installation modules during the assembly process.

[0065] 2. General Multimodal Structural Anomaly Detection for Weakly Textured or Textureless Objects: Traditional image anomaly detection methods heavily rely on texture, local patterns, or high-frequency appearance details, while this invention is entirely based on geometric structural consistency, and therefore applicable to: 1) Textureless geometry detection (such as plastic brackets, metal): This invention provides accurate geometric reference through rendering depth, and can maintain stable detection performance on smooth, reflective, and monochromatic surfaces.

[0066] 2) For objects whose appearance changes significantly with the shooting angle but whose structure remains stable, this invention eliminates the difference in perspective through posture alignment, thereby improving the reliability of detection.

[0067] 3. Robot Vision and Intelligent Sorting: This invention provides structural standards through rendering depth and utilizes cross-modal networks to locate abnormal areas, making it suitable for structural verification tasks in robot vision and automated operations. This includes determining whether components are in place, missing, or exhibiting posture deviations, assisting robots in accurate grasping or assembly operations. Based on the abnormal location information provided by this invention, robots can automatically sort structurally abnormal samples off the production line, improving the level of automation.

[0068] 4. Digital Twin and Online Consistency Detection: In the application of digital twins in Industry 4.0, this invention can be used to compare real-time images with differentiable 3D Gaussian rendering depth across modalities to determine structural deviations or assembly errors; it can also be used to detect in real time on the production line whether products are assembled according to the design state, and is suitable for process quality control.

[0069] like Figure 5 As shown, this embodiment of the invention also provides a structural anomaly detection device based on differentiable rendering alignment, comprising: The data acquisition module 10 is used to acquire the RGB query image to be detected and the textureless 3D reference model; Differentiable representation construction module 20 is used to convert the three-dimensional reference model into a three-dimensional Gaussian distribution representation that supports differentiable rendering, and to render and generate a corresponding reference mask set according to a preset set of reference poses. The pose initialization module 30 is used to extract a foreground mask from the RGB query image, and to select the reference pose that is closest to the RGB query image as the initial pose by calculating the shape matching error between the foreground mask and each mask in the reference mask set. The pose optimization module 40 is used to obtain the rendering mask of the three-dimensional Gaussian distribution under the current pose through differentiable rendering, starting from the initial pose, and iteratively optimize the pose parameters with the goal of minimizing the difference between the rendering mask and the foreground mask, so as to obtain the accurate pose that spatially aligns the three-dimensional reference model with the RGB query image. The depth rendering module 50 is used to perform differentiable depth rendering on the three-dimensional Gaussian distribution representation under the precise pose, and generate a reference depth map that is strictly aligned with the viewpoint of the RGB query image. Anomaly detection module 60 is used to input the RGB query image and the reference depth map into a multimodal anomaly detection network, extract multi-scale features and calculate cross-modal feature similarity, and fuse them to generate an anomaly score map representing structural differences.

[0070] In a specific embodiment, the differentiable representation construction module 20 is specifically used for: The mesh data of the three-dimensional reference model is converted into a set of multiple three-dimensional Gaussian tuples, where each three-dimensional Gaussian tuple parameterizes its position, shape, color, and opacity in three-dimensional space.

[0071] In a specific embodiment, the pose initialization module 30 is specifically used for: Extract the foreground mask from the RGB query image; Calculate the chamfer distance between the foreground mask and each reference mask in the reference mask set; The reference pose corresponding to the minimum chamfer distance is determined as the initial pose.

[0072] In a specific embodiment, the pose optimization module 40 is specifically used for: The L1 distance between the rendering mask and the foreground mask is defined as the pose optimization loss; Gradient backpropagation is performed using the differentiable renderer represented by the three-dimensional Gaussian distribution to calculate the gradient of the pose optimization loss with respect to the pose parameters, and the pose parameters are iteratively updated using the gradient descent method until convergence.

[0073] In a specific embodiment, the anomaly detection module 60 is specifically used for: A dual-branch encoder with shared weights in a multimodal anomaly detection network is used to extract feature maps of the RGB query image and the reference depth map at multiple different resolution scales. For each scale of the feature map, the cosine similarity between the corresponding position vectors of the query image features and the reference depth map features is calculated to obtain the preliminary difference map for each scale. The preliminary difference maps at multiple scales are upsampled and fused to generate the anomaly score map.

[0074] In a specific embodiment, it also includes: The result output module outputs the anomaly detection result and the location information of the abnormal region of the RGB query image based on the anomaly score map.

[0075] In a specific embodiment, the result output module is specifically used for: The anomaly score map is globally pooled and input into a classifier to obtain normal or anomaly classification results at the sample level; Based on the anomaly score map or the fusion features extracted by the multimodal anomaly detection network, the bounding box coordinates of the anomaly region are regressed and generated.

[0076] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned structural anomaly detection device based on differentiable rendering alignment and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0077] The aforementioned structural anomaly detection device based on differentiable rendering alignment can be implemented as a computer program, which can, for example... Figure 6 It runs on the computer device shown.

[0078] Please see Figure 6 , Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0079] See Figure 6 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0080] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to execute a structural anomaly detection method based on differentiable rendering alignment.

[0081] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0082] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a structural anomaly detection method based on differentiable rendering alignment.

[0083] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0084] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps: S1. Obtain the RGB query image to be detected and the textureless 3D reference model; S2. Convert the three-dimensional reference model into a three-dimensional Gaussian distribution representation that supports differentiable rendering, and generate a corresponding reference mask set according to a preset set of reference poses. S3. Extract the foreground mask from the RGB query image, and by calculating the shape matching error between the foreground mask and each mask in the reference mask set, select the reference pose that is closest to the RGB query image as the initial pose. S4. Starting from the initial pose, obtain the rendering mask of the three-dimensional Gaussian distribution under the current pose through differentiable rendering, and iteratively optimize the pose parameters with the goal of minimizing the difference between the rendering mask and the foreground mask, to obtain the accurate pose that spatially aligns the three-dimensional reference model with the RGB query image. S5. Under the precise pose, perform differentiable depth rendering on the three-dimensional Gaussian distribution representation to generate a reference depth map that is strictly aligned with the viewpoint of the RGB query image. S6. Input the RGB query image and the reference depth map into the multimodal anomaly detection network, extract multi-scale features and calculate cross-modal feature similarity, and fuse them to generate an anomaly score map representing structural differences.

[0085] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0086] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0087] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the following steps: S1. Obtain the RGB query image to be detected and the textureless 3D reference model; S2. Convert the three-dimensional reference model into a three-dimensional Gaussian distribution representation that supports differentiable rendering, and generate a corresponding reference mask set according to a preset set of reference poses. S3. Extract the foreground mask from the RGB query image, and by calculating the shape matching error between the foreground mask and each mask in the reference mask set, select the reference pose that is closest to the RGB query image as the initial pose. S4. Starting from the initial pose, obtain the rendering mask of the three-dimensional Gaussian distribution under the current pose through differentiable rendering, and iteratively optimize the pose parameters with the goal of minimizing the difference between the rendering mask and the foreground mask, to obtain the accurate pose that spatially aligns the three-dimensional reference model with the RGB query image. S5. Under the precise pose, perform differentiable depth rendering on the three-dimensional Gaussian distribution representation to generate a reference depth map that is strictly aligned with the viewpoint of the RGB query image. S6. Input the RGB query image and the reference depth map into the multimodal anomaly detection network, extract multi-scale features and calculate cross-modal feature similarity, and fuse them to generate an anomaly score map representing structural differences.

[0088] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code.

[0089] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0090] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0091] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0092] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0093] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0094] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0095] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A structural anomaly detection method based on differentiable rendering alignment, characterized in that, include: Obtain the RGB query image to be detected and the textureless 3D reference model; The three-dimensional reference model is converted into a three-dimensional Gaussian distribution representation that supports differentiable rendering, and a corresponding reference mask set is generated by rendering according to a preset set of reference poses. Extract the foreground mask from the RGB query image, and select the reference pose that is closest to the RGB query image as the initial pose by calculating the shape matching error between the foreground mask and each mask in the reference mask set. Starting from the initial pose, the rendering mask of the three-dimensional Gaussian distribution under the current pose is obtained through differentiable rendering. With the goal of minimizing the difference between the rendering mask and the foreground mask, the pose parameters are iteratively optimized to obtain the accurate pose that spatially aligns the three-dimensional reference model with the RGB query image. Under the precise pose, the three-dimensional Gaussian distribution representation is subjected to differentiable depth rendering to generate a reference depth map that is strictly aligned with the viewpoint of the RGB query image; The RGB query image and the reference depth map are input into a multimodal anomaly detection network to extract multi-scale features and calculate cross-modal feature similarity. The network is then fused to generate an anomaly score map representing structural differences.

2. The structural anomaly detection method based on differentiable rendering alignment according to claim 1, characterized in that, Converting the 3D reference model into a 3D Gaussian distribution representation that supports differentiable rendering specifically includes: The mesh data of the three-dimensional reference model is converted into a set of multiple three-dimensional Gaussian tuples, where each three-dimensional Gaussian tuple parametrically represents its position, shape, color, and opacity in three-dimensional space.

3. The structural anomaly detection method based on differentiable rendering alignment according to claim 1, characterized in that, Extracting a foreground mask from the RGB query image, and calculating the shape matching error between the foreground mask and each mask in the reference mask set, selecting the reference pose closest to the RGB query image as the initial pose, specifically including: Extract the foreground mask from the RGB query image; Calculate the chamfer distance between the foreground mask and each reference mask in the reference mask set; The reference pose corresponding to the minimum chamfer distance is determined as the initial pose.

4. The structural anomaly detection method based on differentiable rendering alignment according to claim 1, characterized in that, With the goal of minimizing the difference between the rendering mask and the foreground mask, the pose parameters are iteratively optimized, specifically including: The L1 distance between the rendering mask and the foreground mask is defined as the pose optimization loss; Gradient backpropagation is performed using the differentiable renderer represented by the three-dimensional Gaussian distribution to calculate the gradient of the pose optimization loss with respect to the pose parameters, and the pose parameters are iteratively updated using the gradient descent method until convergence.

5. The structural anomaly detection method based on differentiable rendering alignment according to claim 1, characterized in that, The RGB query image and the reference depth map are input into a multimodal anomaly detection network to extract multi-scale features and calculate cross-modal feature similarity. These are then fused to generate an anomaly score map representing structural differences. Specifically, this includes: A dual-branch encoder with shared weights in a multimodal anomaly detection network is used to extract feature maps of the RGB query image and the reference depth map at multiple different resolution scales. For each scale of the feature map, the cosine similarity between the corresponding position vectors of the query image features and the reference depth map features is calculated to obtain the preliminary difference map for each scale. The preliminary difference maps at multiple scales are upsampled and fused to generate the anomaly score map.

6. The structural anomaly detection method based on differentiable rendering alignment according to claim 1, characterized in that, Also includes: Based on the anomaly score map, output the anomaly detection result and the location information of the anomaly region of the RGB query image.

7. The structural anomaly detection method based on differentiable rendering alignment according to claim 6, characterized in that, The step of outputting the anomaly detection result and anomaly region location information of the RGB query image based on the anomaly score map specifically includes: The anomaly score map is globally pooled and input into a classifier to obtain normal or anomaly classification results at the sample level; Based on the anomaly score map or the fusion features extracted by the multimodal anomaly detection network, the bounding box coordinates of the anomaly region are regressed and generated.

8. A structural anomaly detection device based on differentiable rendering alignment, characterized in that, include: The data acquisition module is used to acquire the RGB query image to be detected and the textureless 3D reference model; The differentiable representation construction module is used to convert the three-dimensional reference model into a three-dimensional Gaussian distribution representation that supports differentiable rendering, and to render and generate the corresponding reference mask set according to the preset reference pose set. The pose initialization module is used to extract the foreground mask from the RGB query image, and by calculating the shape matching error between the foreground mask and each mask in the reference mask set, select the reference pose that is closest to the RGB query image as the initial pose. The pose optimization module is used to obtain the rendering mask of the three-dimensional Gaussian distribution under the current pose through differentiable rendering, starting from the initial pose, and iteratively optimize the pose parameters with the goal of minimizing the difference between the rendering mask and the foreground mask, so as to obtain the accurate pose that spatially aligns the three-dimensional reference model with the RGB query image. The depth rendering module is used to perform differentiable depth rendering on the three-dimensional Gaussian distribution representation under the precise pose, and generate a reference depth map that is strictly aligned with the viewpoint of the RGB query image. An anomaly detection module is used to input the RGB query image and the reference depth map into a multimodal anomaly detection network, extract multi-scale features and calculate cross-modal feature similarity, and fuse them to generate an anomaly score map representing structural differences.

9. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the structural anomaly detection method based on differentiable rendering alignment as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, can implement the structural anomaly detection method based on differentiable rendering alignment as described in any one of claims 1 to 7.

Images