Three-dimensional non-contact pose estimation, defect detection method, system, device

Abstract

This disclosure provides a three-dimensional non-contact pose estimation and defect detection method, system, and device. The method includes: acquiring scene RGB images simultaneously captured from at least two viewpoints of the workpiece under test, and reconstructing a three-dimensional point cloud model of the workpiece based on the scene RGB images; reprojecting the three-dimensional point cloud model onto the image plane of a reference camera to generate a two-dimensional depth map from the reference camera's perspective; stitching the scene RGB images and the two-dimensional depth map to obtain measured multi-source data; generating multiple sets of rendered multi-source data using a differentiable renderer based on the pose parameters to be optimized, and determining the actual pose of the workpiece under test based on the pose parameters to be optimized corresponding to the rendered multi-source data that satisfy the iteration stopping condition. This disclosure achieves high-precision pose estimation of the workpiece under test, and the pose estimation result has excellent robustness, adaptable to complex industrial inspection conditions.

Description

Technical Field

[0001] This disclosure relates to the field of industrial inspection technology, and in particular to a three-dimensional non-contact pose estimation and defect detection method, system, and equipment. Background Technology

[0002] In the process of mass production and quality inspection of industrial products, surface defect detection is a key link in ensuring product quality. Its detection accuracy, efficiency and automation level directly affect the market competitiveness and operational reliability of the products.

[0003] Existing industrial product surface defect detection technologies are mainly divided into two categories: contact and non-contact. Contact detection relies on tools such as mechanical probes for localized measurements, which is slow and inefficient, and cannot meet the cycle time requirements of modern production lines. In non-contact detection, two-dimensional vision detection is widely used due to its simplicity and fast response, but it can only acquire two-dimensional image information and lacks depth data, making it unable to accurately characterize the three-dimensional morphology of defects. This can easily lead to misjudgments and missed detections, affecting the accuracy of detection.

[0004] To overcome the limitations of two-dimensional inspection, three-dimensional defect detection solutions are currently employed. However, existing three-dimensional inspection technologies share common problems: First, insufficient attitude estimation accuracy makes them unsuitable for high-end and low-end automated production environments with high adaptability to attitude uncertainty, resulting in significant pose calculation errors in complex scenarios. Second, adaptability and inspection efficiency are difficult to balance; either relying on high-precision fixtures to fix the workpiece leads to high deployment costs, or fixture-free conditions are prone to registration failures and missed detections, failing to meet the needs of flexible manufacturing. Furthermore, the high-precision inspection requirements of high-end manufacturing are difficult to meet in low-end manufacturing, limiting the promotion of advanced technologies. Summary of the Invention

[0005] The technical problem to be solved by this disclosure is to overcome the shortcomings of existing three-dimensional defect detection technologies, such as insufficient pose estimation accuracy and difficulty in balancing adaptability and detection efficiency, and to provide a three-dimensional non-contact pose estimation and defect detection method, system, and device.

[0006] This disclosure solves the above-mentioned technical problems through the following technical solution:

[0007] Firstly, a three-dimensional non-contact pose estimation method is provided, including:

[0008] Acquire scene RGB images (scene RGB) of the workpiece under test simultaneously captured from at least two perspectives, and reconstruct a three-dimensional point cloud model of the workpiece under test based on the scene RGB images; wherein, the scene RGB images are images captured of the workpiece under test with a fully bright stripe pattern projected on it;

[0009] The three-dimensional point cloud model is reprojected onto the image plane of the reference camera to generate a two-dimensional depth map from the perspective of the reference camera.

[0010] By stitching together the RGB image of the scene and the two-dimensional depth map, measured multi-source data is obtained;

[0011] The actual pose of the workpiece under test is determined by generating multiple sets of multi-source rendering data based on the pose parameters to be optimized using a differentiable renderer, and then determining the pose parameters to be optimized corresponding to the multi-source rendering data that meet the iteration stopping condition.

[0012] Optionally, before the step of reconstructing the three-dimensional point cloud model of the workpiece to be tested, the method further includes: filtering out the background region in the scene RGB image using a measured mask image; wherein the measured mask image is obtained by image segmentation of the scene RGB image;

[0013] And / or, after the step of reconstructing the three-dimensional point cloud model of the workpiece to be tested, the method further includes at least one of the following operations:

[0014] Perform depth bandpass filtering on the 3D point cloud model;

[0015] Remove statistical outliers from the 3D point cloud model;

[0016] Radius nearest neighbor filtering is performed on the three-dimensional point cloud model.

[0017] Optionally, the scene RGB image and the two-dimensional depth map are stitched together to obtain measured multi-source data, including:

[0018] Joint feature encoding is performed on the scene RGB image and the two-dimensional depth map to generate high-dimensional features;

[0019] The high-dimensional features are used to characterize the measured multi-source data.

[0020] Optionally, the pose parameters to be optimized include a translation vector; the first initial value of the translation vector is determined through the following steps:

[0021] The two-dimensional depth map is back-projected onto the camera coordinate system using the intrinsic parameters of the reference camera to obtain the target point cloud;

[0022] The mean of the three-dimensional coordinates of the target point cloud is determined as the first translation vector.

[0023] Optionally, the pose parameters to be optimized include a rotation matrix; the second initial value of the rotation matrix is ​​determined through the following steps:

[0024] The three-dimensional point cloud model is uniformly rotated and sampled around the first initial value of the translation vector to generate multiple pose templates;

[0025] Use a virtual camera to render templates for each pose to obtain the corresponding rendering mask.

[0026] Calculate the similarity between each rendered mask and the measured mask, and use the rotation matrix corresponding to the rendered mask with the highest similarity as the second initial value; wherein, the measured mask is obtained by image segmentation of the scene RGB image.

[0027] Optionally, the pose parameters to be optimized include a translation vector; the first initial value of the translation vector of the (n+1)th frame scene RGB image is determined based on the translation vector contained in the actual pose estimated from the nth frame scene RGB image.

[0028] And / or, the pose parameters to be optimized include a rotation matrix; the second initial value of the rotation matrix of the (n+1)th frame scene RGB image is determined based on the rotation matrix contained in the actual pose estimated from the nth frame scene RGB image.

[0029] Optionally, the rendered multi-source data includes a rendered mask map and a rendered depth map;

[0030] The loss function of the differentiable renderer is a weighted sum of the mask loss and the depth loss;

[0031] Wherein, the mask loss is the intersection-union ratio loss of the measured mask map of the scene RGB image and the rendered mask map; the depth loss is the normalized mean square error of the two-dimensional depth map and the rendered depth map.

[0032] Secondly, a three-dimensional non-contact defect detection method is provided, including:

[0033] Estimate the actual pose of the workpiece to be measured; the actual pose is determined according to the pose estimation method described in any one of the first aspects;

[0034] The defect-free standard 3D model is converted into a reference point cloud model that is identical to the actual pose of the workpiece under test.

[0035] Defect detection is performed on the three-dimensional point cloud model of the workpiece under test based on the reference point cloud model.

[0036] Optionally, before the step of performing defect detection on the three-dimensional point cloud model of the workpiece to be tested based on the reference point cloud model, the method further includes:

[0037] Find the nearest neighbor of the target point in the reference point cloud model and calculate the Euclidean distance between the target point and the nearest neighbor.

[0038] The geometric deviation between the reference point cloud model and the 3D point cloud model is determined based on the Euclidean distance;

[0039] In response to the geometric deviation being less than or equal to the deviation threshold, a step of defect detection is performed on the three-dimensional point cloud model of the workpiece under test based on the reference point cloud model.

[0040] In response to the geometric deviation being greater than the deviation threshold, the step of estimating the actual pose of the workpiece under test is returned.

[0041] Optionally, defect detection is performed on the three-dimensional point cloud model of the workpiece under test based on the reference point cloud model, including:

[0042] The iterative nearest point algorithm is used to perform point cloud registration between the reference point cloud model and the 3D point cloud model to obtain the transformation relationship between the reference point cloud model and the 3D point cloud model;

[0043] The 3D point cloud model is updated according to the transformation relationship, and defects are detected in the updated 3D point cloud model according to the reference point cloud model.

[0044] Thirdly, a three-dimensional non-contact pose estimation system is provided, including:

[0045] The acquisition module is used to acquire scene RGB images of the workpiece under test simultaneously captured from at least two perspectives, and to reconstruct a three-dimensional point cloud model of the workpiece under test based on the scene RGB images; wherein, the scene RGB images are images captured of the workpiece under test with a fully bright stripe pattern projected on it;

[0046] The generation module is used to reproject the three-dimensional point cloud model onto the image plane of the reference camera to generate a two-dimensional depth map from the perspective of the reference camera.

[0047] The stitching module is used to stitch together the scene RGB image and the two-dimensional depth map to obtain measured multi-source data;

[0048] The optimization module is used to generate multiple sets of multi-source rendering data through a differentiable renderer based on the pose parameters to be optimized, and to determine the actual pose of the workpiece to be tested according to the pose parameters to be optimized corresponding to the multi-source rendering data that meet the iteration stopping condition.

[0049] Fourthly, a three-dimensional non-contact defect detection system is provided, comprising:

[0050] An estimation module is used to estimate the actual pose of the workpiece under test; the actual pose is determined according to the pose estimation method described in any one of the first aspects.

[0051] The conversion module is used to convert a defect-free standard 3D model into a reference point cloud model that is identical to the actual pose of the workpiece under test.

[0052] The detection module is used to perform defect detection on the three-dimensional point cloud model of the workpiece to be tested based on the reference point cloud model.

[0053] Fifthly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein the processor executes the computer program to implement the method described in any one of the first aspects.

[0054] In a sixth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any one of the first aspects.

[0055] In a seventh aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method as described in any one of the first aspects.

[0056] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.

[0057] The positive and progressive effects of this disclosure are as follows: This disclosure achieves high-precision pose estimation of the workpiece under test by fusing multimodal data of visual (RGB) information and geometric (depth / point cloud) information, and by combining differentiable rendering technology for iterative optimization. Moreover, the pose estimation result has excellent robustness and can be adapted to complex industrial inspection conditions. Attached Figure Description

[0058] Figure 1 A schematic diagram of the hardware architecture of a three-dimensional non-contact pose estimation system provided for an exemplary embodiment of this disclosure;

[0059] Figure 2 A flowchart of a three-dimensional non-contact pose estimation method provided as an exemplary embodiment of this disclosure;

[0060] Figure 3 An example diagram illustrating the generation of a depth map during a three-dimensional non-contact pose estimation process, provided as an exemplary embodiment of this disclosure.

[0061] Figure 4 An illustration of a differentiable rendering process in a three-dimensional non-contact pose estimation process provided as an exemplary embodiment of this disclosure;

[0062] Figure 5 A flowchart of another three-dimensional non-contact pose estimation method provided as an exemplary embodiment of the present disclosure;

[0063] Figure 6 A flowchart of a three-dimensional non-contact defect detection method provided as an exemplary embodiment of this disclosure;

[0064] Figure 7 An illustration of the effect of a three-dimensional non-contact defect detection result provided for an exemplary embodiment of this disclosure;

[0065] Figure 8a A schematic diagram of a three-dimensional non-contact pose estimation system provided as an exemplary embodiment of the present disclosure;

[0066] Figure 8b A schematic diagram of a three-dimensional non-contact defect detection system provided as an exemplary embodiment of this disclosure;

[0067] Figure 9 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present disclosure. Detailed Implementation

[0068] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0069] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not be construed as an unnecessary limitation. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0070] Non-contact inspection technologies have developed rapidly in recent years, mainly including systems based on laser scanning, fixed binocular vision, and robots equipped with monocular cameras.

[0071] Laser scanning technology acquires high-density point clouds through point-by-point scanning, offering high spatial resolution and enabling three-dimensional quantitative analysis of defects. However, its scanning process is time-consuming, making it difficult to meet the demands of high-speed online inspection. Furthermore, it is sensitive to environmental factors such as workpiece surface reflection and dust, easily generating noisy point cloud data that interferes with subsequent inspection and analysis.

[0072] Fixed binocular vision systems reconstruct 3D structures based on the principle of parallax, offering high processing efficiency. However, they are limited by fixed installation positions, restricting their adaptability to products of different sizes or structural forms. If the workpiece exhibits tilting, offset, or other posture deviations, and the posture estimation results are inaccurate, it can easily lead to 3D reconstruction distortion, directly reducing the accuracy of defect location and dimensional measurement.

[0073] The detection scheme of robots equipped with monocular cameras has a certain degree of motion flexibility and can achieve multi-angle observation through path planning. However, since monocular vision itself lacks direct depth information, it is necessary to rely on motion reconstruction structures or multi-frame matching algorithms to estimate pose and depth. The accuracy of pose estimation is affected by factors such as initial values ​​and environmental texture features, resulting in poor overall detection stability.

[0074] Further analysis reveals that existing 3D defect detection technologies still face the following common problems:

[0075] 1. Insufficient pose estimation accuracy: Most methods use traditional point cloud registration or image feature matching algorithms, which fail to fully integrate geometric and semantic information in multimodal data (such as point clouds and images) and lack closed-loop optimization mechanisms. This results in large pose calculation errors under conditions such as complex workpiece poses and weak textures in the detection scene, which directly affects the overall detection accuracy.

[0076] 2. Difficulty in balancing system adaptability and inspection efficiency: On the one hand, some inspection systems rely on high-precision fixtures or fixed loading methods to ensure the consistency of workpiece posture, which not only has high deployment costs but also poor adaptability to production scenarios; on the other hand, in fixtureless operation scenarios, the uncertainty of workpiece posture increases significantly, which can easily lead to the failure of registration between the standard model and the measured point cloud, resulting in false alarms and missed detections of defects, making it difficult to meet the quality control requirements of flexible manufacturing scenarios with multiple varieties and small batches.

[0077] It is worth noting that in high-end manufacturing sectors such as automobile manufacturing, existing inspection methods typically require workpieces to have reliable positioning references or to achieve precise positioning through specialized fixtures to ensure inspection reliability. However, such conditions are often difficult to meet in low- and mid-end manufacturing industries, which directly limits the widespread application of non-contact inspection technologies.

[0078] In summary, existing technologies still have significant shortcomings in terms of detection accuracy, environmental adaptability, and system flexibility. There is an urgent need for a method that can achieve deep integration of high-precision pose estimation and 3D defect detection. This method would enable efficient, accurate, and non-contact detection of surface defects in industrial products by accurately acquiring the workpiece pose and dynamically correcting the point cloud registration results, thus meeting the needs of diverse manufacturing scenarios.

[0079] To address the aforementioned issues, this disclosure provides a three-dimensional non-contact pose estimation method that achieves high-precision pose estimation of a workpiece by fusing multimodal data and combining it with differentiable rendering optimization.

[0080] Figure 1 This is a schematic diagram of the hardware architecture of a three-dimensional non-contact pose estimation system provided as an exemplary embodiment of this disclosure. Figure 1As shown, the system includes a server 11, an industrial camera 12, and a structured light projector 13. The number of industrial cameras 12 and structured light projectors 13 can be set according to actual needs. In other implementations, the industrial camera can also be replaced by electronic devices such as mobile phones, tablets, laptops, PDAs (Personal Digital Assistants), and wearable devices (such as smart glasses, smartwatches, etc.) with cameras. This disclosure does not limit this.

[0081] The industrial camera 12 interacts with the server 11 via a network, which can be a wired or wireless network, such as Wi-Fi (Wireless Local Area Network) or 4G (Fourth Generation Mobile Communication Technology).

[0082] Server 11 can be a physical server or a virtual server. Server 11 can run server-side programs for a certain application to implement the relevant business functions of the application. For example, when server 11 runs a three-dimensional non-contact pose estimation program, it can implement the relevant business functions of pose estimation.

[0083] In the technical solution of this embodiment, the server 11 cooperates with the industrial camera 12 and the structured light projector 13: the structured light projector projects a striped pattern of the target state onto the surface of the workpiece to be measured. The target state can be set according to actual needs, for example, six sinusoidal striped patterns with phases shifted sequentially by π / 3 are projected onto the surface of the workpiece to be measured, and their phases are adjusted according to... The beam pattern increases uniformly. Furthermore, the target state includes structured light in both pure white and pure black states. That is, the structured light projector also projects pure white (full brightness) and pure black structured light to acquire scene RGB images and normalize the stripe images. The scene RGB images acquired from the pure white structured light scene are used to construct high-dimensional information. The industrial camera 12 can flexibly adjust its shooting angle and position according to the size and geometry of the workpiece under test, acquiring RGB images of the workpiece under test simultaneously from different perspectives. The RGB images include stripe images and scene RGB images; these images are high signal-to-noise ratio RGB images. The server 11 acquires the RGB image and estimates the pose of the workpiece under test. In other implementations, the server 11 estimates the pose of the workpiece under test based on a grayscale image. The grayscale image can be obtained from an industrial camera or converted from an RGB image. The system has completed precise calibration before pose estimation, including the calibration of the intrinsic parameters (focal length, principal point coordinates, etc.) and extrinsic parameters (rotation matrix and translation vector) of the industrial camera.

[0084] It should be noted that industrial cameras and structured light projectors need to achieve hardware-level synchronization through a synchronization controller to meet the timing consistency requirements of batch inspection.

[0085] In one embodiment, the server directly receives RGB images captured by an industrial camera and performs pose estimation.

[0086] In one embodiment, the industrial camera stores the captured RGB images in a memory, and the server retrieves the RGB images from the memory for pose estimation. The memory can be the server's built-in memory or an external memory independent of the server.

[0087] The pose estimation process will be further explained below.

[0088] Figure 2 A three-dimensional non-contact pose estimation method is provided as an exemplary embodiment of this disclosure, the method comprising the following steps:

[0089] Step 201: Acquire scene RGB images of the workpiece under test simultaneously from at least two perspectives, and reconstruct the three-dimensional point cloud model of the workpiece under test based on the scene RGB images.

[0090] Among them, the scene RGB image is an image obtained by taking pictures of the workpiece under test with a fully bright stripe pattern projected on it.

[0091] The following example illustrates the process of constructing a 3D point cloud model: six sinusoidal stripe patterns are projected onto the workpiece by a structured light projector, and scene RGB images from two perspectives are acquired by two industrial cameras. The two industrial cameras simultaneously capture six sets of deformed scene RGB images (I1-I6) from different perspectives after being modulated by the workpiece surface height. The phase of the scene RGB images is calculated using a six-step phase-shifting algorithm to obtain the wrapping phase φ∈ (-π, π].

[0092] ;

[0093] Then, phase unwrapping is used to convert it into a continuous absolute phase Φ. Combined with the system calibration parameters, the absolute phase is mapped to height information, and finally a three-dimensional point cloud model of the workpiece under test is reconstructed. This three-dimensional point cloud model is also the measured point cloud model.

[0094] Step 202: Reproject the 3D point cloud model onto the image plane of the reference camera to generate a 2D depth map from the perspective of the reference camera.

[0095] The reference camera can be defined according to actual needs. It can be any industrial camera used to capture RGB images of the scene, or it can be set separately according to requirements.

[0096] Two-dimensional depth maps are also known as measured depth maps.

[0097] Step 203: Stitch together the scene RGB image and the two-dimensional depth map to obtain the measured multi-source data.

[0098] In one embodiment, to reduce computation and background interference, the measured mask image is multiplied pixel by pixel with the scene RGB image and the two-dimensional depth map respectively to accurately extract the scene RGB data and depth data containing only the target area of ​​the workpiece under test. The two are then fused into an RGBD segmented data cube with four channels of R, G, B, and D (i.e., measured multi-source data RGBD), providing high-quality multimodal data support for subsequent calculation of the actual pose of the workpiece under test.

[0099] The measured mask image was obtained by image segmentation of the scene's RGB image. The measured mask image is a binary mask image.

[0100] In one embodiment, a deep learning-based instance segmentation model (such as YOLO-Instance or Mask R-CNN) processes the scene's RGB image to accurately identify and segment individual workpieces in the image, outputting a measured mask image of the same size. In the measured mask image, the target region representing the workpiece has a pixel value of 1, while the background has a pixel value of 0, which is used for subsequent target (workpiece)-background separation and loss function calculation.

[0101] Step 204: Generate multiple sets of multi-source rendering data using a differentiable renderer and based on the pose parameters to be optimized. Determine the actual pose of the workpiece to be tested based on the pose parameters to be optimized corresponding to the multi-source rendering data that meet the iteration stopping condition.

[0102] The pose parameters to be optimized include a translation vector and / or a rotation matrix. The initial value of the translation vector is an initial translation vector, and the initial value of the rotation matrix is ​​an initial rotation matrix. The actual pose includes a translation vector and / or a rotation matrix.

[0103] Understandably, the initial translation vector and / or initial rotation matrix are the preliminary pose results obtained, i.e., the initial pose, which cannot accurately represent the actual pose of the workpiece under test, and can only serve as the initial values ​​for subsequent fine calculation and estimation of the actual pose.

[0104] In this embodiment, by fusing multimodal data of visual (RGB) information and geometric (depth / point cloud) information, and combining iterative optimization with differentiable rendering technology, high-precision pose estimation of the workpiece under test is achieved. Moreover, the pose estimation result has excellent robustness and can be adapted to complex industrial inspection conditions.

[0105] In one embodiment, before reconstructing the 3D point cloud model of the workpiece under test, the method further includes: filtering out the background region in the scene RGB image using a measured mask image. The filtered scene RGB image after removing the background region is then used as the data basis for pose estimation.

[0106] In one embodiment, the 3D point cloud model is further subjected to multi-level post-processing to remove noise and background interference, retaining a clean target point cloud. Pose estimation is then performed based on the post-processed 3D point cloud model. Multi-level post-processing may include, but is not limited to, at least one of the following operations:

[0107] Remove statistical outliers from the 3D point cloud model to eliminate noise interference and restore the true 3D geometric features of the workpiece.

[0108] Radius nearest neighbor filtering is performed on the 3D point cloud model to remove isolated points and sparse noise points with fewer than a threshold of neighborhood points;

[0109] Perform depth bandpass filtering on the 3D point cloud model to select valid points based on the target expected depth range.

[0110] If the scene has a known supporting plane, the post-processing of the 3D point cloud model can also include removing the planar background.

[0111] See Figure 3 The 3D point cloud model, after undergoing multiple post-processing stages, is then reprojected onto the image plane of the reference camera to generate a 2D depth map from the perspective of the reference camera.

[0112] In one embodiment, the scene RGB image and two-dimensional depth map are stitched together to obtain measured multi-source data, including: performing joint feature encoding on the scene RGB image and two-dimensional depth map to generate high-dimensional features; and using the high-dimensional features to characterize the measured multi-source data.

[0113] Feature extraction can be achieved, but is not limited to, through a trained neural network. This neural network can employ pre-trained HOG, SIFTCNN, ResNet, etc. This neural network can perform joint feature extraction from the scene's RGB image and two-dimensional depth map, generating a fused feature map to represent high-dimensional features.

[0114] In this embodiment, the measured multi-source data is characterized by high-dimensional features. That is, when comparing with the rendered multi-source data, the high-dimensional features extracted by the neural network are compared, which can improve robustness. Even if the target workpiece has complex situations such as partial occlusion or missing surface texture, the matching accuracy and stability can still be guaranteed.

[0115] In one embodiment, the pose parameters to be optimized include a translation vector. The first initial value of the translation vector, i.e., the initial translation vector, is determined by the following steps: back-projecting the two-dimensional depth map onto the camera coordinate system through the intrinsic parameters of the reference camera to obtain the target point cloud; and determining the mean of the three-dimensional coordinates of all or some of the points contained in the target point cloud as the initial translation vector t0.

[0116] In other implementations, the median (geometric center) of the three-dimensional coordinates of all or some of the points contained in the target point cloud is determined as the initial translation vector t0, or the center point obtained by principal component analysis (PCA) of the three-dimensional coordinates of all or some of the points contained in the target point cloud is determined as the initial translation vector t0.

[0117] Among them, some points included in the target point cloud can be selected as key points of the workpiece to be tested.

[0118] The reference camera can be a virtual camera, which is constructed based on the intrinsic and extrinsic parameters of a real industrial camera obtained from calibration, to ensure that the subsequent rendered image is geometrically consistent with the real image.

[0119] In this embodiment, the interference of outliers on translation estimation can be reduced.

[0120] In one embodiment, the pose parameters to be optimized include a rotation matrix. A template matching strategy is used to determine a second initial value for the rotation matrix, i.e., an initial rotation matrix, specifically:

[0121] S11. Perform uniform rotation sampling on the 3D point cloud model around the initial translation vector to generate multiple pose templates.

[0122] The initial translation vector can be determined by the average of the three-dimensional coordinates of all or some of the points contained in the target point cloud, or it can be specified by the user. This embodiment does not impose any particular limitation on this.

[0123] Multiple pose templates represent the three-dimensional shape that a perfect, defect-free workpiece should present under different inspection perspectives.

[0124] S12. Use a virtual camera to render templates of various poses to obtain the corresponding rendering mask images.

[0125] S13. Calculate the similarity between each rendered mask and the measured mask, and take the rotation matrix corresponding to the rendered mask with the highest similarity as the initial rotation matrix R0.

[0126] The similarity can be determined, but is not limited to, through IoU or the cosine similarity of features in the mask image.

[0127] In one embodiment, step S13 uses high-dimensional features of the mask image extracted by pre-trained neural networks such as HOG, SIFTCNN, and ResNet to perform similarity calculation, thereby improving robustness. Even if the target workpiece has complex situations such as partial occlusion or missing surface texture, the matching accuracy and stability can still be guaranteed.

[0128] In other implementations, the similarity between the high-dimensional features of the three-dimensional point cloud model of the workpiece under test and the high-dimensional features of each posture template is calculated, and the rotation matrix corresponding to the posture template with the highest similarity is used as the initial rotation matrix R0.

[0129] In one embodiment, when generating the pose template, sampling optimization is performed based on the geometric symmetry of the object to reduce unnecessary sampling points.

[0130] In one embodiment, a batch optimization strategy is employed to avoid the optimization process getting trapped in local minima. Specifically, based on the initial pose estimation, a set (e.g., J) of slightly different initial guessed poses are generated around the initial pose. For example, the rotation matrix is ​​sampled in 3D space (including uniform Euler angle sampling along the xyz axes, Fibonacci sphere sampling, etc.), and then the feature information corresponding to the sampled rotation poses is extracted using a neural network. This feature information is then compared with the measured mask image to determine the similarity between the two poses. Finally, several candidate poses with high similarity are selected for subsequent optimization.

[0131] In one embodiment, the parallel computing capabilities of computing devices (such as GPUs) are utilized to simultaneously perform pose transformation and differentiable rendering optimization on the 3D models corresponding to several candidate poses, and the loss value between each rendered RGBD data and the measured RGBD data is calculated in parallel to improve the efficiency of pose estimation.

[0132] In one embodiment, to achieve dynamic tracking of workpieces during the production process, multiple frames of scene RGB images are continuously acquired to obtain an RGB image sequence. Each frame of the scene RGB image sequence is processed separately to estimate the pose of the workpiece during the production process in real time, meeting the dynamic detection requirements of flexible manufacturing.

[0133] In one embodiment, the pose parameters to be optimized include a translation vector. The initial translation vector of the (n+1)th frame of the scene RGB image is determined based on the translation vector contained in the actual pose estimated from the scene RGB images of the previous target frames. n is an integer greater than or equal to 1.

[0134] In one embodiment, the pose parameters to be optimized include a rotation matrix. The initial rotation matrix of the scene RGB image in frame (n+1) is determined based on the rotation matrix contained in the actual pose estimated from the scene RGB images of the previous target frames.

[0135] The target frame number can be set according to the actual situation, for example, setting it to 1, which is the optimization result of the nth frame ( , The initial pose of the (n+1)th frame is used as the weighted result of the optimization results of the RGB images of the scene in multiple frames. When the number of target frames is greater than 1, the initial pose of the (n+1)th frame is used as the weighted result of the optimization results of the RGB images of the scene in multiple frames.

[0136] In this embodiment, to improve efficiency, a temporal inheritance strategy is adopted to determine the initial pose of the (n+1)th frame based on the actual pose estimated from the scene RGB image of the previous target frame. This strategy can significantly reduce the initial pose error, accelerate the optimization convergence speed, and reduce the risk of getting trapped in local optima, thereby supporting real-time dynamic tracking of the workpiece pose.

[0137] The process of optimizing differentiable rendering is described below.

[0138] Based on the initial pose (R0, t0) obtained above, a differentiable pose transformation module is constructed. This module rigidly transforms the standard 3D model from the model coordinate system to the virtual camera coordinate system. Then, the 3D model with the transformed coordinate system is input into a differentiable renderer (e.g., PyTorch3D). Using the calibrated intrinsic parameters of the virtual camera, multi-source rendering data is generated. This multi-source rendering data includes a rendered RGB image and a rendered depth image, meaning it encompasses both color channel and depth channel information. The rendered RGB image is exactly the same size as the actual captured scene RGB image.

[0139] The standard 3D model is a defect-free ideal model obtained by 3D modeling of the workpiece. The standard 3D model can be created, but is not limited to, through CAD software.

[0140] During iterative optimization, the rotation matrix R and translation vector t are parameterized as optimizable variables (e.g., using rotation vectors or quaternions to represent rotation), and the derivative with respect to θ (θ being the pose parameter) is solved using PyTorch's automatic differentiation mechanism. The Adam optimizer is preferably used for gradient descent iterations, gradually updating the pose parameters until the iteration stopping condition is met.

[0141] Where θ is a unified optimizable parameter vector of rotation matrix R and translation vector t, and R and t are geometric representations of pose.

[0142] Iteration stopping conditions may include, but are not limited to, the loss function value falling below a preset threshold or the number of iterations reaching the maximum number of iterations (e.g., 200). The final output R... * With t * This represents the actual pose of the workpiece under test, i.e., the high-precision 6D pose estimation result. The optimization process is visualized as follows: Figure 4 As shown: The red part in the figure is the RGB image of the actual scene, and the gray and white part is the outline of the 3D model rendered under the current optimized pose. The figure shows the optimized trajectory from the initial pose to the converged pose.

[0143] In this embodiment, the entire rendering process is differentiable, supporting gradient backpropagation to pose parameters. Specifically, it leverages the derivative characteristics of RGBD rendering with respect to spatial coordinates (X, Y) to enhance sensitivity to workpiece contours and geometric changes. Simultaneously, the rendered contours possess adjustable smooth edges, ensuring that even with significant initial pose deviations, the rendering mechanism can still output effective gradients, providing reliable guidance for subsequent pose optimization.

[0144] In one embodiment, rendering multi-source data includes rendering a mask map and rendering a depth map. The loss function of the differentiable renderer is a weighted sum of the mask loss and the depth loss.

[0145] Among them, the mask loss is the cross-union ratio loss between the measured mask map and the rendered mask map of the scene RGB image; the depth loss is the normalized mean square error between the two-dimensional depth map and the rendered depth map.

[0146] In this embodiment, a multi-component weighted loss function is used to measure the difference between the measured multi-source data RGBD and the rendered multi-source data RGBD.

[0147] In one embodiment, the loss function of a differentiable renderer is expressed as follows:

[0148] ;

[0149] ;

[0150] ;

[0151] in, Indicates the masking loss; A and B represent the measured mask and the rendered mask, respectively. The closer to 0, the higher the degree of overlap; the depth loss is represented by the normalized mean square error (NMSE) between the 2D depth map in the masked area and the rendered depth map. Represents the depth value of a two-dimensional depth map; This represents the depth value of the rendered depth map; The mean depth value of a two-dimensional depth map; and These are weighting parameters used to balance the importance of the two loss components. and You can configure it according to your actual needs.

[0152] In one embodiment, setting =1.0 and =1.0, meaning both are equally important.

[0153] See Figure 5 The pose estimation process will be further explained below:

[0154] Step 501: Process the scene RGB image and point cloud data using segmentation algorithm and point cloud filtering algorithm to obtain the measured mask image and two-dimensional depth map.

[0155] Step 502: Use a neural network to extract the first feature information of the measured mask image and the two-dimensional depth image, and obtain the initial pose based on the similarity comparison result between the first feature information and the second feature information of the pose template.

[0156] This feature information can be a high-dimensional feature.

[0157] Step 503: Based on the initial pose, transform the standard 3D model into the coordinate system of the reference camera to obtain multiple pose templates.

[0158] The pose of each pose template is represented by a transformation matrix and a translation vector.

[0159] Step 504: Perform differentiable rendering of the pose template based on the camera parameters of the reference camera to obtain the rendering mask map and the rendering depth map.

[0160] Step 505: Iterate using gradient descent to minimize the loss function between the rendered mask and the rendered depth map and the measured mask and the 2D depth map, so as to update the bit transformation matrix and translation vector.

[0161] Step 506: Iterate to obtain the pose estimation result of the high-precision workpiece.

[0162] The pose estimation result of a high-precision workpiece is also the actual pose of the workpiece.

[0163] In one embodiment, pose estimation is implemented using the open-source differentiable rendering library PyTorch3D, based on the PyTorch framework. Experiments were conducted on standard datasets (such as YCB-Video) and multiple real-world industrial scenarios. The results show that for a single target object with a known 3D model, the pose estimation error (measured by metrics such as ADD, ADD-S, or TE) in this embodiment can be stably controlled within the range of 0.5-1.5 pixels or 0.1°-0.5°. The specific accuracy is affected by camera resolution, target size, and environmental complexity. A single static pose estimation process (excluding image acquisition time) takes approximately 50-200 milliseconds, meeting the basic real-time requirements of most industrial online inspection systems. Combined with a dynamic tracking strategy, this embodiment can achieve continuous pose tracking of moving workpieces, exhibiting good stability and responsiveness.

[0164] Further analysis reveals that existing high-precision inspection methods (such as those applied in automotive manufacturing) typically rely on precise workpiece positioning and specialized fixtures to ensure inspection reliability. While these conditions are achievable in highly automated production lines, they are generally difficult to deploy in low- to mid-end manufacturing environments. In contrast, the method in this embodiment does not rely on the strict fixing or preset posture of the workpiece. It achieves accurate registration of arbitrarily placed workpieces through a high-precision posture estimation algorithm, effectively reducing the requirements for loading and positioning accuracy and production line automation levels. Therefore, this embodiment is not only suitable for high-end manufacturing applications with high inspection accuracy requirements, but also adaptable to multi-variety, small-batch production environments with low automation levels, exhibiting strong environmental adaptability and deployment flexibility.

[0165] This embodiment verifies the feasibility and effectiveness of the method in terms of attitude estimation accuracy, system robustness, and operating efficiency. The method can achieve high-precision attitude estimation under different working conditions, providing an accurate pose basis for stable and reliable 3D defect detection, and providing important technical support for the implementation of generalized intelligent detection in cross-level manufacturing scenarios.

[0166] This disclosure also provides a three-dimensional non-contact defect detection method, which includes the following steps:

[0167] Step 601: Estimate the actual pose of the workpiece to be measured.

[0168] The actual pose is determined according to the pose estimation method provided in any of the above embodiments.

[0169] Step 602: Convert the defect-free standard 3D model into a reference point cloud model that is identical to the actual pose of the workpiece to be tested.

[0170] In one embodiment, the conversion formula is expressed as follows:

[0171] ;

[0172] in, Represents a reference point cloud model; A point cloud representing a standard 3D model in its local coordinate system; and Let represent the rotation matrix and translation vector that characterize the actual pose, respectively.

[0173] It accurately characterizes the three-dimensional spatial position and geometric shape of the workpiece in the camera's field of view under an ideal, defect-free condition.

[0174] Step 603: Perform defect detection on the three-dimensional point cloud model of the workpiece under test based on the reference point cloud model.

[0175] The three-dimensional point cloud model of the workpiece to be tested is also the actual measured point cloud model.

[0176] In one embodiment, step 603 involves comparing the reference point cloud model with the 3D point cloud model of the workpiece under test point by point, calculating the geometric deviation, and performing defect detection based on the geometric deviation. The formula for calculating the geometric deviation can be found in the following description and will not be repeated here.

[0177] In one embodiment, the geometric features of the reference point cloud model and the three-dimensional point cloud model of the workpiece under test are extracted respectively, including parameters such as defect area, depth, volume and contour shape, and defect type identification and level classification are performed accordingly.

[0178] In this embodiment, based on high-precision attitude estimation, a reference point cloud model is generated from a standard 3D model. After registration and comparison with the measured point cloud model, precise location, identification, and quantitative analysis of defects can be achieved. This embodiment does not rely on precise pre-positioning of the workpiece, and can adapt to actual production conditions with large attitude changes, effectively improving the accuracy and versatility of defect detection. Therefore, it can be widely used in high-end manufacturing fields such as automotive parts and construction machinery, as well as adapting to the inspection needs of low- and mid-end manufacturing industries. It has excellent flexibility in scenario adaptation, outstanding overall practicality and scalability, and significant engineering application value.

[0179] Furthermore, in this embodiment, the closed-loop collaboration of high-precision attitude estimation and three-dimensional comparison significantly improves the accuracy, robustness and automation level of defect detection, making it particularly suitable for surface quality inspection of complex curved surfaces, highly reflective or precision-assembled industrial products.

[0180] In one embodiment, before defect detection, the three-dimensional point cloud model (actual model) of the workpiece to be tested is preprocessed, including removing environmental noise, outliers and background clutter, to obtain filtered three-dimensional point cloud data containing the workpiece to be tested and its corresponding depth map.

[0181] In one embodiment, in order to achieve effective comparison, the reference point cloud model and the three-dimensional point cloud model of the workpiece under test need to be unified in coordinate system to verify the consistency and integrity of the data.

[0182] In one embodiment, the method further includes the following steps prior to step 603:

[0183] S21. Find the nearest neighbor of the target point in the 3D point cloud model in the reference point cloud model, and calculate the Euclidean distance between the target point and the nearest neighbor.

[0184] The measured point cloud model With reference point cloud model Perform point-by-point comparison. For the measured point cloud model... Each point in the reference point cloud model Find the nearest neighbor of the i-th point and calculate the Euclidean distance d between the two points. This distance d represents the geometric deviation of the i-th point. Where the geometric deviation of the i-th point is... Satisfy the following formula:

[0185] ;

[0186] in, For the measured point cloud model The i-th point in Reference point cloud model Zhongyu The corresponding nearest neighbor, ||·|| represents the L2 norm.

[0187] S22. Determine the geometric deviation between the reference point cloud model and the 3D point cloud model based on the Euclidean distance.

[0188] In one embodiment, for a measured point cloud model containing N points The sum of geometric deviations is shown below:

[0189] ;

[0190] S23. In response to the geometric deviation being less than or equal to the deviation threshold, perform the step of defect detection on the three-dimensional point cloud model of the workpiece under test based on the reference point cloud model.

[0191] S24. In response to a geometric deviation exceeding a deviation threshold, return the step of estimating the actual pose of the workpiece to be measured.

[0192] The deviation threshold T can be set by the user according to the accuracy requirements.

[0193] If the geometric deviation D > T, the attitude estimation error is determined to be too large, and the process returns to the attitude estimation step. If the total geometric deviation D ≤ T, the attitude estimation is determined to meet the accuracy requirements, and the next step of defect detection is performed.

[0194] In this embodiment, attitude verification is performed before defect detection. By confirming the accuracy of the attitude estimation results and the absence of displacement of the object under test during the detection process, the detection deviation caused by attitude error and workpiece displacement is avoided from the root, ensuring the accuracy, authenticity and reliability of the defect detection results. At the same time, the robustness and engineering practicality of the detection are improved, providing reliable technical support for quality control in industrial production.

[0195] In one embodiment, defect detection of the three-dimensional point cloud model of the workpiece under test based on the reference point cloud model includes: performing point cloud registration between the reference point cloud model and the three-dimensional point cloud model using an iterative nearest point algorithm to obtain the transformation relationship between the reference point cloud model and the three-dimensional point cloud model; updating the three-dimensional point cloud model according to the transformation relationship; and performing defect detection on the updated three-dimensional point cloud model based on the reference point cloud model.

[0196] In one embodiment, to improve the comparability and visualization of the difference values, all calculated geometric deviation values ​​d are normalized and mapped to the [0,1] interval. The deviation value at the i-th point after normalization is denoted as... Based on the accuracy requirements of the inspection task and the manufacturing tolerances of the workpiece, a defect judgment threshold T' is set. All points with normalized deviation values ​​d > T' are marked as potential defect points.

[0197] In one embodiment, after obtaining the defect detection results, 3D visualization rendering and defect analysis are also performed. Specifically, this includes at least one of the following:

[0198] Reference point cloud model display: In the 3D visualization window, the reference point cloud model is first rendered using a reference color (e.g., neutral gray or white) and used as the reference model for defect comparison.

[0199] Difference point cloud coloring: See Figure 7 Based on the normalized difference values, each point in the measured point cloud model is colored and rendered. A color mapping table is used to visually represent the degree of difference. The mapping rule, for example, ranges from blue (representing very small or no difference) to blue (representing very small or no difference). ≈0) transitions to red (indicating a significant difference), ≈1). This color gradient can intuitively show the spatial location, size, and severity of defects. In the figure, the gray point cloud represents the reference point cloud model, and the colored point cloud represents the measured point cloud model.

[0200] Defect area highlighting and annotation: Extract point cloud regions with difference values ​​exceeding the threshold T', visually highlight them in a striking manner (such as bold red dots, flashing effects, or bounding boxes), and annotate the defect areas with quantitative information such as the maximum difference value, average difference value, or defect area of ​​the defect area.

[0201] Visualization Interface: Provides an interactive visualization interface that allows users to observe, rotate, and zoom the 3D model from different angles to examine defects in detail.

[0202] Inspection Report: Automatically generates inspection reports that include the number of defects, the 3D coordinate range of each defect, its size, the severity level based on the difference value, and 3D visualization screenshots of the defect area. This provides complete data for workpiece quality assessment, subsequent process optimization, and problem tracing.

[0203] In this embodiment, various three-dimensional defects (such as dents, protrusions, scratches, cracks, etc.) on the surface of a workpiece can be accurately detected. At the same time, the defects can be presented intuitively, quantitatively characterized, and visualized. Abstract pose parameters are transformed into intuitive and analyzable three-dimensional defect information, effectively connecting the technical links between pose estimation and defect analysis. This allows the location, size, severity, and other information of three-dimensional defects to be clearly presented and quantitatively characterized, providing intuitive and reliable information for the accurate judgment, quantitative analysis, and quality control of defects in industrial production. It also provides strong technical support for quality control and quality control decisions in the industrial production process.

[0204] Corresponding to the foregoing method embodiments, this disclosure also provides system embodiments.

[0205] Figure 8a A three-dimensional non-contact pose estimation system is provided as an exemplary embodiment of this disclosure. This system is used to implement the three-dimensional non-contact pose estimation method provided in any of the above embodiments. The system includes:

[0206] The acquisition module 81 is used to acquire scene RGB images of the workpiece under test simultaneously captured from at least two perspectives, and to reconstruct a three-dimensional point cloud model of the workpiece under test based on the scene RGB images; wherein, the scene RGB images are images captured of the workpiece under test with a fully bright stripe pattern projected on it;

[0207] The generation module 82 is used to reproject the three-dimensional point cloud model onto the image plane of the reference camera to generate a two-dimensional depth map from the perspective of the reference camera.

[0208] The stitching module 83 is used to stitch together the scene RGB image and the two-dimensional depth map to obtain measured multi-source data;

[0209] The optimization module 84 is used to generate multiple sets of multi-source rendering data through a differentiable renderer and based on the pose parameters to be optimized, and to determine the actual pose of the workpiece to be tested according to the pose parameters to be optimized corresponding to the multi-source rendering data that meet the iteration stopping condition.

[0210] Optionally, the system further includes: a preprocessing module, used to filter out the background region in the scene RGB image using a measured mask image; wherein the measured mask image is obtained by image segmentation of the scene RGB image;

[0211] The preprocessing module is also used to perform at least one of the following operations after the step of reconstructing the three-dimensional point cloud model of the workpiece under test:

[0212] Perform depth bandpass filtering on the 3D point cloud model;

[0213] Remove statistical outliers from the 3D point cloud model;

[0214] Radius nearest neighbor filtering is performed on the three-dimensional point cloud model.

[0215] Optionally, the splicing module is specifically used for:

[0216] Joint feature encoding is performed on the scene RGB image and the two-dimensional depth map to generate high-dimensional features;

[0217] The high-dimensional features are used to characterize the measured multi-source data.

[0218] Optionally, the pose parameters to be optimized include a translation vector; the system further includes a determination module, used for:

[0219] The two-dimensional depth map is back-projected onto the camera coordinate system using the intrinsic parameters of the reference camera to obtain the target point cloud;

[0220] The mean of the three-dimensional coordinates of the target point cloud is determined as the initial translation vector.

[0221] Optionally, the pose parameters to be optimized include a rotation matrix; the determining module is used for:

[0222] The three-dimensional point cloud model is uniformly rotated and sampled around the initial translation vector to generate multiple pose templates;

[0223] Use a virtual camera to render templates for each pose to obtain the corresponding rendering mask.

[0224] Calculate the similarity between each rendered mask and the measured mask, and use the rotation matrix corresponding to the rendered mask with the highest similarity as the initial rotation matrix; wherein, the measured mask is obtained by image segmentation of the scene RGB image.

[0225] Optionally, the pose parameters to be optimized include translation vectors; the initial translation vector of the (n+1)th frame scene RGB image is determined based on the translation vector contained in the actual pose estimated from the nth frame scene RGB image.

[0226] And / or, the pose parameters to be optimized include an initial rotation matrix; the initial rotation matrix of the (n+1)th frame scene RGB image is determined based on the rotation matrix contained in the actual pose estimated from the nth frame scene RGB image.

[0227] Optionally, the rendered multi-source data includes a rendered mask map and a rendered depth map;

[0228] The loss function of the differentiable renderer is a weighted sum of the mask loss and the depth loss;

[0229] Wherein, the mask loss is the intersection-union ratio loss of the measured mask map of the scene RGB image and the rendered mask map; the depth loss is the normalized mean square error of the two-dimensional depth map and the rendered depth map.

[0230] This disclosure also provides a three-dimensional non-contact defect detection system, see [link to relevant documentation]. Figure 8b This system is used to implement the three-dimensional non-contact defect detection method provided in any of the above embodiments. The system includes:

[0231] Estimation module 85 is used to estimate the actual pose of the workpiece to be measured; the actual pose is determined according to any of the pose estimation methods described above.

[0232] The conversion module 86 is used to convert a defect-free standard 3D model into a reference point cloud model that is identical to the actual pose of the workpiece under test.

[0233] The detection module 87 is used to perform defect detection on the three-dimensional point cloud model of the workpiece to be tested based on the reference point cloud model.

[0234] Optionally, the system also includes a pose verification module for:

[0235] Find the nearest neighbor of the target point in the reference point cloud model and calculate the Euclidean distance between the target point and the nearest neighbor.

[0236] The geometric deviation between the reference point cloud model and the 3D point cloud model is determined based on the Euclidean distance;

[0237] In response to the geometric deviation being less than or equal to the deviation threshold, the detection module is invoked to perform defect detection on the three-dimensional point cloud model of the workpiece under test based on the reference point cloud model;

[0238] In response to the geometric deviation being greater than the deviation threshold, a three-dimensional non-contact pose estimation system is invoked.

[0239] Optionally, the detection system is specifically used for:

[0240] The iterative nearest point algorithm is used to perform point cloud registration between the reference point cloud model and the 3D point cloud model to obtain the transformation relationship between the reference point cloud model and the 3D point cloud model;

[0241] The 3D point cloud model is updated according to the transformation relationship, and defects are detected in the updated 3D point cloud model according to the reference point cloud model.

[0242] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs.

[0243] Figure 9 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the method described in any of the above embodiments. Figure 9 The electronic device 90 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0244] like Figure 9 As shown, the electronic device 90 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including memory 92 and processor 91).

[0245] Bus 93 includes a data bus, an address bus, and a control bus.

[0246] The memory 92 may include volatile memory, such as random access memory (RAM) 921 and / or cache memory 922, and may further include read-only memory (ROM) 923.

[0247] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) program module 924, such program module 924 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0248] The processor 91 executes various functional applications and data processing, such as the methods provided in any of the above embodiments, by running computer programs stored in the memory 92.

[0249] Electronic device 90 can also communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 95. Furthermore, electronic device 90 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 96. As shown, network adapter 96 communicates with other modules of electronic device 90 via bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0250] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0251] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in any of the above embodiments.

[0252] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0253] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the above embodiments.

[0254] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0255] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.

Claims

1. A three-dimensional non-contact pose estimation method, characterized in that, include: Acquire scene RGB images of the workpiece under test simultaneously from at least two perspectives, and reconstruct a three-dimensional point cloud model of the workpiece under test based on the scene RGB images; wherein, the scene RGB images are images of the workpiece under test with a fully bright stripe pattern projected on it; The three-dimensional point cloud model is reprojected onto the image plane of the reference camera to generate a two-dimensional depth map from the perspective of the reference camera. By stitching together the RGB image of the scene and the two-dimensional depth map, measured multi-source data is obtained; The actual pose of the workpiece under test is determined by generating multiple sets of multi-source rendering data based on the pose parameters to be optimized using a differentiable renderer, and then determining the pose parameters to be optimized corresponding to the multi-source rendering data that meet the iteration stopping condition.

2. The three-dimensional non-contact pose estimation method according to claim 1, characterized in that, Before the step of reconstructing the three-dimensional point cloud model of the workpiece to be tested, the method further includes: filtering out the background region in the scene RGB image using a measured mask image; wherein, the measured mask image is obtained by image segmentation of the scene RGB image; And / or, after the step of reconstructing the three-dimensional point cloud model of the workpiece to be tested, the method further includes at least one of the following operations: Perform depth bandpass filtering on the 3D point cloud model to remove statistical outliers from the 3D point cloud model; Radius nearest neighbor filtering is performed on the three-dimensional point cloud model.

3. The three-dimensional non-contact pose estimation method according to claim 1, characterized in that, By stitching together the RGB image of the scene and the two-dimensional depth map, measured multi-source data is obtained, including: Joint feature encoding is performed on the scene RGB image and the two-dimensional depth map to generate high-dimensional features; The high-dimensional features are used to characterize the measured multi-source data.

4. The three-dimensional non-contact pose estimation method according to claim 1, characterized in that, The pose parameters to be optimized include a translation vector; the first initial value of the translation vector is determined through the following steps: The two-dimensional depth map is back-projected onto the camera coordinate system using the intrinsic parameters of the reference camera to obtain the target point cloud; The mean of the three-dimensional coordinates of the target point cloud is determined as the first initial value.

5. The three-dimensional non-contact pose estimation method according to claim 1, characterized in that, The pose parameters to be optimized include a rotation matrix; the second initial value of the rotation matrix is ​​determined through the following steps: The three-dimensional point cloud model is uniformly rotated and sampled around the first initial value of the translation vector to generate multiple pose templates; Use a virtual camera to render templates for each pose to obtain the corresponding rendering mask. Calculate the similarity between each rendered mask and the measured mask, and use the rotation matrix corresponding to the rendered mask with the highest similarity as the second initial value; wherein, the measured mask is obtained by image segmentation of the scene RGB image.

6. The three-dimensional non-contact pose estimation method according to claim 1, characterized in that, The pose parameters to be optimized include translation vectors; the first initial value of the translation vector of the (n+1)th frame scene RGB image is determined based on the translation vector contained in the actual pose estimated from the nth frame scene RGB image. And / or, the pose parameters to be optimized include a rotation matrix; the second initial value of the rotation matrix of the (n+1)th frame scene RGB image is determined based on the rotation matrix contained in the actual pose estimated from the nth frame scene RGB image.

7. The three-dimensional non-contact pose estimation method according to any one of claims 1-6, characterized in that, The rendered multi-source data includes a rendered mask map and a rendered depth map; The loss function of the differentiable renderer is a weighted sum of the mask loss and the depth loss; Wherein, the mask loss is the intersection-union ratio loss of the measured mask map of the scene RGB image and the rendered mask map; the depth loss is the normalized mean square error of the two-dimensional depth map and the rendered depth map.

8. A three-dimensional non-contact defect detection method, characterized in that, include: Estimate the actual pose of the workpiece under test; The actual pose is determined by the pose estimation method according to any one of claims 1-7; The defect-free standard 3D model is converted into a reference point cloud model that is identical to the actual pose of the workpiece under test. Defect detection is performed on the three-dimensional point cloud model of the workpiece under test based on the reference point cloud model.

9. The three-dimensional non-contact defect detection method according to claim 8, characterized in that, Before the step of performing defect detection on the three-dimensional point cloud model of the workpiece under test based on the reference point cloud model, the method further includes: Find the nearest neighbor of the target point in the reference point cloud model and calculate the Euclidean distance between the target point and the nearest neighbor. The geometric deviation between the reference point cloud model and the 3D point cloud model is determined based on the Euclidean distance; In response to the geometric deviation being less than or equal to the deviation threshold, a step of defect detection is performed on the three-dimensional point cloud model of the workpiece under test based on the reference point cloud model. In response to the geometric deviation being greater than the deviation threshold, the step of estimating the actual pose of the workpiece under test is returned.

10. The three-dimensional non-contact defect detection method according to claim 8 or 9, characterized in that, Defect detection is performed on the three-dimensional point cloud model of the workpiece under test based on the reference point cloud model, including: The iterative nearest point algorithm is used to perform point cloud registration between the reference point cloud model and the 3D point cloud model to obtain the transformation relationship between the reference point cloud model and the 3D point cloud model; The 3D point cloud model is updated according to the transformation relationship, and defects are detected in the updated 3D point cloud model according to the reference point cloud model.

11. A three-dimensional non-contact pose estimation system, characterized in that, include: The acquisition module is used to acquire scene RGB images of the workpiece under test simultaneously captured from at least two perspectives, and to reconstruct a three-dimensional point cloud model of the workpiece under test based on the scene RGB images; wherein, the scene RGB images are images captured of the workpiece under test with a fully bright stripe pattern projected on it; The generation module is used to reproject the three-dimensional point cloud model onto the image plane of the reference camera to generate a two-dimensional depth map from the perspective of the reference camera. The stitching module is used to stitch together the scene RGB image and the two-dimensional depth map to obtain measured multi-source data; The optimization module is used to generate multiple sets of multi-source rendering data through a differentiable renderer based on the pose parameters to be optimized, and to determine the actual pose of the workpiece to be tested according to the pose parameters to be optimized corresponding to the multi-source rendering data that meet the iteration stopping condition.

12. A three-dimensional non-contact defect detection system, characterized in that, include: The estimation module is used to estimate the actual pose of the workpiece under test; The actual pose is determined by the pose estimation method according to any one of claims 1-7; The conversion module is used to convert a defect-free standard 3D model into a reference point cloud model that is identical to the actual pose of the workpiece under test. The detection module is used to perform defect detection on the three-dimensional point cloud model of the workpiece to be tested based on the reference point cloud model.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 10.

Images