An intelligent visual detection method and device for component defects

Abstract

The application provides an intelligent visual detection method and device for component defects, which comprises the following steps: loading a light standard three-dimensional model corresponding to a target component from a backend server; synchronously collecting real-time image data and depth data of the component on a conveying mechanism or a clamping mechanism; generating a real-time point cloud model for describing surface geometric features of the component based on the real-time image data and the depth data; matching the real-time point cloud model with the light standard three-dimensional model to locate a point area of interest in a high-defect area of the component; spatially mapping and fusing texture features in the real-time image data and the depth data in the point area of interest; and identifying defects in the point area of interest of the component through a joint probability model. The application can realize high-speed and accurate defect detection of components without complex feature engineering, and greatly reduces the maintenance difficulty and implementation cost of the algorithm.

Description

Technical Field

[0001] This invention relates to the field of industrial automation inspection technology, specifically to an intelligent visual inspection method and device for component defects. Background Technology

[0002] In modern electronic manufacturing and semiconductor packaging, component quality control is a core element in ensuring product reliability. Due to manufacturing process errors such as poor soldering and pin misalignment, external physical damage such as scratches and cracks, and environmental stresses such as oxidation and corrosion, component surfaces are highly susceptible to various geometric deformations and texture defects. Existing technologies include numerous methods for identifying surface defects in components, such as laser scanning, visual imaging, ultrasonic testing, and infrared thermal imaging. However, given the diverse types and shapes of components, single-sensor defect detection methods often have limitations. Therefore, multi-sensor fusion has been researched to improve the accuracy and reliability of defect detection. However, multi-sensor fusion relies on pre-processing feature engineering, requiring manual feature extraction for specific defects to design specialized feature fusion algorithms. Furthermore, the entire process of multi-channel data acquisition and processing, feature extraction, feature fusion, and defect identification involves a large amount of computation. In high-speed production environments, it is difficult to balance the trade-off between defect identification efficiency and accuracy. Summary of the Invention

[0003] Based on the above-mentioned problems, this invention proposes an intelligent visual detection method and device for component defects. Through asynchronous pipeline design and hardware-level synchronization, it can achieve high-speed and accurate defect detection of components without the need for complex feature engineering, which greatly reduces the maintenance difficulty and implementation cost of the algorithm.

[0004] In view of this, a first aspect of the present invention proposes an intelligent visual detection method for component defects, comprising: Load the lightweight standard 3D model corresponding to the target component; Synchronously acquire real-time image and depth data of components on the conveying or clamping mechanism; A real-time point cloud model is generated based on the real-time image data and the depth data to describe the surface geometric features of the component. The real-time point cloud model is matched with the lightweight standard 3D model to locate the region of interest in the high-defect area of ​​the component. Spatial mapping and fusion are performed on the texture features and depth data in the real-time image data within the region of interest. Defects in the region of interest of the component are identified by a joint probability model.

[0005] Optionally, the steps of synchronously acquiring real-time image data and depth data of components on the conveying mechanism or clamping mechanism specifically include: Receive the position signal from the servo driver to trigger synchronous sensor acquisition; The acquired raw depth image is subjected to median filtering for noise reduction and voxel-based downsampling. The lightweight feature data, after denoising and downsampling, is transmitted to the backend server via a high-speed communication interface.

[0006] Optionally, the step of matching the real-time point cloud model with the lightweight standard 3D model to locate the region of interest in the high-defect area of ​​the component specifically includes: The pose matching between the real-time point cloud model and the lightweight standard 3D model is performed on a global scale using fast point feature histograms. The iterative nearest-point algorithm is used to achieve pixel-level spatial overlap within the local area determined by the pose matching; Determine the distribution of deviation features in the lightweight standard 3D model within the overlapping space of the real-time point cloud model. The point of interest region is automatically located based on the deviation characteristics distributed within the high-defect areas of the component.

[0007] Optionally, the step of automatically locating the point of interest region within the high-defect area of ​​the component based on the deviation characteristics specifically includes: Read pre-configured mask data from the lightweight standard 3D model, the mask data being used to define the defect detection shielding area of ​​the component; The areas on the component other than the defect detection shielding area are identified as the high-incidence areas of defects. The distribution of deviation characteristics in the high-incidence area of ​​the defect is analyzed to locate the region of interest.

[0008] Optionally, the step of analyzing the distribution of deviation characteristics in the high-incidence defect area to locate the region of interest specifically includes: Obtain the lower boundary of the pre-configured tolerance threshold range for determining the geometric deviation of the target component. and the upper boundary of the tolerance threshold range ; The vertical distance from each sampling point on the surface of the real-time point cloud model within the area of ​​interest to the surface of the lightweight standard 3D model. The lower boundary of the tolerance threshold range and upper boundary Compare; When any sampling point within the interest area satisfies When this happens, the sampling point is marked as a suspected defect point; When any sampling point within the interest area satisfies When this happens, the sampling point is marked as a defect point.

[0009] Optionally, after the step of marking the sampling point as a defect point, the method further includes: Cluster analysis is performed on each sampling point on the surface of the real-time point cloud model to obtain several defect point clusters; The number of cluster points in the defect point cluster is compared with a preset defect point threshold. When the number of cluster points in any defect point cluster is greater than the defect point threshold, the region where the defect point cluster is located is configured as the point of interest region.

[0010] Optionally, after marking the sampling points as suspected defect points, the method further includes: Cluster analysis is performed on each sampling point on the surface of the real-time point cloud model to obtain several clusters of suspected defect points. The clusters of suspected defect points consist of suspected defect points and defect points whose distance is less than a preset clustering distance. The number of cluster points in the suspected defect point cluster is compared with a preset suspected defect point threshold. When the number of cluster points in any suspected defect point cluster is greater than the suspected defect point threshold, the area where the suspected defect point cluster is located is configured as a secondary verification area. Perform a secondary verification on the secondary verification area.

[0011] Optionally, the real-time point cloud model includes a mapping relationship between each two-dimensional coordinate point in the real-time image data and the three-dimensional spatial coordinates in the spatial coordinate system of the real-time point cloud model. The step of spatially mapping and fusing the texture features and depth data in the real-time image data within the region of interest specifically includes: Within the area of ​​interest, determine a number of target sampling points; Extract the RGB pixel gradient of the target sampling point from the real-time image data. As the texture feature of the component at the target sampling point, wherein Used to represent the coordinates of the target sampling point; Extract the vertical distance of the target sampling point from the depth data. As the geometric deviation of the component at the target sampling point; The joint probability of the component at the target sampling point is obtained by weighted fusing of the texture features and the geometric deviation: .

[0012] Optionally, the joint probability The method, calculated using a pre-built joint probability model, includes the following steps prior to the step of identifying defects in the region of interest of the component using the joint probability model: Acquire large amounts of real-time image and depth data using laboratory data. The joint observation vector of the two sensors is constructed using the real-time image data and depth data: ; The joint observation vector is constructed through manual annotation. and its category tags The observed data sample pairs, among which ; The observed data sample pairs are divided into qualified datasets. and defect dataset ; The probability density distribution of the qualified dataset was calculated using the maximum likelihood estimation method. and the probability density distribution of the defect dataset. ; Define the joint probability function: .

[0013] A second aspect of the present invention provides an intelligent visual inspection device for component defects, comprising a front-end sensing module, a high-speed communication interface, and an edge controller. The front-end sensing module includes an image sensor for acquiring real-time image data of the component and a depth sensor for acquiring depth data of the component. The controller communicates with a back-end server through the high-speed communication interface to implement the intelligent visual inspection method for component defects as described in any of the first aspects of the present invention.

[0014] This invention proposes an intelligent visual detection method and device for component defects. It loads a lightweight standard 3D model corresponding to the target component from a backend server, simultaneously acquires real-time image data and depth data of the component on a conveying or clamping mechanism, generates a real-time point cloud model describing the surface geometric features of the component based on the real-time image data and the depth data, matches the real-time point cloud model with the lightweight standard 3D model to locate regions of interest (POIs) in high-defect areas of the component, and spatially maps and fuses the texture features in the real-time image data and the depth data within the POIs. A joint probability model is then used to identify defects in the POIs of the component. This method enables high-speed and accurate defect detection of components without complex feature engineering, significantly reducing the maintenance difficulty and implementation cost of the algorithm. Attached Figure Description

[0015] Figure 1 This is a flowchart of an intelligent visual detection method for component defects provided in one embodiment of the present invention; Figure 2 This is a schematic diagram of an intelligent visual inspection device for component defects provided in an embodiment of the present invention. Detailed Implementation

[0016] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0017] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0018] In the description of this invention, the term "multiple" refers to two or more. Unless otherwise explicitly defined, the terms "upper," "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. The terms "connect," "install," "fix," etc., should be interpreted broadly. For example, "connect" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "multiple" means two or more.

[0019] In the description of this specification, the terms "one embodiment," "some implementations," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0020] The following description, with reference to the accompanying drawings, illustrates an intelligent visual inspection method and apparatus for component defects according to some embodiments of the present invention.

[0021] like Figure 1 As shown, (right 1) Step 100: Load the lightweight standard 3D model corresponding to the target component.

[0022] Specifically, the lightweight standard 3D model is a geometric data model obtained by downsampling the original 3D model file, such as a CAD model file, from the backend server. This downsampling process includes, but is not limited to, geometric simplification and information dimension stripping. Geometric simplification involves using mesh simplification algorithms to remove internal structural information from the original model, retaining only the external surface information of components. Information dimension stripping involves removing a large amount of non-geometric metadata from the original model file, including material properties, thermodynamic parameters, part numbers, assembly hierarchy relationships, etc.

[0023] The embodiments of the present invention use a model-driven method to realize defect detection of components. For any component, only its three-dimensional model (such as CAD file) needs to be imported to carry out detection. There is no need to retrain the neural network or perform complex feature engineering for each defect type of each component.

[0024] The above steps also include configuring a lower boundary of the tolerance threshold range for determining the geometric deviation of the target component. and the upper boundary of the tolerance threshold range In practice, each different component can be configured with its corresponding tolerance threshold range.

[0025] More specifically, when the intelligent visual inspection device loads the lightweight standard 3D model corresponding to the target component from the backend server, it sets upper and lower tolerance threshold ranges around the surface of the lightweight standard 3D model to determine geometric deviations according to preset process accuracy requirements. and The lower boundary of the tolerance threshold range. and upper boundary These are the lower and upper limits for determining the geometric deviation of the target component relative to the standard shape, respectively.

[0026] Step 200: Synchronously acquire real-time image data and depth data of the components on the conveying mechanism or clamping mechanism.

[0027] In some embodiments of the present invention, step 200 specifically includes: Step 210: Receive the position signal from the servo driver to trigger synchronous sensor acquisition; In the technical solution of this invention, the front-end sensing module consists of an image sensor and a depth sensor. The intelligent vision inspection device monitors the servo feedback signal of the motion control system in real time. When it detects that the component under inspection is in a predetermined shooting pose, it sends a synchronous acquisition command to the image sensor and depth sensor in the front-end sensing module through a hardware trigger interface, thereby realizing the synchronous acquisition of real-time image data and depth data of the component. The hardware trigger interface is an electrical signal path connecting the edge controller and the sensor, used to achieve microsecond-level synchronization between camera exposure and the servo mechanism's positioning time, eliminating temporal errors during motion and significantly reducing the false alarm rate.

[0028] In some embodiments of the present invention, the image sensor is a high frame rate RGB (Red-Green-Blue) camera, used to acquire real-time image data of the component. The depth sensor is a structured light sensor, used to acquire real-time depth data of the component.

[0029] In practical implementation, when the component arrives at the shooting area, the servo motor transmits a signal indicating that it has reached the predetermined position to the FPGA (Field Programmable Gate Array). The FPGA can trigger simultaneous exposure of the high frame rate RGB camera and the structured light sensor within 10 microseconds, thereby ensuring that the imaging position of the high-speed moving component does not shift significantly. The arrival signal is a pulse signal emitted by the servo driver through the digital I / O (Input / Output) port when the mechanical movement reaches the predetermined physical coordinates.

[0030] Step 220: Perform median filtering for noise reduction and voxel-based downsampling on the acquired raw depth image; In high-speed visual inspection systems, the amount of data generated by the front-end sensing module (RGB-D) is enormous. Directly pushing the entire unprocessed raw point cloud stream to the back-end server would cause two fatal problems: first, network bandwidth would be instantly saturated; second, the back-end server would have to expend significant computing power to process invalid noise and redundant data. This invention utilizes an edge controller embedded near the sensor module, such as an FPGA or a high-performance SoC (System-on-Chip), to perform downsampling and noise reduction preprocessing on the acquired raw depth data, ensuring that the data reaching the back-end server is high signal-to-noise ratio and lightweight downsampled data.

[0031] The median filtering denoising process is a nonlinear spatial filtering process that replaces the center pixel with the median value of all values ​​within the pixel's neighborhood. This effectively removes isolated high-depth or extremely low-depth points randomly appearing in the depth sensor, improving the smoothness of the depth map. The voxel-based downsampling operation is a data compression process based on three-dimensional spatial meshing. In its implementation, it divides the space into equally sized cubes and retains only one centroid point in each voxel to achieve downsampling. This significantly reduces the order of magnitude of the point cloud without destroying the overall geometric features of the part, achieving data lightweighting and reducing the computational load on the backend server.

[0032] Step 230: Transmit the denoised and downsampled lightweight feature data to the backend server via a high-speed communication interface.

[0033] In some implementations, the high-speed communication interface can be a high-bandwidth transmission channel such as PCIe (Peripheral Component Interconnect Express), 10GbE (10 Gigabit Ethernet), or Camera Link, to ensure that processed feature data can flow from the edge layer to the back-end server without delay, eliminating communication bottlenecks in the data flow process.

[0034] Step 300: Generate a real-time point cloud model to describe the surface geometric features of the component based on the real-time image data and the depth data.

[0035] In some embodiments of the present invention, step 300 specifically includes: Step 310: After obtaining the real-time image data and depth data of the components, perform downsampling and noise reduction preprocessing on the acquired raw depth data; Step 320: Extract the RGB texture features of the component surface from the real-time image data; Step 330: Use the downsampled depth information and the texture features to perform rapid 3D reconstruction of the visible surface of the component to generate a real-time point cloud model that describes the surface geometry of the component.

[0036] In some embodiments of the present invention, the backend server employs an asynchronous processing pipeline data flow mechanism to achieve continuity in component detection. Specifically, while the frontend sensing module acquires real-time image and depth data of the current component and preprocesses the real-time image and depth data, the backend server performs model matching and defect identification of the previous component in parallel.

[0037] Step 400: Match the real-time point cloud model with the lightweight standard 3D model to locate the region of interest in the high-defect area of ​​the component.

[0038] In some embodiments of the present invention, step 400 specifically includes: Step 410: Use fast point feature histograms to perform pose matching between the real-time point cloud model and the lightweight standard 3D model on a global scale; In practical implementation, the Fast Point Feature Histogram (FPFH) descriptor can be used to transform complex geometries into feature histograms with rotation / translation invariance. Fast feature matching can then be used to determine the approximate initial pose of the component at the time of image capture. The FPFH descriptor is a statistical descriptor generated based on local geometric features, used to encode the spatial structure information of three-dimensional points into high-dimensional vectors. This allows the algorithm to quickly identify the approximate relative pose between the component's current pose and the standard model through feature matching, without relying on the initial pose.

[0039] Step 420: Use the iterative nearest point algorithm to achieve pixel-level spatial overlap within the local area determined by the pose matching; Within the range determined by coarse registration, an improved ICP (Iterative Closest Point) algorithm is used for point-to-point iterative optimization. By continuously minimizing the distance error between point clouds, sub-pixel-level overlap is achieved. The iterative closest point algorithm is a point cloud registration algorithm based on iterative optimization. It updates the transformation matrix by finding the nearest point pair and minimizing the squared error. This is used to perform high-precision rotation and translation fine-tuning based on the initial guesses provided by coarse registration, achieving a leap in accuracy from millimeter to micrometer level.

[0040] The above implementation combines pose matching of the FPFH descriptor with precise localization of the ICP algorithm to achieve high-speed and high-precision matching between the point cloud model and the lightweight standard 3D model, avoiding the massive computational resource consumption caused by matching all data.

[0041] In some implementations, the residual change rate and / or the upper limit of the number of iterations are configured as convergence criteria so that the iterative nearest point algorithm stops in time when the accuracy requirement is met, thus ensuring matching efficiency.

[0042] Step 430: Determine the distribution of deviation features in the lightweight standard 3D model within the real-time point cloud model in the overlapping space; The overlap space is the three-dimensional coordinate space with the minimum sum of pixel coordinate deviations between the real-time point cloud model and the lightweight standard 3D model. Within this overlap space, the real-time point cloud model and the lightweight standard 3D model have the highest degree of overlap. The deviation feature distribution is composed of the positional deviation features (coordinate distances) between corresponding pixels in the real-time point cloud model and the lightweight standard 3D model. Each deviation feature (coordinate distance) is associated with the coordinates of a pixel in the real-time point cloud model and the coordinates of a pixel in the lightweight standard 3D model. When a pixel in the real-time point cloud model and its corresponding pixel in the lightweight standard 3D model coincide in position, their coordinate values ​​are the same, and the feature value of the deviation feature is 0.

[0043] Step 440: Automatically locate the point of interest region in the high-defect area of ​​the component based on the deviation characteristics.

[0044] In some embodiments of the present invention, step 440 specifically includes: Step 441: Read the pre-configured mask data from the lightweight standard 3D model, the mask data being used to define the defect detection shielding area of ​​the component; Step 442: Determine the areas on the component other than the defect detection shielding area as the high-incidence areas of defects. The high-incidence areas of defects include key areas such as solder joints, pin edges, and package boundaries of the component.

[0045] Step 443: Analyze the distribution of deviation characteristics in the high-incidence area of ​​the defect to locate the region of interest.

[0046] In some embodiments of the present invention, step 443 specifically includes: Step 443-10: Obtain the lower boundary of the pre-configured tolerance threshold range for determining the geometric deviation of the target component. and the upper boundary of the tolerance threshold range ; Step 443-20: Calculate the vertical distance from each sampling point on the surface of the real-time point cloud model within the interest area to the surface of the lightweight standard 3D model. The lower boundary of the tolerance threshold range and upper boundary Compare; Step 443-30: When any sampling point within the interest area satisfies When the sampling point is selected, it is marked as a qualified point.

[0047] In this embodiment, arrive The range is the range of random errors allowed by the manufacturing process, or the acceptable range.

[0048] Steps 443-40: When any sampling point within the region of interest satisfies When this happens, the sampling point is marked as a defect point.

[0049] In this embodiment, the vertical distance Greater than the upper boundary The situation was clearly determined to be either geometric deformation or structural damage.

[0050] Steps 443-50: When any sampling point within the interest area satisfies When this happens, the sampling point is marked as a suspected defect point; In this embodiment, arrive The range is a critical value range, which serves as a buffer zone to avoid a large number of false alarms caused by manufacturing tolerances or measurement fluctuations. The measurement fluctuations include depth measurement errors caused by light flicker in the detection environment, random noise from the sensor, or slight vibrations.

[0051] The embodiments of this invention employ an RGB-D fusion defect detection mode, enabling simultaneous monitoring of geometric deformation and surface texture anomalies. Here, RGB in RGB-D refers to the RGB texture features of the component surface extracted from real-time image data, and D in RGB-D refers to the depth features of the component surface extracted from depth data.

[0052] In some embodiments of the present invention, after steps 443-40, the method further includes: Steps 443-41: Perform cluster analysis on each sampling point on the surface of the real-time point cloud model to obtain several defect point clusters; Steps 443-42: Compare the number of cluster points in the defect point cluster with a preset defect point threshold; Steps 443-43: When the number of cluster points in any defect point cluster is greater than the defect point threshold, the region where the defect point cluster is located is configured as the region of interest.

[0053] In some embodiments of the present invention, after steps 443-50, the method further includes: Steps 443-51: Perform cluster analysis on each sampling point on the surface of the real-time point cloud model to obtain several clusters of suspected defect points. The clusters of suspected defect points consist of suspected defect points and defect points with a distance less than a preset clustering distance. Steps 443-52: Compare the number of cluster points in the suspected defect point cluster with the preset suspected defect point threshold; Steps 443-53: When the number of cluster points in any suspected defect point cluster is greater than the suspected defect point threshold, the region where the suspected defect point cluster is located is configured as a secondary verification region. Steps 443-54: Perform a secondary verification on the secondary verification area.

[0054] It should be understood that the clustering step for identified defects precedes the clustering step for suspected defects. Once a defect cluster is constructed, even if suspected defects exist within its region, they will not be reconstructed into a suspected defect cluster. In a practical implementation, after completing the defect clustering step, the region containing the constructed defect cluster can be configured as a shielded region, ensuring that suspected defects within these shielded regions are not included in the clustering scope during subsequent suspected defect clustering steps.

[0055] Preferably, in steps 443-41 and 443-51, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is used to cluster the defect points and the suspected defect points. In a specific implementation, the neighborhood radius and minimum number of points for the DBSCAN algorithm can be pre-configured as clustering parameters based on empirical data. The neighborhood radius should be greater than the sampling distance from the point cloud model. For example, the neighborhood radius can be configured as an integer multiple (e.g., 3 or 5 times the sampling distance) of the sampling points from the point cloud model.

[0056] Furthermore, the front-end sensing module includes a first sensing module and a second sensing module spaced apart. The second sensing module is a verification module and is also equipped with an image sensor and a depth sensor. The first sensing module acquires real-time image data and depth data of the component in a first imaging area, and the second sensing module acquires real-time image data and depth data of the component in a second imaging area. The second imaging area is located behind the first imaging area along the movement direction of the component.

[0057] In some embodiments of the present invention, the relative angle (i.e., imaging angle) between the second sensing module and the second imaging region is different from the relative angle between the first sensing module and the first imaging region. Steps 443-54 can be implemented in the following ways: The second sensing module is controlled to acquire the real-time image data and depth data of the component again; The suspected defect points in the secondary verification area are re-verified based on the re-acquired real-time image data and depth data. Based on the verification results, the region where the suspected defect point cluster is located is configured as a region of interest or a region of non-interest.

[0058] Furthermore, in the step of re-verifying the suspected defect points in the secondary verification area based on the reacquired real-time image data and depth data, when the vertical distance of any suspected defect point in the secondary verification area is... Still greater than the lower boundary When the suspected defect point is determined as a defect point; when the vertical distance of any suspected defect point in the secondary verification area is... Smaller than the lower boundary At that time, the suspected defect points are determined as qualified points.

[0059] Furthermore, in the step of configuring the region where the suspected defect point cluster is located as a region of interest or a region of non-interest based on the verification results, cluster analysis is performed again on the secondary verification region after the sampling point type is re-determined to construct defect point clusters, and defect point clusters in the secondary verification region with a number of cluster points greater than the defect point threshold are configured as regions of interest.

[0060] Step 500: Spatial mapping and fusion of texture features and depth data in the real-time image data within the region of interest.

[0061] In some embodiments of the present invention, the real-time point cloud model includes a mapping relationship between each two-dimensional coordinate point in the real-time image data and the three-dimensional spatial coordinates in the spatial coordinate system where the real-time point cloud model is located. Step 500 specifically includes: Step 510: Determine several target sampling points within the area of ​​interest; Step 520: Extract the RGB pixel gradient of the target sampling point from the real-time image data. As the texture feature of the component at the target sampling point, wherein Used to represent the coordinates of the target sampling point; The RGB pixel gradient These are vector values ​​based on the brightness or color differences between adjacent pixels in the image space, typically calculated using the Sobel or Canny operators. The RGB pixel gradient... It can be used as a core characteristic indicator for judging surface defects such as scratches, stains, and color variations.

[0062] In some implementations, step 520 specifically includes: Step 521: Calculate the RGB pixel gradient value of each two-dimensional coordinate point in the two-dimensional coordinate system of the real-time image data. ,in These are the coordinate values ​​in the two-dimensional coordinate system of the real-time image data; Step 522: Read each two-dimensional coordinate point from the real-time image data from the real-time point cloud model. The three-dimensional spatial coordinates of the real-time point cloud model in the spatial coordinate system The mapping relationship between them; Step 523: Based on the mapping relationship, calculate the RGB pixel gradient values ​​in the two-dimensional coordinate system. Mapped to the three-dimensional spatial coordinates in the spatial coordinate system of the real-time point cloud model Obtain the RGB pixel gradient .

[0063] Step 530: Extract the vertical distance of the target sampling point from the depth data. As the geometric deviation of the component at the target sampling point; The vertical distance from each sampling point on the surface of the real-time point cloud model within the area of ​​interest to the surface of the lightweight standard 3D model. The distance between the sampling points on the surface of the point cloud model and the surface projection points of the lightweight standard 3D model is a physical indicator used in this invention to quantify and measure the geometric deformation of components, such as flattening, bulging, and pitting.

[0064] Step 540: Weightedly fuse the texture features and the geometric deviation to obtain the joint probability of the component at the target sampling point: .

[0065] Using the technical solution described above, the system can identify complex composite defects. For example, a seemingly flat solder joint, although its height remains unchanged, may have a dark oxide layer on its surface. Using the aforementioned joint probability model, this type of composite defect can be effectively identified.

[0066] Furthermore, the joint probability Calculated using a pre-built joint probability model, prior to step 600, the following is also included: Step 010: Acquire a large amount of real-time image and depth data using laboratory data; Step 020: Construct a joint observation vector from the two sensors using the real-time image data and depth data: ; Step 030: Construct the joint observation vector using manual annotation. and its category tags The observed data sample pairs, among which ;; Step 040: Divide the observed data sample pairs into qualified datasets. and defect dataset ; Step 050: Calculate the probability density distribution of the qualified dataset using the maximum likelihood estimation method. and the probability density distribution of the defect dataset. ; Step 060: Define the joint probability function: .

[0067] In the above embodiments, This indicates that the test is successful. Indicates a defect.

[0068] Step 600: Defect identification is performed on the region of interest of the component using a joint probability model.

[0069] In some implementations, step 600 specifically includes: Step 610: Build Characteristic plane; Step 620: In the Plot the pass probability density curve on the characteristic plane and defect probability density curve ; Step 630: Solve for the qualified probability density curve and the defect probability density curve The equation of the intersection point: , Any intersection point in the intersection equation satisfy ; Step 640: Using the intersection equation as the decision boundary, the... On the characteristic plane One side is designated as the defective side. One side is determined to be the qualified side; Step 650: Construct the feature vector of the sampling points in the region of interest. ; Step 660: Identify the feature vector of the sampling points in the region of interest. In the The distribution of characteristic planes; Step 670: Feature vector of sampling points in the region of interest The distribution ratio between the defective side and the qualified side is used to determine whether the region of interest is a defective region.

[0070] like Figure 2 As shown, the second aspect of the present invention proposes an intelligent visual inspection device for component defects, including a front-end sensing module, a high-speed communication interface, and an edge controller. The front-end sensing module includes an image sensor for acquiring real-time image data of the component and a depth sensor for acquiring depth data of the component. The controller communicates with a back-end server through the high-speed communication interface to implement the intelligent visual inspection method for component defects as described in any one of the first aspects of the present invention.

[0071] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0072] As described above, these embodiments of the present invention do not exhaustively cover all details, nor do they limit the invention to the specific embodiments described. Clearly, many modifications and variations can be made based on the above description. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to effectively utilize the invention and its modifications. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A smart visual inspection method for component defects, characterized in that, include: Load the lightweight standard 3D model corresponding to the target component; Synchronously acquire real-time image and depth data of components on the conveying or clamping mechanism; A real-time point cloud model is generated based on the real-time image data and the depth data to describe the surface geometric features of the component. The real-time point cloud model is matched with the lightweight standard 3D model to locate the region of interest in the high-defect area of ​​the component. Spatial mapping and fusion are performed on the texture features and depth data in the real-time image data within the region of interest. Defects in the region of interest of the component are identified by a joint probability model.

2. The intelligent visual inspection method for component defects according to claim 1, characterized in that, The specific steps for synchronously acquiring real-time image and depth data of components on the conveying or clamping mechanism include: Receive the position signal from the servo driver to trigger synchronous sensor acquisition; The acquired raw depth image is subjected to median filtering for noise reduction and voxel-based downsampling. The lightweight feature data, after denoising and downsampling, is transmitted to the backend server via a high-speed communication interface.

3. The intelligent visual inspection method for component defects according to claim 1, characterized in that, The step of matching the real-time point cloud model with the lightweight standard 3D model to locate the region of interest in the high-defect area of ​​the component specifically includes: The pose matching between the real-time point cloud model and the lightweight standard 3D model is performed on a global scale using fast point feature histograms. The iterative nearest-point algorithm is used to achieve pixel-level spatial overlap within the local area determined by the pose matching; Determine the distribution of deviation features in the lightweight standard 3D model within the overlapping space of the real-time point cloud model. The point of interest region is automatically located based on the deviation characteristics distributed within the high-defect areas of the component.

4. The intelligent visual inspection method for component defects according to claim 3, characterized in that, The steps for automatically locating the point of interest region based on the deviation characteristics distributed within the high-defect area of ​​the component specifically include: Read pre-configured mask data from the lightweight standard 3D model, the mask data being used to define the defect detection shielding area of ​​the component; The areas on the component other than the defect detection shielding area are identified as the high-incidence areas of defects. The distribution of deviation characteristics in the high-incidence area of ​​the defect is analyzed to locate the region of interest.

5. The intelligent visual inspection method for component defects according to claim 4, characterized in that, The specific steps of analyzing the deviation feature distribution in the high-incidence defect area to locate the point of interest region include: Obtain the lower boundary of the pre-configured tolerance threshold range for determining the geometric deviation of the target component. and the upper boundary of the tolerance threshold range ; The vertical distance from each sampling point on the surface of the real-time point cloud model within the area of ​​interest to the surface of the lightweight standard 3D model. The lower boundary of the tolerance threshold range and upper boundary Compare; When any sampling point within the interest area satisfies When this happens, the sampling point is marked as a suspected defect point; When any sampling point within the interest area satisfies When this happens, the sampling point is marked as a defect point.

6. The intelligent visual inspection method for component defects according to claim 5, characterized in that, After the step of marking the sampling points as defect points, the method further includes: Cluster analysis is performed on each sampling point on the surface of the real-time point cloud model to obtain several defect point clusters; The number of cluster points in the defect point cluster is compared with a preset defect point threshold. When the number of cluster points in any defect point cluster is greater than the defect point threshold, the region where the defect point cluster is located is configured as the point of interest region.

7. The intelligent visual inspection method for component defects according to claim 6, characterized in that, After the step of marking the sampling points as suspected defect points, the method further includes: Cluster analysis is performed on each sampling point on the surface of the real-time point cloud model to obtain several clusters of suspected defect points. The clusters of suspected defect points consist of suspected defect points and defect points whose distance is less than a preset clustering distance. The number of cluster points in the suspected defect point cluster is compared with a preset suspected defect point threshold. When the number of cluster points in any suspected defect point cluster is greater than the suspected defect point threshold, the area where the suspected defect point cluster is located is configured as a secondary verification area. Perform a secondary verification on the secondary verification area.

8. The intelligent visual inspection method for component defects according to claim 1, characterized in that, The real-time point cloud model includes the mapping relationship between each two-dimensional coordinate point in the real-time image data and the three-dimensional spatial coordinates in the spatial coordinate system of the real-time point cloud model. The step of spatially mapping and fusing the texture features and depth data in the real-time image data within the region of interest specifically includes: Within the area of ​​interest, determine a number of target sampling points; Extract the RGB pixel gradient of the target sampling point from the real-time image data. As the texture feature of the component at the target sampling point, wherein Used to represent the coordinates of the target sampling point; Extract the vertical distance of the target sampling point from the depth data. As the geometric deviation of the component at the target sampling point; The joint probability of the component at the target sampling point is obtained by weighted fusing of the texture features and the geometric deviation: 。 9. The intelligent visual inspection method for component defects according to claim 8, characterized in that, The joint probability The method, calculated using a pre-built joint probability model, includes the following steps prior to the step of identifying defects in the region of interest of the component using the joint probability model: Acquire large amounts of real-time image and depth data using laboratory data. The joint observation vector of the two sensors is constructed using the real-time image data and depth data: ； The joint observation vector is constructed through manual annotation. and its category tags The observed data sample pairs, among which ; The observed data sample pairs are divided into qualified datasets. and defect dataset ; The probability density distribution of the qualified dataset was calculated using the maximum likelihood estimation method. and the probability density distribution of the defect dataset. ; Define the joint probability function: 。 10. An intelligent visual inspection device for component defects, characterized in that, The device includes a front-end sensing module, a high-speed communication interface, and an edge controller. The front-end sensing module includes an image sensor for acquiring real-time image data of components and a depth sensor for acquiring depth data of components. The controller communicates with a back-end server through the high-speed communication interface to implement the intelligent visual detection method for component defects as described in any one of claims 1-9.

Images