Machine vision-based riveting post-quality detection method and riveting equipment

Abstract

The application discloses a riveting quality detection method based on machine vision and a riveting device, which combines photometric stereo vision technology and single-point laser height measurement, calculates surface normal and albedo through acquisition of a plurality of illumination image sequences, calibrates a three-dimensional topography reconstruction process by using a reference height value obtained by a single-point laser displacement device, and generates a high-precision calibrated height map, so that the method effectively overcomes the high reflection and shadow interference of the rivet surface, significantly improves the detection capability of micro-cracks, pits and other defects, and further realizes reliable quality judgment through multi-dimensional feature fusion extraction. In this way, the three-dimensional geometric topography and surface optical characteristics of the object are effectively separated, the interference of highlights and shadows on detection is fundamentally eliminated, and therefore the micro-defects such as cracks and pits which are difficult to be found by traditional visual methods can be accurately identified and quantified, and the precision and reliability of the riveting quality detection are significantly improved.

Description

Technical Field

[0001] This application relates to the field of quality inspection, and more specifically, to a machine vision-based method and riveting equipment for post-riveting quality inspection. Background Technology

[0002] Riveting, as a crucial permanent joining process, plays a vital role in many industrial manufacturing sectors. During the riveting process, fluctuations in materials, equipment, or process parameters can lead to micro-defects such as cracks and pits on the rivet surface. These defects directly affect the reliability of the connection and the overall quality and safety of the product. Therefore, precise and efficient automated quality inspection of the riveted rivets is a necessary step to ensure production quality and mitigate potential risks.

[0003] To improve riveting quality, existing technologies have explored various approaches. For example, Chinese patent CN114192725B discloses an automated method for preventing missed or incorrect riveting. This method monitors air pressure changes to determine if there are procedural errors such as missed or incorrect riveting, focusing on verifying the presence and correctness of materials before the riveting process begins. However, this type of method cannot detect surface morphology defects formed after riveting. For surface defect detection, the industry typically uses machine vision technology, but traditional visual inspection methods face significant challenges when applied to highly reflective rivet surfaces. Rivets usually have smooth, curved metal surfaces that produce strong specular reflections under unidirectional lighting, creating bright glare areas and deep shadows. This uneven lighting severely interferes with image acquisition. The limited dynamic range of industrial cameras makes it difficult to simultaneously capture details in both bright and shadow areas, resulting in the loss of information about minor defects such as cracks and dents in these areas. Therefore, traditional visual algorithms struggle to effectively overcome the interference of high reflectivity and shadows, leading to high rates of missed and false detections of minor defects, failing to meet the increasingly stringent industrial quality control requirements.

[0004] Therefore, we look forward to an optimized machine vision-based quality inspection solution for riveting. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a machine vision-based method and equipment for post-riveting quality inspection. This method combines photometric stereo vision technology with single-point laser height measurement. By acquiring multi-illumination image sequences, it calculates the surface normal and albedo, and uses the reference height value obtained from the single-point laser displacement device to calibrate the three-dimensional topography reconstruction process, generating a high-precision calibrated height map. This method effectively overcomes the interference of high reflectivity and shadows on the rivet surface, significantly improving the detection capability for defects such as microcracks and dents. Furthermore, reliable quality judgment is achieved through multi-dimensional feature fusion extraction. In this way, the three-dimensional geometric topography and surface optical properties of the object are effectively separated, fundamentally eliminating the interference of highlights and shadows on detection. This allows for accurate identification and quantification of minute defects such as cracks and dents that are difficult to detect using traditional vision methods, significantly improving the accuracy and reliability of post-riveting quality inspection.

[0006] According to one aspect of this application, a machine vision-based method for post-riveting quality inspection is provided, comprising:

[0007] In response to the detection trigger signal, the original image sequence and reference height value of the detected rivet are acquired using an industrial camera and a single-point laser displacement device;

[0008] Image preprocessing and surface normal calculation are performed on the original image sequence of the rivet to obtain the X-direction normal map, Y-direction normal map, Z-direction normal map and RGB albedo map;

[0009] Based on the reference height value, the X-direction normal map, Y-direction normal map, and Z-direction normal map are subjected to three-dimensional topography correction and reconstruction to obtain the calibrated height map;

[0010] Multi-dimensional defect features were extracted from the calibrated height map, RGB albedo map, X-direction normal map, and Y-direction normal map to obtain a candidate defect list;

[0011] Based on the candidate defect list, defects are classified and finally judged to obtain the final quality inspection result, which is used to indicate whether the quality of the inspected rivet is qualified.

[0012] According to another aspect of this application, a riveting post-riveting quality inspection device based on machine vision is provided, comprising:

[0013] The information acquisition module is used to acquire the original image sequence and reference height value of the rivet being detected through an industrial camera and a single-point laser displacement device in response to the detection trigger signal.

[0014] The image processing module is used to perform image preprocessing and surface normal calculation on the original image sequence of the rivet being inspected to obtain the X-direction normal map, Y-direction normal map, Z-direction normal map and RGB albedo map;

[0015] The 3D topography correction and reconstruction module is used to perform 3D topography correction and reconstruction on the X-direction normal map, Y-direction normal map and Z-direction normal map based on the reference height value to obtain the calibrated height map;

[0016] The multi-dimensional defect feature extraction module is used to extract multi-dimensional defect features from the calibrated height map, RGB albedo map, X-direction normal map and Y-direction normal map to obtain a candidate defect list;

[0017] The quality inspection module is used to classify and make final judgments on the candidate defect list to obtain the final quality inspection result, which indicates whether the quality of the inspected rivet is qualified.

[0018] Compared with existing technologies, this application provides a machine vision-based method and riveting equipment for post-riveting quality inspection. It combines photometric stereo vision technology with single-point laser height measurement. By acquiring multi-illumination image sequences, it calculates surface normals and albedo, and uses reference height values ​​obtained from a single-point laser displacement device to calibrate the three-dimensional topography reconstruction process, generating a high-precision calibrated height map. This method effectively overcomes the interference of high reflectivity and shadows on the rivet surface, significantly improving the detection capability for defects such as microcracks and dents. Furthermore, it achieves reliable quality judgment through multi-dimensional feature fusion extraction. In this way, the three-dimensional geometric topography and surface optical properties of the object are effectively separated, fundamentally eliminating the interference of highlights and shadows on detection. This allows for accurate identification and quantification of minute defects such as cracks and dents that are difficult to detect using traditional vision methods, significantly improving the accuracy and reliability of post-riveting quality inspection. Attached Figure Description

[0019] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 This is a flowchart of a machine vision-based post-riveting quality inspection method according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of the data flow of a machine vision-based post-riveting quality inspection method according to an embodiment of this application;

[0022] Figure 3 This is a block diagram of a machine vision-based riveting post-riveting quality inspection riveting device according to an embodiment of this application. Detailed Implementation

[0023] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0024] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0025] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0026] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0027] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0028] The technical solution of this application proposes a machine vision-based method for quality inspection after riveting. Figure 1 This is a flowchart of a machine vision-based post-riveting quality inspection method according to an embodiment of this application. Figure 2 This is a system architecture diagram of a machine vision-based post-riveting quality inspection method according to an embodiment of this application. Figure 1 and Figure 2As shown, the machine vision-based post-riveting quality inspection method according to an embodiment of this application includes the following steps: S1, in response to a detection trigger signal, acquiring the original image sequence and reference height value of the rivet to be inspected using an industrial camera and a single-point laser displacement device; S2, performing image preprocessing and surface normal calculation on the original image sequence of the rivet to obtain an X-direction normal map, a Y-direction normal map, a Z-direction normal map, and an RGB albedo map; S3, based on the reference height value, performing three-dimensional topography correction and reconstruction on the X-direction normal map, the Y-direction normal map, and the Z-direction normal map to obtain a calibrated height map; S4, extracting multi-dimensional defect features from the calibrated height map, the RGB albedo map, the X-direction normal map, and the Y-direction normal map to obtain a candidate defect list; S5, classifying and finally judging defects based on the candidate defect list to obtain a final quality inspection result, wherein the final quality inspection result is used to indicate whether the quality of the inspected rivet is qualified.

[0029] Specifically, in S1, in response to a detection trigger signal, the original image sequence and reference height value of the rivet being inspected are acquired using an industrial camera and a single-point laser displacement device. It should be understood that the acquisition of the original image sequence is crucial for separating the geometric normal information and material albedo information of the rivet surface through subsequent photometric stereoscopic calculations. This is essential for overcoming the loss of defect information caused by high reflectivity and shadows in traditional visual inspection and is the foundation for realizing the three-dimensional morphological feature analysis of minute defects (such as cracks and dents). The acquisition of the reference height value is to address the inherent uncertainty of the integral constant and the accumulated error problem generated during the photometric stereoscopic integral reconstruction of the three-dimensional morphology. This reference height value serves as a high-precision anchor point for global calibration of the reconstructed relative height field, thereby ensuring that the final height map has absolute accuracy at the real physical scale, which is crucial for the quantitative assessment of defects.

[0030] The original image sequence of the rivets being inspected refers to a set of images captured by an industrial camera on the same rivet under illumination from a single light source in different directions and with different spectra. Specifically, in the technical solution of this application, the original image sequence of the rivets being inspected includes images of the upper red rivet, upper green rivet, upper blue rivet, lower red rivet, lower green rivet, lower blue rivet, left red rivet, left green rivet, left blue rivet, right red rivet, right green rivet, and right blue rivet. The reference height value is a scalar floating-point number representing the absolute physical height of a specific reference point on the rivet surface in the device coordinate system, measured by a single-point laser displacement device.

[0031] In practice, once the rivet to be inspected is conveyed to the preset inspection station and stabilizes, the position sensor or upper control system (such as a PLC) in the system will send a detection trigger signal. This signal simultaneously activates the industrial camera and the single-point laser displacement device. The industrial camera performs high-speed sequential imaging of the rivet under illumination conditions of multiple different directions and spectra. Specifically, the system sequentially activates red (R), green (G), and blue (B) light sources located above, below, to the left, and to the right of the rivet. The industrial camera performs one exposure each time a single light source is illuminated, thereby acquiring a series of images. At the same time, the single-point laser displacement device emits a laser beam towards a preset reference point on the surface of the rivet (usually the highest point of the rivet) and receives the reflected light. The absolute height of this point is accurately calculated using principles such as triangulation or confocal methods, and this measured value is recorded as the reference height value. This series of image acquisition and single-point height measurement process is completed in a very short time to ensure that the rivet to be inspected does not shift during this period.

[0032] Specifically, in step S2, image preprocessing and surface normal calculation are performed on the original image sequence of the rivet to be detected to obtain X-direction normal map, Y-direction normal map, Z-direction normal map, and RGB albedo map. It should be understood that in machine vision, the brightness value of each pixel in an image is the result of the combined effects of the object's surface shape, material (albedo), and illumination direction. For highly reflective surfaces, even small changes in illumination can lead to drastic highlights and shadows in the image, which can mask or distort subtle brightness changes caused by surface defects (such as cracks and pits), making it extremely difficult to detect defects directly from the original image. Therefore, in the technical solution of this application, by performing image preprocessing and surface normal calculation on the original image sequence of the rivet to be detected, the original image data, which is strongly correlated with illumination conditions, is converted into a representation that describes the inherent physical properties of the object's surface and is independent of illumination, laying a solid data foundation for subsequent high-precision three-dimensional morphology reconstruction and multi-dimensional defect analysis. Specifically, by employing the photometric stereo method, two key intrinsic properties can be derived from multiple images under different lighting conditions: the first is the surface normal, which describes the three-dimensional orientation of the surface at each point and directly reflects the object's geometry; the second is the albedo, which describes the surface's ability to reflect light and reflects the object's intrinsic color and material. In this way, subsequent defect detection algorithms can be performed in the pure geometric domain (normal map) and the material domain (albedo map), respectively, thus completely eliminating the interference of highlights and shadows and greatly improving the robustness and accuracy of detection.

[0033] The X-direction normal map, Y-direction normal map, and Z-direction normal map are three independent grayscale images. The grayscale value of each pixel in the image represents the component of the unit normal vector at the corresponding point on the surface of the measured object along the X, Y, and Z coordinate axes. The RGB albedo map is a color image that represents the intrinsic reflectance of the object's surface to red, green, and blue light after excluding all lighting and geometric effects, i.e., the true color of the object.

[0034] The specific execution method of this step is based on the principle of photometric stereo vision, and combines multispectral information for step-by-step processing and synthesis. Specifically, the red, green and blue color channels are first calculated independently, and then the calculation results are selectively synthesized.

[0035] First, the red channel is processed. This sub-step includes: extracting the upper red rivet image, lower red rivet image, left red rivet image, and right red rivet image; calculating the average illumination intensity of the upper red rivet image, lower red rivet image, left red rivet image, and right red rivet image to obtain the R-channel albedo map; and processing the upper red rivet image, lower red rivet image, left red rivet image, and right red rivet image using a photometric stereo method to obtain the R-channel X-direction normal map, R-channel Y-direction normal map, and R-channel Z-direction normal map. Specifically, for any pixel (x, y) in the image, the R-channel albedo ρ of that point is first estimated by averaging its brightness values ​​in the four red channel images. r (x,y); subsequently, based on the Lambertian reflection model I=ρ(L·N), where I is pixel brightness, ρ is albedo, L is the unit illumination direction vector, and N is the unit surface normal vector. Since ρ r Given that L is also a pre-calibrated known quantity for each image, we can construct a normal vector N = [N... x N y N z ] T This is an overdetermined system of linear equations. For example, for four images, the system of equations can be expressed as follows:

[0036] [I u p / ρ r ] = [L u p T ][N x ]

[0037] [I d own / ρ r ] = [L d own T ][N y ]

[0038] [I l eft / ρr ] = [L l eft T ][N z ]

[0039] [I r ight / ρ r ] = [L r ight T ]

[0040] This system of equations can be solved using the least squares method, thus obtaining the normal vector N(x,y) for that pixel. Then, N(x,y) is used to solve for all pixels. x N y N z The components are combined separately to form the normal map of the R channel in the X, Y, and Z directions;

[0041] Next, the exact same processing flow was applied to the green and blue channels. Specifically, the images of the top green rivet, bottom green rivet, left green rivet, and right green rivet were extracted; the G-channel albedo map, G-channel X-direction normal map, G-channel Y-direction normal map, and G-channel Z-direction normal map of the top green rivet, bottom green rivet, left green rivet, and right green rivet images were calculated; the images of the top blue rivet, bottom blue rivet, left blue rivet, and right blue rivet images were extracted; and the B-channel albedo map, B-channel X-direction normal map, B-channel Y-direction normal map, and B-channel Z-direction normal map of the top blue rivet, bottom blue rivet, left blue rivet, and right blue rivet images were calculated.

[0042] Finally, the results are synthesized and selected. Specifically, the R-channel albedo map, G-channel albedo map, and B-channel albedo map are synthesized to obtain the RGB albedo map; and the G-channel X-direction normal map, G-channel Y-direction normal map, and G-channel Z-direction normal map are used as the X-direction normal map, Y-direction normal map, and Z-direction normal map, respectively. Synthesizing the RGB albedo map involves merging the three single-channel albedo maps along the color dimension to form a standard color image. The normal map selection uses only the normal map calculated from the green channel as the final output. This is generally because most industrial camera sensors (such as CMOS / CCDs using Bayer filters) have the highest quantum efficiency and best signal-to-noise ratio for green light, thus the normal data calculated based on the green channel image also has the best accuracy and stability.

[0043] Taking the scheme of this application as an example, the system retrieves 12 original images acquired in the previous stage from memory. First, the red channel is processed; the system extracts four red light images: top, bottom, left, and right. For a pixel at image coordinates (256, 312), its brightness value (0-255) in the four images may be 220, 45, 150, and 95, respectively. The system calculates its R-channel albedo value as (220+45+150+95) / 4 = 127.5. Subsequently, using the calibrated four red light source direction vectors and these four brightness values, the normal vector of that point is obtained by solving a least-squares problem, for example... The system stores the values ​​0.15, -0.32, and 0.93 (after normalization and type conversion) at positions (256, 312) of the R channel X, Y, and Z normal maps, respectively. This calculation is performed in parallel for all pixels. Next, the system processes the green and blue channel images in the same way to obtain the G and B channel albedo maps and normal maps. Finally, the system merges the three single-channel albedo maps (R, G, and B) into a single RGB color albedo map and outputs the G channel X, Y, and Z normal map as the final normal map, which is then passed to the next processing module.

[0044] Specifically, in step S3, based on the reference height value, the X-direction normal map, Y-direction normal map, and Z-direction normal map are reconstructed using three-dimensional topography correction to obtain a calibrated height map. It should be understood that a simple normal map only provides the local orientation of each point on the object's surface. To identify and quantify three-dimensional defects such as pit depth and scratch width, this normal information must be integrated and reconstructed into a height map. However, any integration-based reconstruction method essentially only yields a relative height map, whose overall height has an unknown floating offset (i.e., the integration constant), and may introduce cumulative errors such as low-frequency distortion during the integration process. This makes direct quantitative evaluation of defects from the relative height map unreliable. Therefore, in the technical solution of this application, a reference height value precisely measured by a single-point laser displacement device is introduced as an absolute benchmark to accurately correct and calibrate this floating, potentially distorted relative height map, thereby generating a calibrated height map with absolute physical meaning and high precision in the global coordinate system, providing a solid data foundation for subsequent defect feature extraction and final quality assessment.

[0045] In practice, this step is a multi-stage calculation process, which includes three core sub-steps: relative height field integration, absolute reference parameter calculation, and global height calibration.

[0046] First, the first sub-step is performed: Fourier transform-based relative height field integration is performed on the X-direction normal map, Y-direction normal map, and Z-direction normal map to obtain the relative height map. During this process, the system first determines the relative height map from the normal map (N... x N y N z The gradient fields of the surface in the X and Y directions are calculated. The specific calculation process is expressed by the following formula:

[0047]

[0048] Here, p(x, y) represents the gradient field in the X direction, and q(x, y) represents the gradient field in the Y direction. Subsequently, the system uses a Fourier transform to convert this gradient field to the frequency domain, performs integration in the frequency domain, and then uses an inverse Fourier transform to convert it back to the spatial domain, thereby reconstructing the surface topography. This method of integration in the frequency domain has higher computational efficiency and better global consistency compared to path integration in the spatial domain, and can effectively suppress high-frequency noise. The values ​​of each point in the obtained relative height map only represent their relative height relationships.

[0049] Next, the second sub-step is executed: based on the reference point pixel coordinates and the system scale factor, the absolute height offset is obtained by solving the absolute reference affine transformation parameters relative to the relative height map. Specifically, a positive direction neighborhood is defined in the relative height map with the reference point pixel coordinates as the center; a dynamic peak search is performed on the positive direction neighborhood to obtain the actual peak coordinates; and a local expected height estimation based on a Gaussian kernel is performed on the actual peak coordinates to obtain a robust reference relative height as the relative height value. That is, firstly, the random offset of the workpiece position is actively compensated, and the true height peak point under the current measurement is dynamically searched and located in the neighborhood around the preset reference point. Then, with the true peak point as the center, the height values ​​in the local neighborhood are weighted and averaged using a Gaussian kernel function to obtain a robust reference relative height value that suppresses random noise and is more statistically significant, denoted as H. ref After obtaining the robust relative height value, the offset is calculated. Specifically, based on the system scale factor, the offset between the relative height value and the reference height value is calculated using the following formula to obtain the absolute height offset:

[0050] O offset =H ref -S scale ·h relative

[0051] Among them, H ref h is the reference height value. relative S is the relative height value. scaleHere, the absolute height offset is a scalar value representing the height difference between the zero datum plane of the relative height map and the zero plane of the real physical world coordinate system.

[0052] Specifically, it should be understood that if the calibration benchmark in the technical solution of this application is based on a deterministic dependence on a single pixel (that is, assuming a static and precise mapping relationship between the laser height measurement point and the preset pixel coordinates in the reconstructed height map), many technical problems will arise in practical implementation. Specifically, in real industrial production scenarios, physical uncertainties such as slight positioning deviations of the workpiece, production line vibrations, or equipment thermal deformation cause the position of the actual height peak to drift dynamically. Directly accepting the relative height value of the preset coordinates will introduce significant errors due to missing the true peak value. In addition, sensor noise or minor optical defects on the surface may form accidental outliers on a single pixel of the reconstructed height map. This mechanism has no fault tolerance for such random noise. Once the preset coordinates happen to fall into an outlier, a completely incorrect relative height will be extracted, causing the absolute height calibration of the entire three-dimensional shape to fail. Its inherent single-point fragility makes it unable to meet the requirements of high-reliability quality inspection.

[0053] To overcome the aforementioned shortcomings, this preferred technical solution introduces a robust calibration method based on dynamic peak finding and local expectation estimation. The first step of this method is to perform dynamic peak localization and region of interest definition. It should be understood that random offsets in the physical position of the workpiece must be actively compensated for, rather than passively accepting a potentially invalid static coordinate system. The execution process begins by defining a search neighborhood Ω, centered on the preset reference point pixel coordinates, representing a reasonable range of peak drift.

[0054] Subsequently, an optimization search is performed on the relative height map within this neighborhood to locate the maximum height value. This dynamically and adaptively identifies the most accurate peak pixel coordinates of the current workpiece surface for each independent measurement. It should be understood that the output peak coordinates can track minute displacements of the workpiece in real time, fundamentally eliminating systematic biases introduced by positional mismatches. This process can be expressed by the following formula:

[0055]

[0056] whereΩ={(i,j)∣|ii ref |≤R,|jj ref |≤R}

[0057] Next, a local expected height estimation based on a Gaussian kernel is performed. Even if the true peak coordinates are located, the value of this single point may still be contaminated by local high-frequency noise. A mechanism must be established to suppress this random interference and extract a more statistically meaningful reference height. The execution process involves constructing a two-dimensional Gaussian kernel function G centered on the actual peak coordinates obtained in the first step. This kernel function is then convolved with the local region around the peak point using the relative height map to calculate a weighted average of the height values ​​of all pixels in that neighborhood. The purpose of this execution is to transform the extraction of the reference height from sampling a single discrete point to a probabilistic expectation estimation of the local surface topology. The result is a robust reference relative height as the relative height value. This value incorporates neighborhood information and smooths out the random fluctuations of individual pixels through Gaussian weighting. The result exhibits strong insensitivity to noise and small outliers, significantly improving the signal-to-noise ratio and stability of the reference benchmark. This process can be expressed by the following formula:

[0058]

[0059] Where σ is the Gaussian kernel standard deviation.

[0060] In this way, by introducing dynamic peak finding and local expectation estimation, the calibration process is transformed from a fragile dependence on discrete single points to a robust statistical analysis of local geometric features. This significantly enhances the stability and accuracy of the 3D topography reconstruction algorithm in the face of physical positioning tolerances and sensor noise. Therefore, it ensures that even in uncertain industrial environments, the system can still generate high-precision, highly repeatable 3D measurement data, providing a solid data foundation for the reliable identification and quantitative evaluation of subsequent minor defects (such as cracks and dents), thereby fundamentally improving the overall performance and reliability of the automated quality inspection system.

[0061] Subsequently, the third sub-step is performed: based on the absolute height offset and the system scale factor, a global pixel-level absolute height calibration is performed on the relative height map to obtain the calibrated height map. In this process, the relative height value H of each pixel is first... relative (i,j) is obtained through the system scale factor S scale Convert to physical scale, then add the absolute height offset O calculated in the previous step. offset After this global, pixel-level affine transformation, the final calibrated height map shows that each pixel value represents the actual physical height of the corresponding point on the rivet surface in the device coordinate system. Specifically, this process is expressed by the formula:

[0062] H calibrated (i,j)=S scale ·H relative (i,j)+O offset

[0063] Among them, S scale The system scale factor is 0, and the absolute height offset is 0. offset H relative (i,j) represents the relative height value of the relative height map at pixel position (i,j).

[0064] Taking the scheme of this application as an example, the system receives the X, Y, and Z direction normal maps output by the previous module, as well as the reference height value H measured by the laser displacement device. ref = 9.523mm. First, the system performs a Fourier transform-based integration on the normal map to generate a relative height map H. relative Assuming the preset reference point pixel coordinates are (250, 250), the system scale factor S scale The value is 0.005 mm / unit. The system performs a dynamic peak search within a 10x10 neighborhood centered at (250, 250), finding the actual peak at (252, 249). Subsequently, a Gaussian kernel is used to perform a weighted average of the local height centered at (252, 249), yielding a robust reference relative height value of 1850.2. Next, the system calculates the absolute height offset: O offset =9.523 - 1850.2 * 0.005 = 0.272 mm. Finally, the system iterates through each pixel (i, j) in the relative height map and applies the formula:

[0065] H calibrated (i,j)=H relative (i,j)*0.005+0.272

[0066] Finally, a calibrated height map is generated, which is then passed to the subsequent defect extraction module for analysis.

[0067] Specifically, in step S4, multi-dimensional defect feature extraction is performed on the calibrated height map, RGB albedo map, X-direction normal map, and Y-direction normal map to obtain a candidate defect list. It should be understood that defects on rivet surfaces are diverse, with varying physical manifestations. For example, geometric defects such as cracks, pits, and protrusions are essentially local abrupt changes in three-dimensional morphology, and are therefore most prominent on the calibrated height map. Material defects such as oil stains, rust, and material color differences manifest as differences in surface optical reflectance properties, and can only be accurately identified in the RGB albedo map, which excludes the influence of illumination. Furthermore, some very shallow scratches or minor morphological changes, although not noticeable on the height map, can cause a sharp change in the surface normal direction, thus forming a strong anomalous signal in the normal map. Any single-dimensional information may be insensitive to a certain type of defect. Therefore, in the technical solution of this application, by parallel analysis of these mutually orthogonal data dimensions, and by comprehensively utilizing multi-dimensional information such as shape, color, and orientation, a complementary detection system is constructed to capture all possible defects to the maximum extent, avoid missed detections due to single information dimensions, and thus provide a high-recall set of candidate defects for the final quality judgment.

[0068] In practice, the first step is to extract geometric defects from the calibrated height map. This process involves calculating the Gaussian curvature and average curvature of the calibrated height map. Since geometric defects such as cracks and pits appear as areas with extremely high local curvature, the system identifies all geometrically abnormal areas as candidate geometric defects by performing threshold segmentation on the calculated curvature map.

[0069] Next, spectral pseudocolor extraction is performed on the RGB albedo map to identify spectral pseudocolor defects. This process specifically involves analyzing the color information of the RGB albedo map. Specifically, the system first establishes a color model for standard metallic materials; then, for each pixel in the image, the system analyzes its color characteristics, such as calculating its chromaticity coordinates or converting it to a color space like HSV. Any pixel area deviating from this standard color model, such as appearing yellowish due to oil stains or dark reddish due to oxidation, is marked as a spectral anomaly area.

[0070] Furthermore, normal anomaly extraction is performed on the X-direction and Y-direction normal maps to obtain normal anomaly defects. Specifically, this process involves calculating the gradient of the normal map, i.e., the vector field composed of the X / Y-direction normal maps. Surface discontinuities, such as crack edges, can cause drastic changes in the normal direction, thus forming strong signals on the normal gradient map. Subsequently, the system performs thresholding and morphological operations (such as connected component analysis and denoising) on ​​this gradient map to identify linear or point-like normal anomaly regions.

[0071] Subsequently, feature fusion and candidate generation are performed on geometric defects, spectral false-color defects, and normal anomaly defects to obtain the candidate defect list. This process is not a simple merging, but rather a spatial correlation analysis of the anomalous regions from the three sources. For example, the system performs the following logical judgment: if a region has both geometric and normal anomalies but a normal spectrum, it is highly likely to be a real crack or pit; if a region has spectral anomalies but normal geometry and normality, it is highly likely to be a false defect, such as oil stains or discoloration. Finally, the system stores regions that meet the conditions for real defects and their attributes, including location, size, depth, length, curvature value, and spectral features, into the candidate defect list. Here, the candidate defect list is a data structure collection, where each element represents a potential defect that needs further confirmation and contains preliminary attribute information such as the defect's location and size.

[0072] Taking the scheme of this application as an example, suppose there is a tiny dent on a rivet being inspected, and there is slight rust at the bottom of the dent. The system activates the multi-dimensional feature extraction module. First, the geometric defect extraction submodule calculates the curvature of the calibrated height map. The high curvature value at the dent exceeds a preset threshold, and the area is accurately marked as a geometric defect. At the same time, the spectral pseudo-color extraction submodule processes the RGB albedo map. It establishes the silver-gray color of most of the rivet area as a standard color model and finds that the color at the location corresponding to the dent is a reddish-brown that is significantly different from the model. Therefore, this area is also marked as a spectral pseudo-color defect. Meanwhile, the normal anomaly extraction submodule, when analyzing the normal map, finds that the normal direction of the edge region of the dent changes drastically, and its gradient value far exceeds the normal threshold. Therefore, this edge ring region is also marked as a normal anomaly defect. Finally, the feature fusion submodule receives these three results and finds that they highly overlap in space. The system judges according to the built-in rules that the area has geometric, normal, and spectral anomalies, which meets the definition of a serious defect. The system generates an entry for the region, recording its center coordinates, area, maximum depth calculated from the height map, and "reddish-brown" spectral characteristics. This entry is added to the candidate defect list, awaiting final classification and judgment.

[0073] Specifically, in step S5, defects are classified and finally judged based on the candidate defect list to obtain the final quality inspection result, which indicates whether the quality of the inspected rivet is qualified. It should be understood that the candidate defect list generated in the previous step may contain actual serious defects, acceptable minor flaws, or even some non-critical pseudo-defects. Without classification and judgment, this raw data cannot directly guide production decisions. Therefore, in the embodiments of this application, a configurable quality control rule base is introduced to rigorously examine the quantitative characteristics of each candidate defect, determine whether it exceeds the acceptable tolerance range, and finally integrate the judgment results of all defects to give an authoritative conclusion on the overall quality of the rivet, thereby realizing a closed loop of automated quality control. That is, according to preset industrial quality standards, all potential defects detected are accurately classified and their severity assessed, and based on this, a final binary (qualified / unqualified) judgment is made on the overall quality status of the inspected rivet.

[0074] In practice, the system first iterates through each candidate defect in the candidate defect list. During this process, the system processes each defect entry in the list generated in the previous step, obtaining all the quantitative feature information it contains, such as location, size, depth, length, curvature value, spectral features, etc.

[0075] Next, the quantitative characteristics of the candidate defects are compared with preset thresholds in the quality control rule base. During this process, the system loads a predefined quality control rule base, which defines detailed acceptance criteria for different types of defects (such as cracks and dents). For example, this rule base might contain the following sample rules: for "crack" defects, the acceptance criteria are: minimum length not exceeding 0.2 mm, maximum width not exceeding 0.05 mm, and minimum depth not exceeding 0.01 mm; for "dent" defects, the acceptance criteria are: minimum diameter not exceeding 0.1 mm and minimum depth not exceeding 0.02 mm. The system then compares the actual measured values ​​of the currently traversed candidate defects one by one with these criteria.

[0076] Then, based on the comparison results, each candidate defect is classified. That is, based on the comparison results of the previous step, the system assigns a classification label to each candidate defect. For example, if the measured depth of a pit is 0.03mm, exceeding the 0.02mm threshold in the rule base, then the defect will be classified as "excessively deep pit". If all features are within the acceptable range, it may be classified as "acceptable" or "acceptable defect".

[0077] Subsequently, a comprehensive final quality inspection result is generated. During this process, the system summarizes the classification results of all candidate defects. The judgment logic is as follows: if any candidate defect is classified as unqualified (e.g., "excessively long crack," "excessively deep dent," etc.), the final quality inspection result of the entire inspected rivet is judged as unqualified; only when the candidate defect list is empty, or when all candidate defects in the list are classified as qualified or acceptable, will the final quality inspection result be judged as qualified.

[0078] Finally, a detailed defect report is generated when necessary. If the final quality inspection result is non-compliant, the system will generate a detailed defect report, which will clearly list the details of all non-compliant defects, including the defect type, its precise location in the image coordinate system and 3D physical coordinate system, and specific dimensional measurements. This report can be used for subsequent process analysis and quality traceability, and can be highlighted on the human-machine interface to remind operators.

[0079] Taking the solution of this application as an example, the system receives a list of candidate defects containing one defect entry. This entry describes a pit with the following quantified characteristics: center coordinates (301, 415), diameter 0.15 mm, and depth 0.025 mm. The system loads a quality control rule base, where the rules for "pit" are: minimum diameter ≤ 0.1 mm, minimum depth ≤ 0.02 mm. The system begins comparison and finds that the measured diameter of the pit (0.15 mm) is greater than 0.1 mm, and the measured depth (0.025 mm) is greater than 0.02 mm. Therefore, this defect is classified as "pit over-diameter" and "pit over-deep". Due to the presence of a defect, the system immediately determined the final quality inspection result of the rivet to be unqualified. Subsequently, the system generated a defect report, which included: "Defect type: pit; Location: (301, 415); Diagnosis: over-diameter, over-depth; Measurement values: Diameter = 0.15mm (standard ≤ 0.1mm), Depth = 0.025mm (standard ≤ 0.02mm)". This signal was sent to the production line PLC, triggering the rejection mechanism to remove the unqualified workpiece from the production line. At the same time, the contents of the defect report popped up and were highlighted on the monitoring screen.

[0080] In summary, the machine vision-based post-riveting quality inspection method according to the embodiments of this application is explained. It combines photometric stereo vision technology with single-point laser height measurement, calculates surface normals and albedo by acquiring multi-illumination image sequences, and calibrates the three-dimensional topography reconstruction process using reference height values ​​obtained from a single-point laser displacement device to generate a high-precision calibrated height map. This method effectively overcomes the interference of high reflectivity and shadows on the rivet surface, significantly improving the detection capability for defects such as microcracks and dents. Furthermore, reliable quality judgment is achieved through multi-dimensional feature fusion extraction. In this way, the three-dimensional geometric topography and surface optical properties of the object are effectively separated, fundamentally eliminating the interference of highlights and shadows on detection. This enables accurate identification and quantification of minute defects such as cracks and dents that are difficult to detect using traditional vision methods, significantly improving the accuracy and reliability of post-riveting quality inspection.

[0081] Furthermore, a riveting equipment for post-riveting quality inspection based on machine vision is also provided.

[0082] Figure 3 This is a block diagram of a machine vision-based riveting quality inspection device according to an embodiment of this application. Figure 3 As shown, the riveting equipment 300 based on machine vision for post-riveting quality inspection according to an embodiment of this application includes: an information acquisition module 310, used to acquire the original image sequence of the rivet to be inspected and a reference height value through an industrial camera and a single-point laser displacement device in response to a detection trigger signal; an image processing module 320, used to perform image preprocessing and surface normal calculation on the original image sequence of the rivet to obtain an X-direction normal map, a Y-direction normal map, a Z-direction normal map and an RGB albedo map; a three-dimensional shape correction and reconstruction module 330, used to perform three-dimensional shape correction and reconstruction on the X-direction normal map, the Y-direction normal map and the Z-direction normal map based on the reference height value to obtain a calibrated height map; a multi-dimensional defect feature extraction module 340, used to extract multi-dimensional defect features from the calibrated height map, the RGB albedo map, the X-direction normal map and the Y-direction normal map to obtain a candidate defect list; and a quality inspection module 350, used to classify defects and make a final judgment based on the candidate defect list to obtain a final quality inspection result, wherein the final quality inspection result is used to indicate whether the quality of the inspected rivet is qualified.

[0083] As described above, the machine vision-based riveting quality inspection device 300 according to the embodiments of this application can be implemented in various wireless terminals, such as servers with machine vision-based riveting quality inspection algorithms. In one possible implementation, the machine vision-based riveting quality inspection device 300 according to the embodiments of this application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the machine vision-based riveting quality inspection device 300 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the machine vision-based riveting quality inspection device 300 can also be one of many hardware modules of the wireless terminal.

[0084] Alternatively, in another example, the machine vision-based post-riveting quality inspection riveting device 300 and the wireless terminal can also be separate devices, and the machine vision-based post-riveting quality inspection riveting device 300 can be connected to the wireless terminal via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0085] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A machine vision-based method for quality inspection after riveting, characterized in that, include: In response to the detection trigger signal, the original image sequence and reference height value of the detected rivet are acquired using an industrial camera and a single-point laser displacement device; Image preprocessing and surface normal calculation are performed on the original image sequence of the rivet to obtain the X-direction normal map, Y-direction normal map, Z-direction normal map and RGB albedo map; Based on the reference height value, the X-direction normal map, Y-direction normal map, and Z-direction normal map are subjected to 3D topographic correction and reconstruction to obtain the calibrated height map, including: The relative height map is obtained by performing relative height field integration based on Fourier transform on the X-direction normal map, Y-direction normal map, and Z-direction normal map; Based on the reference point pixel coordinates and the system scale factor, the absolute height offset is obtained by solving the absolute reference affine transformation parameters relative to the relative height map for the reference height value. This includes: defining a positive direction neighborhood in the relative height map with the reference point pixel coordinates as the center; performing dynamic peak search on the positive direction neighborhood to obtain the actual peak coordinates; performing local expected height estimation based on a Gaussian kernel on the actual peak coordinates to obtain a robust reference relative height as the relative height value; and calculating the offset between the relative height value and the reference height value based on the system scale factor to obtain the absolute height offset. Based on the absolute height offset and the system scale factor, a global pixel-level absolute height calibration is performed on the relative height map to obtain the calibrated height map. Multi-dimensional defect features were extracted from the calibrated height map, RGB albedo map, X-direction normal map, and Y-direction normal map to obtain a candidate defect list; Based on the candidate defect list, defects are classified and finally judged to obtain the final quality inspection result, which is used to indicate whether the quality of the inspected rivet is qualified.

2. The machine vision-based post-riveting quality inspection method according to claim 1, characterized in that, The original image sequence of the rivets to be detected includes the image of the upper red rivet, the upper green rivet, the upper blue rivet, the lower red rivet, the lower green rivet, the lower blue rivet, the left red rivet, the left green rivet, the left blue rivet, the right red rivet, the right green rivet, and the right blue rivet.

3. The machine vision-based post-riveting quality inspection method according to claim 1, characterized in that, Image preprocessing and surface normal calculation are performed on the original image sequence of the inspected rivets to obtain X-direction normal maps, Y-direction normal maps, Z-direction normal maps, and RGB albedo maps, including: Extract the images of the top red rivet, bottom red rivet, left red rivet, and right red rivet; Calculate the average illumination intensity of the upper red rivet image, lower red rivet image, left red rivet image, and right red rivet image to obtain the R-channel albedo map; The images of the upper red rivet, lower red rivet, left red rivet, and right red rivet were processed using a photometric stereo method to obtain the R-channel X-direction normal map, R-channel Y-direction normal map, and R-channel Z-direction normal map.

4. The machine vision-based post-riveting quality inspection method according to claim 3, characterized in that, Image preprocessing and surface normal calculation are performed on the original image sequence of the inspected rivets to obtain X-direction normal maps, Y-direction normal maps, Z-direction normal maps, and RGB albedo maps, including: Extract the images of the top green rivet, bottom green rivet, left green rivet, and right green rivet; Calculate the G-channel albedo map, G-channel X-direction normal map, G-channel Y-direction normal map, and G-channel Z-direction normal map for the top, bottom, left, and right green rivet images; Extract the images of the top blue rivet, bottom blue rivet, left blue rivet, and right blue rivet; Calculate the B-channel albedo map, B-channel X-direction normal map, B-channel Y-direction normal map, and B-channel Z-direction normal map of the upper blue rivet image, lower blue rivet image, left blue rivet image, and right blue rivet image; The R-channel albedo map, G-channel albedo map, and B-channel albedo map are synthesized to obtain the RGB albedo map; and The X-direction normal map, Y-direction normal map, and Z-direction normal map of the G-channel are used as the X-direction normal map, Y-direction normal map, and Z-direction normal map, respectively.

5. The machine vision-based post-riveting quality inspection method according to claim 1, characterized in that, Based on the system scale factor, the absolute height offset is obtained by calculating the offset between the relative height value and the reference height value using the following formula: in, For reference height value, This is a relative height value. This is the system scaling factor.

6. The machine vision-based post-riveting quality inspection method according to claim 1, characterized in that, The relative height map is calibrated at the global pixel level based on the absolute height offset and the system scale factor to obtain the calibrated height map. This includes: performing global pixel-level absolute height calibration on the relative height map based on the absolute height offset and the system scale factor using the following formula to obtain the calibrated height map: [Formula omitted for brevity] in, The system scale factor is [value], and the absolute height offset is [value]. , Relative height map at pixel position The relative height value at that location.

7. The machine vision-based post-riveting quality inspection method according to claim 1, characterized in that, Multi-dimensional defect feature extraction was performed on the calibrated height map, RGB albedo map, X-direction normal map, and Y-direction normal map to obtain a candidate defect list, including: Geometric defects are extracted from the calibrated height map to obtain the geometric defects. Spectral pseudocolor extraction is performed on the RGB albedo map to obtain spectral pseudocolor defects; Normal anomalies are extracted from the X-direction and Y-direction normal maps to obtain normal anomaly defects. The candidate defect list is obtained by feature fusion and candidate generation for geometric defects, spectral false color defects, and normal anomaly defects.

8. A riveting equipment for post-riveting quality inspection based on machine vision, characterized in that, include: The information acquisition module is used to acquire the original image sequence and reference height value of the rivet being detected through an industrial camera and a single-point laser displacement device in response to the detection trigger signal. The image processing module is used to perform image preprocessing and surface normal calculation on the original image sequence of the rivet being inspected to obtain the X-direction normal map, Y-direction normal map, Z-direction normal map and RGB albedo map; The 3D topography correction and reconstruction module is used to perform 3D topography correction and reconstruction on the X-direction normal map, Y-direction normal map, and Z-direction normal map based on the reference height value to obtain a calibrated height map, including: The relative height map is obtained by performing relative height field integration based on Fourier transform on the X-direction normal map, Y-direction normal map, and Z-direction normal map; Based on the reference point pixel coordinates and the system scale factor, the absolute height offset is obtained by solving the absolute reference affine transformation parameters relative to the relative height map for the reference height value. This includes: defining a positive direction neighborhood in the relative height map with the reference point pixel coordinates as the center; performing dynamic peak search on the positive direction neighborhood to obtain the actual peak coordinates; performing local expected height estimation based on a Gaussian kernel on the actual peak coordinates to obtain a robust reference relative height as the relative height value; and calculating the offset between the relative height value and the reference height value based on the system scale factor to obtain the absolute height offset. Based on the absolute height offset and the system scale factor, a global pixel-level absolute height calibration is performed on the relative height map to obtain the calibrated height map. The multi-dimensional defect feature extraction module is used to extract multi-dimensional defect features from the calibrated height map, RGB albedo map, X-direction normal map and Y-direction normal map to obtain a candidate defect list; The quality inspection module is used to classify and make final judgments on the candidate defect list to obtain the final quality inspection result, which indicates whether the quality of the inspected rivet is qualified.

Images