A Machine Vision-Based Method and System for Monitoring Skin Riveting Quality

Abstract

This application discloses a machine vision-based method and system for monitoring the quality of riveting on skin, relating to the field of quality inspection technology in aerospace manufacturing engineering. The method includes: simultaneously acquiring multimodal data from a coaxial structured light 3D sensor, an industrial camera, and an infrared thermal imager to comprehensively obtain the morphology, texture, and temperature information of the rivet area; projecting the 3D point cloud data onto a radial mesh using polar coordinates to generate a measurement height function and calculate a shape difference function; extracting shape energy features using a weighted p-norm integral method, while simultaneously extracting the fractal dimension, texture features, and temperature rise index of the rivet contour to construct a rivet distribution map structure, and defining adjacency weights through Euclidean distance and stress transfer paths between nodes; and outputting riveting defect analysis results based on a graph convolutional network with a shape gating mechanism by dynamically adjusting the adjacency weights. This improves the accuracy of riveting defect detection.

Description

Technical Field

[0001] This application relates to the field of quality inspection technology in aerospace manufacturing engineering, and in particular to a method and system for monitoring the quality of skin riveting based on machine vision. Background Technology

[0002] Riveting is the primary connection method between aircraft skin and frame, and its processing quality directly affects the overall strength and aerodynamic performance of the wing structure. Current riveting processes are still mainly manual, which is limited by ergonomics and worker experience, leading to problems such as fatigue and missed inspections. Literature indicates that in aerospace manufacturing, traditional riveting quality inspection relies on experienced technicians visually identifying surface quality. However, uneven lighting and curved surfaces make it difficult to detect micro-cracks and dents, resulting in low inspection efficiency and difficulty in tracing quality information.

[0003] With the development of technology, automation and machine vision technologies have been introduced into riveting inspection to improve production efficiency. In fuselage skin riveting, modern industry urgently needs to utilize machine vision to replace manual labor in dangerous or repetitive operations. This not only improves productivity and product quality but also enhances employee safety. At the same time, because the riveting area is often a large-sized curved surface, drilling and riveting errors are unavoidable. Therefore, how to quickly identify whether riveting parameters meet manufacturing standards has become a hot research topic in the industry. Summary of the Invention

[0004] This application provides a machine vision-based method, system, storage medium, computer program product, and electronic device for monitoring the quality of skin riveting, in order to at least solve the problems of low detection efficiency, insufficient defect identification accuracy, and poor adaptability to complex riveting structures in current related technologies.

[0005] In a first aspect, embodiments of this application provide a machine vision-based method for monitoring the quality of skin riveting. The method includes: simultaneously acquiring a two-dimensional grayscale image, a three-dimensional point cloud, and an infrared temperature map of at least one rivet region using a coaxial structured light 3D sensor, an industrial camera, and an infrared thermal imager; establishing a polar coordinate system with the rivet center as the origin and projecting the three-dimensional point cloud onto a radial grid to generate a measurement height function; calculating a shape difference function by combining the ideal height function corresponding to an ideal rivet model; extracting shape energy features based on a weighted p-norm integral from the shape difference function; extracting the fractal dimension and texture features of the rivet contour from the two-dimensional grayscale image; and extracting a temperature rise index from the infrared temperature map; and constructing a structure based on the shape energy features, fractal dimension, texture features, and temperature rise index of each rivet region. A rivet distribution map structure is constructed. Each node of the rivet distribution map structure uniquely corresponds to a rivet region. The node features include the morphological energy features, fractal dimension, texture features, and temperature rise index of the corresponding rivet region. The edge connections of the rivet distribution map structure are defined by the Euclidean distance between adjacent rivet regions being less than a threshold distance and being on the same stress transmission path. The adjacency weight represents the product of the distance attenuation factor between nodes and the correlation of the residual stress field. The node features of each node in the rivet distribution map structure are updated based on a graph convolutional network with a morphological gate mechanism, and the riveting defect analysis results of each rivet region are output through the updated node features. The graph convolutional network with a morphological gate mechanism is used to dynamically adjust the adjacency weights according to the deviation between the node morphological energy value and the gate threshold, thereby optimizing the information propagation process.

[0006] Secondly, embodiments of this application provide a machine vision-based skin riveting quality monitoring system. The system includes: a data acquisition unit, used to simultaneously acquire two-dimensional grayscale images, three-dimensional point clouds, and infrared temperature maps of at least one rivet region based on a coaxial structured light 3D sensor, an industrial camera, and an infrared thermal imager; a morphology difference analysis unit, used to establish a polar coordinate system with the rivet center as the origin and project the three-dimensional point cloud onto a radial grid to generate a measurement height function, and calculate a morphology difference function by combining it with the ideal height function corresponding to an ideal rivet model; a feature extraction unit, used to extract morphology energy features based on weighted p-norm integrals from the morphology difference function, extract the fractal dimension and texture features of the rivet contour from the two-dimensional grayscale image, and extract temperature rise indicators from the infrared temperature map; and a graph structure construction unit, used to construct a graph based on the morphology energy features, fractal dimension, and texture features of each rivet region. A rivet distribution map structure is constructed using shape dimension, texture features, and temperature rise index. Each node in the rivet distribution map structure uniquely corresponds to a rivet region, and the node features include the shape energy features, fractal dimension, texture features, and temperature rise index of the corresponding rivet region. The edge connections of the rivet distribution map structure are defined by the Euclidean distance between adjacent rivet regions being less than a threshold distance and being on the same stress transmission path. The adjacency weight represents the product of the distance attenuation factor between nodes and the correlation with the residual stress field. A graph convolutional network update unit is used to update the node features of each node in the rivet distribution map structure based on a graph convolutional network with shape gating mechanism, and outputs the riveting defect analysis results of each rivet region through the updated node features. The graph convolutional network with shape gating mechanism is used to dynamically adjust the adjacency weights according to the deviation between the node shape energy value and the gating threshold, thereby optimizing the information propagation process.

[0007] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the machine vision-based skin riveting quality monitoring method of any embodiment of this application.

[0008] Fourthly, embodiments of this application provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the machine vision-based skin riveting quality monitoring method of any embodiment of this application.

[0009] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the machine vision-based skin riveting quality monitoring method of any embodiment of this application.

[0010] The machine vision-based skin riveting quality monitoring method and system provided in this application can achieve at least the following technical effects:

[0011] (1) By combining a coaxial structured light 3D sensor, an industrial camera, and an infrared thermal imager to simultaneously acquire 3D point clouds, 2D grayscale images, and infrared temperature maps, multidimensional data of the riveting area can be comprehensively obtained, providing all-round support for defect identification of curved surfaces and complex riveting areas. In addition, by establishing a polar coordinate system at the rivet center and projecting the 3D point cloud onto a radial grid to generate a measurement height function, and combining it with the ideal height function corresponding to the ideal rivet model to calculate the morphological differences, the morphological differences of the riveting surface can be accurately calculated, and riveting quality problems can be automatically identified and quantified, significantly improving the ability to identify minute morphological deformations.

[0012] (2) By aggregating multi-dimensional data such as the morphological energy characteristics, texture characteristics, and temperature rise index of the rivet area into a unified graph structure, and defining the connection relationship between nodes through Euclidean distance and stress transfer path, the interaction between rivets and stress transfer during the riveting process can be reflected more intuitively. In addition, a graph convolutional network (GCN) with morphological gating mechanism is introduced for riveting defect analysis. The adjacency weights are dynamically adjusted based on the deviation between the morphological energy value and the gating threshold. By combining actual physical guidance information, the GCN intelligently optimizes the information propagation process, making the defect identification process not only more accurate, but also able to adaptively adjust the analysis focus according to the morphological characteristics of the riveting surface, which can better adapt to different riveting conditions and improve the robustness and accuracy of the model.

[0013] This technical solution, by integrating multimodal data acquisition and advanced graph convolutional network algorithms, achieves automated and high-precision riveting quality monitoring, significantly overcoming the limitations of traditional manual inspection. Combining multiple data sources—3D sensors, industrial cameras, and infrared thermal imagers—it accurately identifies minute morphological changes during the riveting process, especially in complex riveting structures, efficiently identifying and locating defects such as cracks and dents. Through optimization of morphology gating mechanisms and information propagation weights, it not only improves the accuracy of defect identification but also achieves efficient and intelligent quality control during riveting quality assessment, providing a more reliable quality monitoring solution for industries such as aerospace manufacturing. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart illustrating an example of a machine vision-based skin riveting quality monitoring method according to an embodiment of this application is shown.

[0016] Figure 2 A flowchart illustrating an example of calculating adjacency weights according to an embodiment of this application is shown.

[0017] Figure 3 This document shows an example of dynamically adjusting adjacency weights in a graph convolutional network with a topography gating mechanism according to an embodiment of this application.

[0018] Figure 4 A simulation diagram illustrating an example of the effect of noise intensity on detection accuracy according to an embodiment of this application is shown.

[0019] Figure 5 A simulation diagram illustrating an example of the comparison of recognition accuracy between a benchmark algorithm and the solution proposed in this application for four typical defect types is shown.

[0020] Figure 6 A structural block diagram of an example of a machine vision-based skin riveting quality monitoring system according to an embodiment of this application is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] It should be noted that, among the current related technologies, some experts and scholars have proposed some novel technical directions for monitoring the quality of skin riveting using machine vision technology.

[0023] The core of a machine vision inspection system lies in the processing of acquired images, with image segmentation being a crucial step. Commonly used iterative thresholding, maximum entropy thresholding, and maximum inter-class variance thresholding methods struggle to clearly separate the target from the background under uneven lighting conditions. While circle detection based on Hough transform can locate rivets, standard algorithms are computationally intensive, have strict edge requirements, and suffer from significant accumulation of invalid data and quantization errors in parameter space projection.

[0024] Specifically, circle detection algorithms based on the standard Hough transform have advantages such as robustness to noise and insensitivity to local defects, but they also suffer from drawbacks such as high computational cost, large memory consumption in parameter space projection, and high requirements for edge quality. To address these issues, researchers proposed an improved Hough transform algorithm: an arbitrary point is selected on the edge of the circle based on its gradient direction, and another intersection point is found by searching horizontally to the right. The circle center is iteratively updated using the perpendicular bisector, and votes are accumulated in the parameter space until the range exceeds the limit. This reduces the search for invalid parameters and lowers the computational cost, but it still relies on a single-modality two-dimensional grayscale image and cannot identify minute cracks or dents around rivets.

[0025] The digital twin skin assembly quality inspection method proposed in patent CN114663763B combines deep learning, RFID chips, and virtual workshop simulation. Specifically, it involves adjusting lighting and camera parameters, acquiring skin surface images, segmenting the images, classifying defects using a ResNet-50 network, writing the classification results into RFID and tracing process parameters, and finally improving the parameters through virtual simulation. It should be noted that while this method improves inspection efficiency to some extent, it also has limitations: first, it uses traditional convolutional neural networks, requiring large-scale labeled data and long training time; second, the classification results mainly target defect types, making it difficult to quantify the degree of defects; and third, the deep model lacks physical constraints and cannot explain the specific causes of crack formation.

[0026] In addition, for the inspection of riveting quality of box-shaped metal workpieces, some literature has proposed combining an improved Retinex image enhancement algorithm with multi-threshold defect segmentation. Specifically, the high reflectivity of the rivet material leads to uneven surface brightness. To eliminate the influence of illumination, a two-step improved Retinex algorithm is adopted: first, the illumination component is estimated and the reflection component is obtained through Gaussian filtering; then, a piecewise gamma function is used to enhance the contrast of the region of interest. The mathematical model of this algorithm represents the original image as the product of the illumination component and the reflection component, and the illumination is obtained through logarithmic transformation. After enhancement, OTSU threshold segmentation is applied, and a multi-threshold strategy is used to remove noise. Finally, the geometric shape and Zernike moment features of the rivets are extracted, and batch detection is achieved using Extreme Learning Machine (ELM) classification. This method achieves good results on metal box-shaped workpieces, but the segmentation and feature extraction process still relies on threshold setting and manual features, making it difficult to extend to aircraft skin rivets with complex shapes and smaller sizes.

[0027] In addition, some researchers have proposed using deep convolutional neural networks to detect defects in large rivets. For example, the YOLO model can achieve a detection accuracy of 97%. However, this model is only suitable for large rivets and still struggles to identify morphological differences and micro-cracks in millimeter-sized rivets, failing to meet the precision requirements of aerospace manufacturing engineering. Convolutional networks typically process two-dimensional images, do not consider the actual physical structure and material properties, and are easily affected by noise or changes in lighting.

[0028] In summary, current methods employ traditional image processing algorithms and deep learning models but lack specific designs tailored to the physical laws of riveting, resulting in insufficient adaptability to minor defects and complex working conditions.

[0029] It should be understood that the above description of the relevant technologies is intended only to help the public better understand the inventive spirit and motivation of this application, and is not intended to limit this application. Furthermore, the technical solutions described in the above-mentioned relevant technologies are not prior art, and may also be undisclosed technical solutions, such as those under research or in the laboratory stage.

[0030] The technical solutions in this application, including the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information, comply with relevant laws and regulations and do not violate public order and good morals.

[0031] Figure 1 A flowchart illustrating an example of a machine vision-based skin riveting quality monitoring method according to an embodiment of this application is shown.

[0032] Regarding the execution subject of the method in the embodiments of this application, it can be any controller or processor with computing or processing capabilities, such as a skin riveting quality monitoring platform. By making full use of multimodal images while introducing physical constraints and adaptive data fusion mechanisms, it can achieve high-precision, real-time, and traceable skin riveting quality monitoring, thereby improving the accuracy of micro-defect detection and supporting process optimization.

[0033] like Figure 1 As shown, in step S110, based on the coaxial structured light 3D sensor, industrial camera and infrared thermal imager, at least one rivet area is simultaneously acquired as a two-dimensional grayscale image, a three-dimensional point cloud and an infrared temperature map.

[0034] In some implementations, the coaxial structured light 3D sensor acquires precise 3D data of the rivet area by projecting light rays and receiving reflected light, enabling high-precision and short-time point cloud acquisition. Specifically, the coaxial structured light 3D sensor projects light rays of a known pattern (e.g., stripes or dot matrix) onto the riveting surface and measures the 3D topography of the surface by receiving the deformation of the returned light, thereby accurately capturing the surface shape and minute height variations of the rivet area.

[0035] In addition, industrial cameras are used to capture high-resolution two-dimensional grayscale images of rivet areas, which can provide more information about important surface features such as texture and gloss. Their high resolution helps to accurately identify the contours, cracks and surface defects of rivets.

[0036] In addition, the temperature distribution during the riveting process can reflect whether the rivet connection is uniform and whether there are quality problems caused by local overheating, such as thermal stress or riveting defects. Infrared thermal imagers are used to capture infrared temperature maps, allowing for real-time monitoring of temperature changes in the rivet area during the riveting process.

[0037] The layout design of the data acquisition unit needs to fully consider the collaborative work between devices and the optimal utilization of physical space. Specifically, the 3D camera and polarized light source are coaxially configured to ensure the acquisition of high-quality 3D point cloud data at the optimal angle. A ring-shaped polarized light source and polarization filter are used to eliminate specular reflections and improve the accuracy of data acquisition. The industrial camera and 3D camera are precisely calibrated to ensure accurate alignment of point cloud data and images, avoiding data misalignment. The infrared thermal imager is arranged in a ring or staggered layout according to the shape and size of the riveting area, capturing temperature changes during the riveting process in real time. All sensors are coordinated through a unified synchronous control system to ensure the synchronous acquisition and efficient transmission of multimodal data.

[0038] By synchronously acquiring multi-dimensional data from three sensors, the shape, texture, and temperature information of the riveting area can be fully captured at the same time. This avoids information loss or deviation caused by the limitations of a single sensor. Through the fusion of multi-source data, it provides comprehensive support for defect identification of curved surfaces and complex riveting areas.

[0039] In some examples of embodiments of this application, various types of sensor data can also be preprocessed to ensure the data quality of subsequent defect analysis.

[0040] More specifically, the improved Retinex algorithm is used to normalize the illumination of the two-dimensional grayscale image, the distortion is corrected for the three-dimensional point cloud, and the temperature field is smoothed and denoised for the infrared temperature map.

[0041] Regarding the explanation of illumination normalization processing, in some examples of the embodiments of this application, the rivet area often has uneven illumination conditions, especially during the riveting process, the direction and intensity of the light source may change, resulting in uneven brightness on the rivet surface in the image. In order to reduce the influence of illumination and highlight surface details, an improved Retinex algorithm is used for illumination normalization processing.

[0042] Specifically, the acquired 2D grayscale image is preprocessed, using Gaussian filtering to estimate the illumination component L(x,y) to initially remove local illumination variations in the image. A logarithmic transformation is then applied to separate the image into a reflection component R(x,y) and an illumination component L(x,y), thus separating the reflection component that reflects the essential morphology of the rivet surface. Furthermore, an adaptive contrast enhancement method is introduced based on the traditional Retinex algorithm, making the brightness contrast of the rivet area more prominent, especially under conditions of significant illumination variations, effectively avoiding interference from excessively dark or bright areas in subsequent feature extraction. Finally, the reflection component R(x,y) is normalized to unify illumination differences, thereby enhancing the visibility of rivet surface defects (such as cracks and dents). Illumination normalization eliminates image brightness differences caused by ambient light variations, significantly improving the contrast of rivet surface details, enhancing defect visibility, and reducing the impact of uneven illumination on defect identification.

[0043] Regarding the details of distortion correction for 3D point clouds, it should be noted that during the 3D point cloud acquisition process, factors such as the sensor's viewing angle, focal length, and lens distortion can cause distortion in the acquired point cloud data, affecting the accurate reconstruction of the rivet's surface morphology. To improve the accuracy of the point cloud, distortion correction must be performed on the acquired 3D point cloud.

[0044] Specifically, standard image calibration methods (such as the Zhang Zhengyou calibration method) can be used to calibrate the 3D sensor and obtain a lens distortion model. The distortion model mainly includes radial distortion and tangential distortion, which are obtained by calibrating the point cloud and camera intrinsic parameters, respectively. Then, based on the known calibration parameters and distortion model, distortion correction is performed on the acquired 3D point cloud to correct geometric errors and thus restore the true rivet surface morphology. Therefore, by correcting the distortion of the 3D point cloud, geometric errors caused by sensor distortion are eliminated, significantly improving the accuracy of the point cloud data.

[0045] Regarding the details of temperature field smoothing and denoising processing for infrared temperature maps, it should be noted that temperature maps acquired by infrared thermal imagers are often affected by noise, especially in areas with drastic temperature changes, where noise can interfere with accurate temperature field calculations. Therefore, a smoothing and denoising algorithm is used to process the infrared temperature maps to improve the stability and reliability of the temperature field data.

[0046] Specifically, a Gaussian smoothing filter can be used to denoise the temperature map, effectively reducing random noise in the image while preserving the trend of temperature changes and avoiding over-smoothing that could affect the validity of the temperature data. After denoising, the temperature distribution model of the rivet area is reconstructed, thus accurately reflecting the temperature changes generated during the riveting process.

[0047] In step S120, a polar coordinate system is established with the rivet center as the origin, and the three-dimensional point cloud is projected onto the radial mesh to generate a measurement height function. The shape difference function is calculated by combining the ideal height function corresponding to the ideal rivet model.

[0048] Here, a polar coordinate system is established based on 3D point cloud data, and it is projected onto a radial grid to generate a measurement height function. Then, by comparing it with the height function of an ideal rivet model, the morphological differences of the riveting area can be calculated.

[0049] Specifically, a polar coordinate system is defined on the riveting surface with the center of the rivet as the origin. Point cloud data is converted into radial and angular coordinates, and this coordinate transformation supports a unified evaluation of the morphology of different regions. Using the polar coordinate system, surface data can be projected onto a radial grid, where each grid point represents the height information of a small region. Each point in these grids has a height value, generating a measured height function that can represent the height variations at various points on the surface in detail. By comparing the measured height function with the ideal height function of the ideal rivet, a morphology difference function is obtained, which can clearly reveal the differences between the riveting area and the ideal model, especially for detecting minute deformations, dents, or protrusions.

[0050] Therefore, it can accurately capture minute morphological deviations in the riveting area, especially for complex riveting shapes and surfaces, and can efficiently reveal defects caused by manufacturing errors or process problems, providing quantitative morphological analysis.

[0051] In step S130, morphological energy features based on weighted p-norm integrals are extracted from the morphological difference function, fractal dimension and texture features of the rivet contour are extracted from the two-dimensional grayscale image, and temperature rise index is extracted from the infrared temperature map.

[0052] In some implementations, based on the morphology difference function calculated in the previous step, as well as the two-dimensional grayscale image and infrared temperature map, morphology energy features, fractal dimension and texture features, and temperature rise index are extracted respectively, thereby comprehensively describing the surface quality, texture complexity and thermal effects of the riveting area.

[0053] Specifically, extracting the morphological energy features from the weighted p-norm integral of the morphological difference function can quantify the degree of difference in surface morphology, thereby reflecting the quality of the riveting area. By integrating the morphological difference function, the irregularity and defect energy of the rivet surface can be effectively quantified. The use of the weighted p-norm allows for flexible adjustment of the emphasis on morphological deviations according to the weights of different regions, thus improving the accuracy of feature extraction.

[0054] Fractal dimension and texture features are extracted from two-dimensional grayscale images. Fractal dimension measures the complexity and self-similarity of the rivet profile and can effectively capture subtle texture variations on the riveted surface. Texture feature extraction reveals potential defects such as localized damage, wear, or cracks on the surface by analyzing the pixel distribution in the grayscale image.

[0055] Infrared temperature maps provide temperature rise indicators that reflect the heat distribution during the riveting process. By extracting temperature rise information from infrared temperature maps, especially temperature fluctuations in local areas, it is possible to help identify thermal inhomogeneity or potential thermal damage during the riveting process.

[0056] In step S140, a rivet distribution map structure is constructed based on the morphological energy characteristics, fractal dimension, texture characteristics, and temperature rise index of each rivet region.

[0057] Here, each node in the rivet distribution graph structure uniquely corresponds to a rivet region. Node features include the morphological energy characteristics, fractal dimension, texture features, and temperature rise index of the corresponding rivet region. Edge connections in the rivet distribution graph structure are defined by the Euclidean distance between adjacent rivet regions being less than a threshold distance and located on the same stress transmission path. The adjacency weight represents the product of the distance attenuation factor between nodes and the correlation with the residual stress field. For example, if the Euclidean distance between two rivet regions is less than a set threshold and they are located on the same stress transmission path, they are connected as an edge in the graph structure, and the adjacency weight of the edge connection expresses the correlation between the distance attenuation factor between nodes and the residual stress field.

[0058] Therefore, by constructing a rivet distribution map structure, a graph structure integrating the characteristics of all rivet regions is provided, and the spatial information of the rivet regions is closely combined with the mechanical transmission path based on adjacency relationships. The rivet distribution map structure can not only represent the spatial relationship between rivets, but also analyze the quality of the riveting area from a global perspective, especially the mutual influence between adjacent rivets during the riveting process, and can reveal potential stress concentration areas and areas of non-uniform quality.

[0059] In step S150, the node features of each node in the rivet distribution map structure are updated based on the graph convolutional network with topology gating mechanism, and the riveting defect analysis results of each rivet region are output through the updated node features.

[0060] By updating the node features in the rivet distribution map using GCN, the feature information of each node comes not only from itself, but also from the propagation information of its neighboring nodes. By fusing neighborhood information through graph convolution operations, the spatial relationship and mechanical coupling effect between rivets can be captured, making defect analysis more accurate.

[0061] Here, a graph convolutional network with a topography gating mechanism is used to dynamically adjust adjacency weights based on the deviation between the node topography energy value and the gating threshold, thereby optimizing the information propagation process. The core of the gating mechanism is to control the propagation intensity of information between adjacent nodes through topography energy differences, avoiding interference from redundant information, thus focusing on regions with significant topography differences. This maintains high-precision defect analysis capabilities under different industries or production conditions. Therefore, by intelligently optimizing the information propagation process and dynamically adjusting adjacency weights through a graph convolutional network with a topography gating mechanism, information transmission becomes more accurate, significantly improving the accuracy and robustness of riveting defect analysis and enabling it to adapt to the characteristics of different riveting regions.

[0062] Regarding the implementation details of step S120, in some examples of embodiments of this application, the three-dimensional point cloud is mapped to a polar coordinate grid using bilinear interpolation to generate a measurement height function. Specifically, a polar coordinate system is established with the center of the rivet as the origin. Then, the acquired three-dimensional point cloud data is converted to a radial grid in the polar coordinate system to generate the corresponding measurement height function.

[0063] For example, 3D point cloud data of the rivet area is acquired using a coaxial structured light 3D sensor. This data contains the spatial coordinates of each point on the rivet surface, specifically (x, y, z). Then, the rivet center is set as the origin of the polar coordinate system, and the radial distance *r* in polar coordinates is defined as the distance from the rivet center to a point on the rivet surface, and the angle *θ* is defined as the azimuth angle from the rivet center to the measurement point. Furthermore, a bilinear interpolation method is used to map the 3D point cloud data onto a radial grid in the polar coordinate system. The purpose of interpolation is to fill in the unmeasured areas on the polar coordinate grid, ensuring the smoothness of the height function across the entire region by calculating intermediate values ​​between adjacent points.

[0064] By projecting the three-dimensional surface of the rivet area onto a polar coordinate system, with r and θ as independent variables, a measurement height function z(r,θ) in the polar coordinate system is obtained, which describes the height distribution of the actual rivet surface.

[0065] The ideal height function is determined by the theoretical model of rivet plastic deformation, reflecting the ideal rivet height value at a corresponding radial distance. Specifically, the plastic deformation model of the rivet is used, considering the deformation of the rivet during the riveting process. Through this model, the ideal height distribution of the rivet at different radial distances r can be determined, for example, assuming that the rivet surface has a certain circular or elliptical profile.

[0066] The ideal height function can be expressed by the following formula:

[0067] z ideal (r)=k1r α +k2, Equation (1)

[0068] In the formula, zideal (r) represents the height of the ideal rivet model at a radial distance r, indicating the geometric height distribution of the ideal rivet; k1, k2, α are material constants calibrated through finite element simulation.

[0069] For example, the plastic deformation of the rivet is simulated using the finite element analysis (FEA) method, and parameters such as k1, k2, and α are calibrated using the simulation results. k1 and k2 are material constants calculated by finite element simulation, and α is a parameter determined based on the geometry of the rivet and the plastic deformation model. This constructs an ideal height function for the rivet and provides a standard shape of the rivet surface for comparison in actual riveting processes.

[0070] Calculate the topography difference function based on the difference between the actual height function and the ideal height function:

[0071] h(r,θ)=z(r,θ)-z ideal (r), Equation (2)

[0072] In the formula, z(r,θ) is the measurement height function based on polar coordinates, representing the actual height value of the rivet surface in the polar coordinate system; r is the radial distance in polar coordinates, representing the radial distance from the center point of the rivet to the measurement point on the rivet surface; θ is the angle in polar coordinates, representing the angle of the measurement point on the rivet surface relative to the center point of the rivet; h(r,θ) represents the morphological difference function.

[0073] Here, h(r,θ) represents the height difference between the actual rivet surface and the ideal surface, reflecting the morphological deviation of the rivet surface. The morphological difference function can reveal the deformation of the rivet surface during riveting, especially in areas of riveting defects (such as cracks, protrusions, or depressions). Areas with large differences can be further used for defect localization and analysis.

[0074] Regarding the details of extracting morphological energy features in step S130, in some examples of embodiments of this application, the morphological energy features of the rivet region are calculated in the defined region of the morphological difference function.

[0075] Specifically, the morphological energy features are extracted by a calculation method based on weighted p-norm integrals. Weighted p-norm integrals help to calculate the morphological differences in the rivet region, especially when different parts of the rivet (such as the edge and the center) have different importance. The morphological features of different regions can be highlighted by weighted adjustment.

[0076] For example, h(r,θ) based on the rivet surface, which represents the height difference between the actual rivet surface and the ideal rivet surface, can better locate and analyze morphological deviations during the riveting process. The morphological energy feature E is obtained by performing a weighted p-norm integral on the morphological difference function h(r,θ) over the defined region Ω of the rivet area.

[0077] E=(∫ Ω w(r)|h(r,θ)| p rdrdθ) 1 / p Equation (3)

[0078] In the formula, E represents the morphological energy feature, the weighting function w(r) is the weighting function of the rivet edge region, representing the weighting value of the rivet edge region; Ω is the definition region of the morphological difference function h(r,θ), used to represent the entire rivet surface; p is the weighting norm exponent, representing the weighting factor when calculating morphological energy, used to adjust the sensitivity to the morphology of different parts, p≥1.

[0079] Here, by analyzing the entire area Ω of the rivet surface, covering all areas from the rivet center to the measurement point on the rivet surface, it is ensured that the morphological changes of all rivet areas can be calculated and analyzed.

[0080] The weighted p-norm integral method can effectively extract the morphological energy features of the rivet surface. In particular, when dealing with the edge and center regions of the rivet, the weights can be adjusted according to the importance of different regions to improve the ability to identify edge defects (such as cracks and dents).

[0081] More specifically, the weighting function w(r) takes a value of 1 in the rivet edge region r∈(0.8R,R], meaning that the morphological changes in the edge region contribute significantly to the overall morphological energy. Therefore, the morphological deviation in the edge region will have a significant impact on the overall analysis. In the rivet center region, r∈[0.2R,0.8R] linearly decreases to 0, where R is the rivet radius. The weighting function value gradually decreases with increasing distance, linearly decaying from 1 to 0. The morphological changes in the center region are given a smaller weight to avoid overemphasizing the morphological deviation in this region.

[0082] It should be noted that during riveting, edge areas are often more prone to defects such as cracks, burrs, or irregular deformations than the central area. Morphological changes in edge areas require higher weighting for more accurate defect detection. The central area exhibits less morphological change and typically does not involve severe defects; therefore, it receives lower weighting to avoid interference during analysis. Thus, by using weighted p-norm integration, the calculation of morphological energy features can be optimized, allowing for more sensitive detection of defects in the rivet edge area during defect detection, while the influence of the rivet center area is moderately reduced, thereby improving the accuracy of defect identification.

[0083] Regarding the details of extracting the fractal dimension in step S130, in some examples of embodiments of this application, the box-counting method can be used to calculate the fractal dimension corresponding to the two-dimensional grayscale image. The box-counting method is a method for calculating image complexity by using the minimum number of boxes covering the image, and is particularly suitable for surfaces and contours with complex shapes. By statistically analyzing the minimum number of boxes covering the rivet contour, the geometric complexity of the rivet contour can be reflected, thus serving as a basis for morphological analysis.

[0084] More specifically, the rivet outline is covered with a square grid of side length ε, and the minimum number of boxes N(ε) covering the rivet outline is counted.

[0085] Here, a square grid with a side length of ε is used to cover the rivet outline. Each grid cell (box) is used to cover a small area of ​​the outline. As the grid precision increases (i.e., the side length ε decreases), the required number of grid cells N(ε) increases. For each ε value, the minimum number of boxes N(ε) is calculated, which is the minimum number of boxes required to cover the entire rivet outline.

[0086] For example, the side length ε of the boxes is gradually changed. For each box size, the number of boxes N(ε) required to cover the rivet outline is calculated and counted. Starting with larger boxes, the side lengths are gradually decreased until the appropriate accuracy is achieved. Each time the box size is adjusted, the number of boxes needs to be recalculated to ensure that no area is missed during coverage.

[0087] In the box dimension method, the relationship between the number of boxes N(ε) and the side length ε of the boxes is usually logarithmic, conforming to the following formula:

[0088] logN(ε)~-D f logε, Equation (4)

[0089] In the formula, D f Fractal dimension is a key indicator for measuring the complexity of a rivet's profile.

[0090] Furthermore, within the range of ε, a linear relationship between logN(ε) and log(1 / ε) is fitted, and the slope of the regression line is taken as the fractal dimension, which is the desired fractal dimension D. f It can quantify the complexity of the rivet profile.

[0091] The box dimension method can be used to accurately calculate the fractal dimension D of the rivet profile. f This reflects the complexity of the rivet's surface morphology. In particular, defects such as cracks and burrs usually increase the fractal dimension, and changes in the fractal dimension can serve as an important basis for defect characterization, which helps to achieve defect detection on complex surface morphologies.

[0092] The details of extracting texture features and temperature rise indicators in step S130 can be partially referenced from similar operations in current related technologies. For example, texture features are extracted by analyzing the gray-level changes of the rivet outline, such as Gray Level Co-occurrence Matrix (GLCM) analysis, to capture local texture information in the image; the temperature rise indicator is obtained by acquiring the temperature changes during the riveting process through an infrared sensor, and then calculating the temperature rise value within the corresponding time period to assess whether overheating or uneven heating occurs during the riveting process.

[0093] Figure 2 A flowchart illustrating an example of calculating adjacency weights according to an embodiment of this application is shown.

[0094] like Figure 2 As shown, in step S210, the stress distribution map of the riveting area is obtained, and the correlation coefficient of the residual stress field between different nodes is extracted.

[0095] Here, a stress distribution map of the riveting area is pre-established using finite element simulation software, and the residual stress values ​​σ at the corresponding positions of node i and node j are extracted from the stress distribution map of the riveting area. i and σ j The Pearson correlation coefficient between node i and node j is calculated based on the continuity of stress distribution, and used as the corresponding correlation coefficient of residual stress field.

[0096] Specifically, a mechanical model of the riveting area is established using finite element simulation software. By applying stress and constraints, the distribution of residual stress during the riveting process is simulated. Based on the geometric model and material parameters of the rivet, a mechanical model of the riveting area is established using finite element simulation software (such as ANSYS), considering stress changes and the generation of residual stress during the riveting process.

[0097] Furthermore, during the simulation, considering the contact force of the rivets, deformation during the riveting process, and stress transfer, a residual stress distribution map of the riveting area is generated. Each node represents a small region, and the residual stress value of that region is output. The residual stress values ​​σ at the corresponding positions of nodes i and j are extracted from the simulation results. i and σ j This reflects the physical stress state of the riveting area.

[0098] σ ij The Pearson correlation coefficient is used to quantify the residual stress relationship between node i and node j. Its calculation can be based on the deviation of the stress value of each node from the mean stress in the region, reflecting the degree of correlation between the stresses between nodes.

[0099] Specifically, for nodes i and j, the mean residual stress in their respective regions is calculated. and Right now:

[0100]

[0101] In the formula, N is the number of sampling points in the region. and Let i and j represent the stress values ​​of node i and node j at the kth sampling point, respectively.

[0102] Based on the calculated residual stress and mean stress, the Pearson correlation coefficient σ between node i and node j is calculated. ij :

[0103]

[0104] Equation (6) measures the correlation of residual stress values ​​between two nodes, ranging from [-1, 1]. A value close to 1 indicates a strong correlation, close to 0 indicates no correlation, and close to -1 indicates a negative correlation. The Pearson correlation coefficient calculation can effectively quantify the stress transfer relationship between nodes within the rivet region, thereby identifying closely related node regions under mechanical action.

[0105] In step S220, the adjacency weight is calculated by fusing the correlation coefficient between the Euclidean distance between nodes and the residual stress field.

[0106] Specifically, the Euclidean distance between nodes and the correlation of the residual stress field jointly determine the adjacency weight between nodes within the riveting area. This weight reflects the physical relationship between nodes and serves as the initial weight base value for information propagation in the graph convolutional network.

[0107] First, calculate the Euclidean distance d between node i and node j. ij , representing the geometric distance between two nodes in space:

[0108]

[0109] In the formula, x i ,y i ,z i and x j ,y j ,z j These are the spatial coordinates of node i and node j, respectively.

[0110] Combined with the Euclidean distance d between nodes ij The correlation coefficient σ with the residual stress field ij Calculate the adjacency weight A ij :

[0111] A ij =exp(-β·d ij )·σij Equation (8)

[0112] In the formula, β is the distance attenuation coefficient, which is calibrated through stress transmission attenuation experiments and has a value of 0.5 ± 0.1 mm. -1 This coefficient reflects the degree to which distance affects adjacency relationships; A ij This represents the adjacency weight corresponding to the edge connection between nodes i and j.

[0113] In Equation (8), the adjacency weights, calculated based on the correlation coefficient between Euclidean distance and residual stress field, determine the information propagation intensity between nodes in the graph convolutional network. By combining Euclidean distance and Pearson correlation coefficient, the adjacency weights can accurately capture the mechanical correlation between nodes in the riveting region.

[0114] By combining the stress distribution map of the riveting area generated by finite element simulation, the Pearson correlation coefficient and Euclidean distance between nodes are accurately calculated, thereby dynamically calculating the adjacency weight. This quantifies the mechanical correlation between nodes and optimizes the information propagation process based on the stress transmission relationship between nodes. Adjacency weight calculation driven by physics and mechanics can more accurately identify potential defects in the riveting area, improving the accuracy of riveting quality assessment and defect location in complex riveting structures.

[0115] Figure 3 A flowchart illustrating an example of dynamically adjusting adjacency weights in a graph convolutional network with a topography gating mechanism according to an embodiment of this application is shown.

[0116] like Figure 3 As shown, in step S310, the gating factor is calculated.

[0117] The gating factor is a value generated by comparing the shape and energy features of a node with a preset ideal rivet shape and energy threshold using a sigmoid function. The gating factor is used to adjust the weights of information propagation in a graph convolutional network, thereby achieving targeted focusing on potential riveting defect areas.

[0118] Specifically, the shape energy feature value E of node i is obtained. i Compare it with the preset ideal rivet morphology energy threshold T E Compare the node's morphological energy eigenvalues ​​E. i This is a key feature extracted based on the morphological changes in the rivet region, used to measure the morphological difference of the node region relative to the ideal rivet model. The morphological energy feature E of each node i is... i Energy threshold T for the preset ideal rivet morphology E The comparison is made between the morphology energy threshold T of the ideal rivet. E These are standard values ​​set through simulation or experience, representing the morphological standard of the rivet surface under normal conditions.

[0119] Then, the comparison result is input into the S-type function converter to generate a gating factor between 0 and 1.

[0120]

[0121] In the formula, g i γ represents the gating weight of node i; γ is the threshold kurtosis parameter, which takes values ​​in the range [5,10] and is used to control the kurtosis of the S-shaped function converter.

[0122] Based on the calculated gating factor g i The information propagation weights of node i are adjusted. Nodes with large shape energy deviations (i.e., E...) i With T E Nodes with larger differences in shape and energy will receive higher weights, while nodes with normal shape and energy will receive lower weights.

[0123] In step S320, during the node feature update process, the calculated gating factor is multiplied by the adjacency weight to adjust the information propagation weight, so that nodes with large shape energy deviations receive higher weights during information aggregation, and nodes with normal shape energy have their weights reduced during information aggregation, thereby achieving directional focusing on abnormal rivet regions.

[0124] Here, in graph convolutional networks with topology gating mechanisms, node feature updates are no longer solely influenced by neighboring node information but also dynamically adjusted by the gating factor. Specifically, in graph convolutional networks, node feature updates depend on the information of their neighboring nodes. During each information propagation, the gating factor g of each node is calculated. i The adjacency weights are dynamically adjusted during the node feature update process.

[0125] Furthermore, the gating factor g i Compared with the original adjacency weight A ij Multiplying them together yields the new, adjusted dynamic adjacency weights.

[0126]

[0127] When information is propagated through graph convolution, nodes with large shape and energy deviations will have a greater impact on information aggregation, while nodes with normal shapes will have a reduced role in information propagation.

[0128] In this embodiment, a gating factor is generated by calculating the difference between the morphological energy features of a node and the ideal rivet model. This ensures that rivet regions with large morphological deviations receive higher weights during information aggregation, while nodes with normal morphology are less affected. Thus, dynamically adjusting the adjacency weights allows the graph convolutional network to focus on riveting defect regions, improving the accuracy and precision of defect identification. This significantly enhances the reliability and effectiveness of quality assessment in complex riveting structures.

[0129] It should be noted that the graph convolutional network with shape gating mechanism in this application can also be called a Physically-Guided Multimodal Fusion Network (PGMM-FN). It consists of multiple layers, including an input layer, a feature extraction layer, a graph convolutional layer, a gating mechanism layer, and an output layer. The input layer first receives multimodal data from 3D point clouds, 2D grayscale images, and infrared temperature maps, converting them into a unified feature vector. The feature extraction layer performs preliminary feature extraction on these data, such as texture, morphology, and temperature rise features, and performs data normalization for subsequent processing.

[0130] In the graph convolutional layer, node features are updated through graph convolution operations. Specifically, the update of node features depends not only on the information of neighboring nodes but also on the regulation of a shape gating mechanism, dynamically adjusting the weights of information propagation. For each node, new node features are calculated using graph convolution operations based on the features of its neighboring nodes and a pre-calculated gating factor. By modeling the interactions, physical relationships, and mechanical transmission between nodes, the graph convolutional network can capture subtle changes in riveting defects at both spatial and mechanical levels, especially in complex riveting regions.

[0131] Furthermore, during training, PGMM-FN can employ supervised learning methods, where the loss function is optimized based on detection accuracy (e.g., cross-entropy loss). The network continuously updates node features and adjacency weights during training until convergence and optimal performance is achieved. The training dataset includes various riveting defect samples, including defect types such as "missing rivets," "incomplete riveting," "head cracking," and "riveting misalignment." Each sample contains multimodal data (grayscale image, depth map, infrared image), with corresponding labels indicating the type and location of the riveting defect. During each forward propagation, the network utilizes graph convolution operations and gating mechanisms to compute node features, analyzes these features, and ultimately outputs the riveting defect analysis results for each rivet region. For example, in each rivet region, PGMM-FN outputs a prediction of whether a defect exists in that region and the type of defect through its output layer. This includes the defect's location, type (e.g., "crack" or "misalignment"), and confidence value for each defect, providing a detailed defect analysis report. This facilitates timely location of riveting defects by quality inspectors, improving the accuracy and reliability of the riveting process.

[0132] To verify the effectiveness of the proposed technical solution, details of some comparative experiments will be disclosed below to evaluate the detection performance of the system under different noise conditions and different defect types, and to compare it with the benchmark method.

[0133] (I) Experimental Datasets and Benchmark Methods

[0134] Given the lack of multimodal samples for skin riveting in existing public datasets, this study constructed a custom experimental dataset. This dataset contains 500 rivet samples, covering five riveting defect types: "qualified, missing rivet, incomplete riveting, upset head crack, and riveting misalignment." Each sample simultaneously records four different image data types: a 2D grayscale image, a 3D depth map, an infrared image, and the corresponding riveting parameters. This data provides comprehensive support for the training and validation of subsequent detection algorithms.

[0135] To simulate the impact of different noise conditions on the performance of the riveting quality inspection system, this experiment introduced Gaussian noise of varying intensities into the image data, with the noise intensity η ranging from 0 to 0.5. The introduction of noise was used to test the system's robustness under different environmental disturbances.

[0136] For comparison, we selected the improved Retinex+ELM classification method as a typical benchmark method, which combines image processing and classification algorithms to address the impact of image noise on riveting quality detection.

[0137] In the experiment, the proposed Physically Guided Multimodal Fusion Network (PGMM-FN) method was compared with the two benchmark methods to verify its advantages in noise interference and defect type identification.

[0138] (II) Indicators and Experimental Setup

[0139] To comprehensively evaluate the performance of the riveting defect detection system, this experiment uses the detection accuracy (Acc) as the main evaluation index, defined as follows:

[0140]

[0141] In the formula, T p and T n These represent the number of correctly identified defective rivets and normal rivets, respectively, with M being the total number of samples. This metric is used to measure the algorithm's detection performance on the overall dataset.

[0142] Furthermore, to evaluate the system's robustness under different noise conditions, we introduce a curve showing the relationship between noise intensity η and accuracy. By analyzing the accuracy changes under different noise intensities, we further evaluate the algorithm's adaptability to noise.

[0143] For different defect types (such as "missing rivets" and "incomplete riveting"), this experiment also recorded the recognition accuracy of each defect in order to analyze the detection effect of each type of defect in detail and compare the advantages and disadvantages of each algorithm in different defect detection.

[0144] (III) Experimental Results and Discussion

[0145] Figure 4 A simulation diagram illustrating an example of the effect of noise intensity on detection accuracy according to an embodiment of this application is shown. The horizontal axis represents noise intensity η, ranging from 0 to 0.5, and the vertical axis represents detection accuracy. Figure 4 The detection accuracy of the baseline algorithm was compared with that of the PGMM-FN method proposed in this paper.

[0146] like Figure 4 As shown, the accuracy of the benchmark algorithm decreases significantly with increasing noise intensity, especially under medium-to-high noise conditions, where the accuracy rapidly drops below 0.8. In contrast, the proposed PGMM-FN method exhibits stronger robustness with increasing noise intensity, maintaining an accuracy consistently above 0.8, and even stably identifying riveting defects under high noise conditions. This demonstrates that the PGMM-FN method effectively improves the system's resistance to noise interference, maintaining high detection accuracy in more complex noisy environments.

[0147] Figure 5A simulation diagram illustrating an example of the comparison of recognition accuracy between a benchmark algorithm and the solution proposed in this application for four typical defect types is shown.

[0148] like Figure 5 As shown, the baseline method (the improved Retinex+ELM method) has an accuracy of less than 0.8 when dealing with defects such as "upsetting head cracking" and "riveting misalignment," and cannot effectively identify these defects. This is because the method is not capable of identifying defects with large morphological changes and those easily affected by light and reflection, especially in the case of complex riveting structures, where it is difficult to accurately extract minute morphological differences, resulting in low detection accuracy.

[0149] In comparison, the PGMM-FN method proposed in this paper demonstrates higher accuracy in detecting all defect types, with an accuracy exceeding 0.88 for all defect types. Particularly noteworthy is its accuracy exceeding 0.9 in identifying "missing rivets" and "incomplete riveting" defects. Therefore, the physically guided fusion network effectively integrates data features from 3D point clouds, 2D grayscale images, and infrared temperature maps. By utilizing multimodal analysis of the morphological differences of various defects, it enhances the ability to distinguish complex defects (such as cracks and misalignments).

[0150] This paper addresses the limitations of existing skin riveting quality inspection methods by proposing a multimodal machine vision-based method and system for skin riveting quality monitoring. Compared with traditional improved Hough transform or simple deep learning methods, it has the following advantages:

[0151] 1) Combining multimodal acquisition with physical modeling: Grayscale images, depth images, and infrared images are acquired simultaneously using coaxial structured light and polarized light sources, and residual stress fields are established using finite element simulation to provide physical references for subsequent feature extraction and model construction.

[0152] 2) Physical characteristics such as morphology energy and fractal dimension: Features such as morphology energy integral, curvature and fractal dimension for rivet deformation are proposed, which help to identify micro-cracks and complex defects and make the test results physically interpretable.

[0153] 3) Physics-guided multimodal fusion network: A graph convolution model with shape gating and force weights is designed to achieve adaptive fusion of three-dimensional shape, two-dimensional texture and temperature information, which greatly improves the robustness to noise and multiple defect types.

[0154] Experimental results demonstrate that the PGMM-FN method presented in this paper exhibits superior performance in noisy environments and with complex defect types, achieving a significantly higher detection accuracy than the improved Retinex+ELM method. Future development can further enhance the system's adaptability and scalability in practical production by incorporating more refined thermo-mechanical coupling models, reinforcement learning control strategies, and large-scale real-world datasets.

[0155] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0156] Figure 6 A structural block diagram of an example of a machine vision-based skin riveting quality monitoring system according to an embodiment of this application is shown.

[0157] like Figure 6 As shown, the machine vision-based skin riveting quality monitoring system 600 includes a data acquisition unit 610, a morphology difference analysis unit 620, a feature extraction unit 630, a graph structure construction unit 640, and a graph convolutional network update unit 650.

[0158] The data acquisition unit 610 is used to simultaneously acquire two-dimensional grayscale images, three-dimensional point clouds, and infrared temperature maps of at least one rivet area based on a coaxial structured light three-dimensional sensor, an industrial camera, and an infrared thermal imager.

[0159] The morphology difference analysis unit 620 is used to establish a polar coordinate system with the rivet center as the origin and project the three-dimensional point cloud onto the radial grid to generate a measurement height function, and calculate the morphology difference function in combination with the ideal height function corresponding to the ideal rivet model.

[0160] The feature extraction unit 630 is used to extract morphological energy features based on weighted p-norm integrals from the morphological difference function, extract fractal dimension and texture features of rivet contours from the two-dimensional grayscale image, and extract temperature rise indexes from the infrared temperature map.

[0161] The graph structure building unit 640 is used to construct a rivet distribution map structure based on the morphological energy characteristics, fractal dimension, texture characteristics, and temperature rise index of each rivet region. Each node of the rivet distribution map structure corresponds uniquely to a rivet region, and the node features include the morphological energy characteristics, fractal dimension, texture characteristics, and temperature rise index of the corresponding rivet region. The edge connection of the rivet distribution map structure is defined by the Euclidean distance between adjacent rivet regions being less than a threshold distance and being on the same stress transmission path. The adjacency weight represents the product of the distance attenuation factor between nodes and the correlation of the residual stress field.

[0162] The graph convolutional network update unit 650 is used to update the node features of each node in the rivet distribution map structure based on the graph convolutional network with a topology gating mechanism, and output the riveting defect analysis results of each rivet region through the updated node features; the graph convolutional network with a topology gating mechanism is used to dynamically adjust the adjacency weights according to the deviation between the node topology energy value and the gating threshold, thereby optimizing the information propagation process.

[0163] In some embodiments, this application provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, server, or network device) to perform the steps of any of the machine vision-based skin riveting quality monitoring methods described above.

[0164] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of any of the above-described machine vision-based skin riveting quality monitoring methods.

[0165] In some embodiments, this application also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of a machine vision-based skin riveting quality monitoring method.

[0166] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0167] The electronic devices in this application can exist in various forms, including but not limited to: mobile communication devices, ultra-mobile personal computer devices, portable entertainment devices, or other airborne electronic devices with data interaction functions.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A machine vision-based method for monitoring the quality of skin riveting, characterized in that, The method includes: Based on a coaxial structured light 3D sensor, an industrial camera and an infrared thermal imager, two-dimensional grayscale images, three-dimensional point clouds and infrared temperature maps of at least one rivet area are acquired simultaneously. A polar coordinate system is established with the rivet center as the origin, and the three-dimensional point cloud is projected onto a radial grid to generate a measurement height function. The morphological difference function is calculated by combining the ideal height function corresponding to the ideal rivet model. The shape energy features based on the weighted p-norm integral are extracted from the shape difference function, the fractal dimension and texture features of the rivet contour are extracted from the two-dimensional grayscale image, and the temperature rise index is extracted from the infrared temperature map. A rivet distribution map structure is constructed based on the morphological energy characteristics, fractal dimension, texture characteristics, and temperature rise index of each rivet region. Each node of the rivet distribution map structure uniquely corresponds to a rivet region, and the node features include the morphological energy characteristics, fractal dimension, texture characteristics, and temperature rise index of the corresponding rivet region. The edge connection of the rivet distribution map structure is defined by the Euclidean distance between adjacent rivet regions being less than a threshold distance and being on the same stress transmission path. The adjacency weight represents the product of the distance attenuation factor between nodes and the correlation of the residual stress field. The graph convolutional network with topology gating mechanism updates the node features of each node in the rivet distribution graph structure, and outputs the riveting defect analysis results of each rivet region through the updated node features; the graph convolutional network with topology gating mechanism is used to dynamically adjust the adjacency weights according to the deviation between the node topology energy value and the gating threshold, thereby optimizing the information propagation process; The step of establishing a polar coordinate system with the rivet center as the origin and projecting the three-dimensional point cloud onto a radial mesh to generate a measurement height function, and then calculating the morphology difference function by combining the ideal height function corresponding to the ideal rivet model, includes: A measurement height function is generated by mapping the 3D point cloud to a polar coordinate grid using bilinear interpolation. The ideal height function is determined by the theoretical model of rivet plastic deformation: ， In the formula, For the ideal rivet model at radial distance The height value at that point represents the geometric height distribution of an ideal rivet; These are the material constants calibrated through finite element simulation; Calculate the morphological difference function: ， In the formula, This is a height measurement function based on polar coordinates, representing the actual height value of the rivet surface in the polar coordinate system; It is the radial distance in polar coordinates, representing the radial distance from the center point of the rivet to the measurement point on the rivet surface; It is an angle in polar coordinates, representing the angle of the measurement point on the rivet surface relative to the center point of the rivet; Represents the morphological difference function; The step of extracting morphological energy features based on weighted p-norm integrals from the morphological difference function includes: Within the defined region of the morphology difference function, calculate the morphological energy characteristics of the rivet region: ， In the formula, Representing morphological energy characteristics, weighting function is a weighting function for the rivet edge region, representing the weighted value of the rivet edge region; Morphological difference function The defined area is used to represent the entire rivet surface; The weighted norm exponent represents the weighting factor used in calculating morphological energy, adjusting the sensitivity to different parts of the morphology. ; Among them, the weight function In the rivet edge area The value is 1, in the center area of ​​the rivet. Decrease linearly to 0, The radius is the rivet radius.

2. The method according to claim 1, characterized in that, After the coaxial-structured optical 3D sensor, industrial camera, and infrared thermal imager simultaneously acquire the 2D grayscale image, 3D point cloud, and infrared temperature map of the target rivet area, the method further includes: The two-dimensional grayscale image is processed by illumination normalization using an improved Retinex algorithm, the three-dimensional point cloud is processed by distortion correction, and the infrared temperature map is processed by temperature field smoothing and denoising.

3. The method according to claim 1, characterized in that, The extraction of the fractal dimension includes: The fractal dimension of the two-dimensional grayscale image is calculated using the box-counting method: - By side length Given a square grid covering the rivet outline, find the minimum number of boxes that cover the rivet outline. ; - Fit within the range of values of the linear relationship between and , and take the slope of the regression line as the fractal dimension.

4. The method according to claim 1, characterized in that, The calculation of the adjacency weight includes: The stress distribution map of the riveting area was obtained using finite element simulation software, and nodes were extracted from the stress distribution map of the riveting area. and nodes Residual stress value at the corresponding location and Calculation nodes based on stress distribution continuity With nodes The Pearson correlation coefficient between them is used as the corresponding residual stress field correlation coefficient; Combining Euclidean distance between nodes Correlation coefficient with residual stress field Calculate the adjacency weight: ， In the formula, This is the distance attenuation coefficient, calibrated through stress transmission attenuation experiments; Represents a node and The adjacency weights corresponding to the edges connecting them.

5. The method according to claim 1, characterized in that, The graph convolutional network with topography gating mechanism is used to dynamically adjust adjacency weights through the following operations: Calculate the gating factor: - Get Nodes morphological energy characteristic values Compare it with the preset ideal rivet morphology energy threshold. Compare; - Input the comparison result into the sigmoid function transformer to generate a gating factor between 0 and 1: ， In the formula, Represents a node Gating weights; The threshold kurtosis parameter has a value range of [5, 10] and is used to control the conversion kurtosis of the S-function converter; During the node feature update process, the calculated gating factor is multiplied by the adjacency weight to adjust the information propagation weight. This results in nodes with large shape and energy deviations receiving higher weights during information aggregation, while nodes with normal shape and energy have their weights reduced during information aggregation. This enables directional focusing on abnormal rivet regions.

6. A machine vision-based skin riveting quality monitoring system, characterized in that, The system is used to implement the method as described in any one of claims 1-5; the system comprises: The data acquisition unit is used to simultaneously acquire two-dimensional grayscale images, three-dimensional point clouds, and infrared temperature maps of at least one rivet area based on a coaxial structured light three-dimensional sensor, an industrial camera, and an infrared thermal imager. The morphology difference analysis unit is used to establish a polar coordinate system with the rivet center as the origin and project the three-dimensional point cloud onto the radial grid to generate a measurement height function, and calculate the morphology difference function by combining the ideal height function corresponding to the ideal rivet model. The feature extraction unit is used to extract morphological energy features based on weighted p-norm integrals from the morphological difference function, extract fractal dimension and texture features of rivet contours from the two-dimensional grayscale image, and extract temperature rise indexes from the infrared temperature map. The graph structure construction unit is used to construct a rivet distribution graph structure based on the morphological energy characteristics, fractal dimension, texture characteristics, and temperature rise index of each rivet region. Each node of the rivet distribution graph structure uniquely corresponds to a rivet region, and the node features include the morphological energy characteristics, fractal dimension, texture characteristics, and temperature rise index of the corresponding rivet region. The edge connection of the rivet distribution graph structure is defined by the Euclidean distance between adjacent rivet regions being less than a threshold distance and being on the same stress transmission path. The adjacency weight represents the product of the distance attenuation factor between nodes and the correlation of the residual stress field. The graph convolutional network update unit is used to update the node features of each node in the rivet distribution map structure based on the graph convolutional network with a topology gating mechanism, and output the riveting defect analysis results of each rivet region through the updated node features; the graph convolutional network with a topology gating mechanism is used to dynamically adjust the adjacency weights according to the deviation between the node topology energy value and the gating threshold, thereby optimizing the information propagation process.

Images