A method, system and processor for detecting the head height of a bolt

Abstract

The present invention relates to the technical field of visual detection, and discloses a method, system and processor for detecting the head height of a bolt. The method comprises: inputting a head image of a high-locking bolt into an encoder-decoder for feature extraction to obtain bolt head contour feature data and bolt head depth feature data; performing structured light projection on the head of the high-locking bolt to obtain three-dimensional reconstruction data of the bolt head; constructing a comprehensive loss function; performing relaxation variable conversion on the three-dimensional reconstruction data of the bolt head to obtain a convex optimization model for bolt head measurement; calculating edge area depth values ​​in combination with the bolt head depth feature data to obtain bolt head detection data; performing feature fusion calculation through a Gaussian mixture model to obtain a bolt head height detection result. The present invention improves the accuracy of bolt head height detection.

Description

Technical Field

[0001] The present invention relates to the field of visual detection technology, and in particular to a method, system and processor for detecting the head height of a bolt. Background Art

[0002] As a key connecting component in mechanical equipment such as aerospace, railways and ships, the accurate detection of the head height of high-lock bolts is of great significance for the safe operation of equipment. Traditional mechanical contact detection methods, such as strain gauges and pressure gauges, have problems such as low detection efficiency, possible damage to the bolt surface, and limited detection accuracy in actual applications, making it difficult to meet the requirements of industrial automation and quality control.

[0003] With the development of computer vision technology, non-contact detection methods based on image processing have gradually become a research hotspot. However, due to the reflective properties of the head surface of high-lock bolts and the nonlinear coupling effect of the inclination angle of the bolt during installation and the lighting environment, existing visual detection methods often have problems such as large detection errors and poor robustness when dealing with such complex scenes. Summary of the invention

[0004] The present invention provides a method, a system and a processor for detecting the head height of a bolt, and the present invention improves the accuracy of detecting the head height of the bolt.

[0005] In a first aspect, the present invention provides a method for detecting the head height of a bolt, the method comprising:

[0006] The head image of the high-lock bolt is input into the encoder-decoder for feature extraction to obtain the bolt head contour feature data and the bolt head depth feature data;

[0007] A three-dimensional reconstruction mathematical model is established according to the bolt head contour feature data, and structured light projection is performed on the head of the high-locking bolt to obtain three-dimensional reconstruction data of the bolt head;

[0008] Performing cross entropy loss calculation on the bolt head profile feature data, and performing mean square error loss calculation on the bolt head depth feature data, to obtain a comprehensive loss function;

[0009] Based on the comprehensive loss function, the three-dimensional reconstruction data of the bolt head is subjected to slack variable conversion, and a sequential convex programming method is used to construct an objective function to obtain a convex optimization model for bolt head measurement;

[0010] Performing edge detection on the bolt head contour feature data, and calculating the edge area depth value in combination with the bolt head depth feature data to obtain bolt head detection data;

[0011] According to the bolt head detection data and the bolt head measurement convex optimization model, feature fusion calculation is performed through a Gaussian mixture model to obtain a bolt head height detection result.

[0012] In a second aspect, the present invention provides a bolt head height detection system, the bolt head height detection system comprising:

[0013] A feature extraction module is used to input the head image of the high-lock bolt into the encoder-decoder for feature extraction to obtain the bolt head contour feature data and the bolt head depth feature data;

[0014] A projection module is used to establish a three-dimensional reconstruction mathematical model according to the bolt head contour feature data, and perform structured light projection on the head of the high-locking bolt to obtain three-dimensional reconstruction data of the bolt head;

[0015] A calculation module, used to perform cross entropy loss calculation on the bolt head profile feature data, and perform mean square error loss calculation on the bolt head depth feature data, to obtain a comprehensive loss function;

[0016] A construction module is used to perform slack variable conversion on the three-dimensional reconstruction data of the bolt head based on the comprehensive loss function, and to construct an objective function using a sequential convex programming method to obtain a convex optimization model for bolt head measurement;

[0017] A detection module, used to perform edge detection on the bolt head contour feature data, and calculate the edge area depth value in combination with the bolt head depth feature data to obtain bolt head detection data;

[0018] The fusion module is used to perform feature fusion calculation through a Gaussian mixture model based on the bolt head detection data and the bolt head measurement convex optimization model to obtain the bolt head height detection result.

[0019] A third aspect of the present invention provides a processor, which is used to execute the above-mentioned method for detecting the head height of a bolt.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned method for detecting the head height of a bolt.

[0021] In the technical solution provided by the present invention, by constructing a dual-channel encoder-decoder network structure, the parallel extraction of the bolt head contour feature and the depth feature is realized, wherein the first channel uses the BatchNormalization layer and the ReLU activation function to extract the contour feature, and the second channel introduces the jump connection structure and the LeakyReLU activation function to extract the depth feature, which effectively solves the problem of insufficient feature extraction. The structured light projection technology is combined with the phase unwrapping algorithm for three-dimensional reconstruction. By establishing the reflectivity parameter model and the attitude parameter model, the reflective characteristics and spatial attitude characteristics of the bolt surface are accurately described, and the problem of the difficulty of modeling complex surfaces in traditional methods is overcome. The relaxation variable is introduced to transform the reflective intensity term and the tilt angle term, and the sequential convex programming method is used to construct the objective function. The augmented Lagrangian model and the alternating direction multiplier method are used for optimization and solution, and the non-convex optimization problem is successfully converted into a convex optimization problem. Combined with sub-pixel edge detection and morphological processing technology, the accuracy of the contour feature is improved through opening and closing operations and connected domain analysis, and the depth data is optimized by surface fitting and outlier removal methods, which improves the accuracy of the depth value calculation in the edge area. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0023] Figure 1 Schematic diagram of the steps of a method for detecting the height of a bolt head in an embodiment of the present invention;

[0024] Figure 2 Schematic diagram of the structure of the bolt head height detection system in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Embodiments of the present invention provide a method, system and processor for detecting the head height of a bolt. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of a method for detecting the head height of a bolt in an embodiment of the present invention includes:

[0027] Step S1, inputting the head image of the high-lock bolt into the encoder-decoder for feature extraction to obtain the bolt head contour feature data and the bolt head depth feature data;

[0028] It is understandable that the execution subject of the present invention may be a bolt head height detection system, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0029] Specifically, the head image of the high-lock bolt is processed into two feature extraction channels, namely the contour feature extraction channel and the depth feature extraction channel. Each channel corresponds to an independent encoder-decoder network to extract the contour and depth information of the bolt head respectively. For the contour feature extraction channel, a first encoder-decoder network based on a convolutional neural network is constructed. The encoder of the network consists of five convolutional layers, and each layer is connected to the BatchNormalization layer and the ReLU activation function after convolution, so as to effectively avoid the gradient vanishing problem in the training process and improve the training efficiency of the model. The convolution kernel size of the convolutional layer is set to 3×3, the step size is 2, and the number of channels increases layer by layer, which are 64, 128, 256, 512 and 512 respectively. Through the convolution operation, the network extracts local features in the bolt head image at multiple scales, gradually reduces the size of the image and improves the abstract level of the feature map. After five layers of convolution operations, the image is deconvolved through the decoder part of the network. The deconvolution layer is also set to five layers, and each layer is connected to the BatchNormalization layer and the ReLU activation function after deconvolution. The purpose of the deconvolution operation is to restore the spatial resolution of the image and reconstruct the contour features of the image. The convolution kernel size of each deconvolution layer is 3×3, the step size is 2, and the number of channels is 512, 256, 128, 64 and 1 respectively. The final output of the initial contour feature data is the contour information of the bolt head. At the same time, a second encoder-decoder network is constructed for the depth feature extraction channel. The encoder part of the second network uses a convolution layer with a skip connection structure. This structure can effectively retain the low-level details of the input image and avoid the loss of depth information in the layer-by-layer convolution process. The encoder also consists of five convolution layers, each of which is followed by a BatchNormalization layer and a LeakyReLU activation function. The convolution kernel size is 3×3, the step size is 2, and the number of channels is 64, 128, 256, 512 and 512 respectively. Through the convolution operation, the second encoder can extract the depth information in the bolt head image and gradually reduce the size of the image. The deconvolution part consists of five deconvolution layers, each of which is followed by a BatchNormalization layer and a LeakyReLU activation function. In this part, deconvolution is not only used to restore spatial resolution, but also incorporates the jump connection features corresponding to the encoder in each deconvolution layer, so that the deep feature information can be reconstructed more accurately during the restoration process, thereby avoiding information loss and obtaining the initial deep feature data of the bolt head. The initial contour feature data and initial depth feature data are normalized and feature aligned respectively. The purpose of normalization is to map the feature data to a uniform scale to avoid the problem of weight imbalance between different features due to inconsistent dimensions.Feature alignment ensures that the features extracted from different channels can be accurately aligned in space, making subsequent feature fusion and detection more efficient and accurate, and ultimately obtaining the bolt head contour feature data and bolt head depth feature data.

[0030] Step S2, establishing a three-dimensional reconstruction mathematical model according to the bolt head profile feature data, and performing structured light projection on the head of the high-locking bolt to obtain three-dimensional reconstruction data of the bolt head;

[0031] Specifically, a mathematical model for three-dimensional reconstruction is established based on the contour feature data of the bolt head. The contour feature data of the bolt head is converted into initial position data in the spatial coordinate system through spatial coordinate mapping. The structured light data of the bolt head is obtained by using a structured light projector to generate a stripe spot pattern and projecting it onto the surface of the bolt head. Phase calculation and unwrapping processing are performed on the structured light data of the bolt head, and depth information is extracted from the projected stripe spot pattern. The depth discontinuity problem caused by periodic blur is eliminated by the phase unwrapping algorithm to obtain the depth parameter of the bolt head. The reflection coefficient of the bolt surface is calculated through the depth parameter of the bolt head. The reflection coefficient is an important parameter that describes the interaction characteristics between light and the surface, and is related to the surface material, illumination angle and surface roughness. According to the initial position data and depth parameters, the surface reflection matrix of the bolt head is obtained. Based on the surface reflection matrix of the bolt head, the angle between the surface normal vector and the light is calculated. The angle relationship reflects the directionality of the light when it is reflected on the surface, and then determines the distribution of the reflected light intensity. By calculating the angle between the surface normal vector and the light, the spatial posture data of the bolt head, that is, the orientation and positioning information of the bolt head in three-dimensional space, is obtained. The mapping relationship between light intensity and position is established according to the depth parameter of the bolt head and the spatial posture data of the bolt head. By constructing the reconstruction function of the bolt head, the light intensity data on the bolt surface is corresponded to the actual position in the spatial coordinate system. In the reconstruction function, the light intensity value and the corresponding spatial position are combined through a specific functional relationship to form a complete surface model to describe the geometric shape of the bolt head. The reconstruction function of the bolt head is globally optimized. The reconstruction model is adjusted according to the actual data to improve the accuracy and robustness of the reconstruction result. In this process, the optimization algorithm iteratively optimizes the parameters of the reconstruction function to reduce the error between the model output and the actual measurement data. Through global optimization, it is ensured that the three-dimensional reconstruction model can fit the shape of the real bolt head to the maximum extent. The optimized three-dimensional reconstruction parameters are input into the reconstruction function, and the three-dimensional reconstruction data of the bolt head is obtained by spatial back-projection calculation. Using the mapping relationship between light intensity and position, the two-dimensional image data is reversely mapped back to the three-dimensional space, the three-dimensional structure of the bolt head is restored, and the three-dimensional reconstruction data of the bolt head is obtained.

[0032] Step S3, performing cross entropy loss calculation on the bolt head profile feature data, and performing mean square error loss calculation on the bolt head depth feature data, to obtain a comprehensive loss function;

[0033] Specifically, the bolt head profile feature data and the standard profile feature data are normalized to eliminate the dimension difference and ensure that the data are compared on a uniform scale. After normalization, the probability distributions of the profile feature data and the standard profile feature data are converted to the same range to obtain the profile feature probability data. The negative logarithm of the profile feature data is calculated to obtain the profile cross entropy loss data. By comparing the difference between the predicted profile probability distribution and the standard profile data, the accuracy and error of the model in profile feature extraction are evaluated. At the same time, the bolt head depth feature data and the standard depth feature data are difference calculated to evaluate the numerical error of the depth data and obtain the depth feature error data, which describes the deviation between the model predicted depth and the actual standard depth. The depth feature error data is averaged to obtain the depth mean square error loss data. The mean square error loss function is used for regression tasks. It measures the accuracy of the prediction by calculating the square of the difference between the predicted value and the true value and averaging them. The profile cross entropy loss data and the depth mean square error loss data. The first loss weight coefficient is set according to the profile cross entropy loss data, which determines the relative importance of the profile loss in the entire loss function. The weight coefficient is determined according to the requirements for contour feature accuracy in the actual task. If the contour feature is more important in the final detection task, the first loss weight coefficient will be set larger, so that the contour loss occupies a larger weight in the optimization process. By multiplying the contour cross entropy loss data with the weight coefficient, the contour loss weighted data is calculated to reflect the contribution of the contour feature loss in the entire model training. Similarly, for the depth mean square error loss data, a second loss weight coefficient is set according to its relative importance in the final task. By weighting the depth loss data, the depth loss weighted data is obtained to quantify the role of the depth feature in the loss function. The contour loss weighted data and the depth loss weighted data are linearly combined to obtain a comprehensive loss function.

[0034] Step S4, performing slack variable conversion on the three-dimensional reconstruction data of the bolt head based on the comprehensive loss function, and constructing the objective function using a sequential convex programming method to obtain a convex optimization model for bolt head measurement;

[0035] Specifically, the reflective intensity term in the three-dimensional reconstruction data of the bolt head is replaced by a relaxation variable. The reflective intensity is a key factor affecting illumination and surface reflection in three-dimensional reconstruction. It needs to be relaxed when constructing the optimization model so that the illumination effect in the optimization process can be effectively controlled. The first constraint is introduced to describe the physical limitation and adjustable range of the reflective intensity, and a reflective intensity relaxation model is obtained. The relaxed model can process the reflective intensity data in a more flexible way, so that it can adapt to the solution requirements in the optimization process. The tilt angle term in the three-dimensional reconstruction data of the bolt head is replaced by a relaxation variable. The tilt angle is a key parameter describing the angle between the bolt surface and the projection light, which directly affects the reflection direction of the light and the three-dimensional reconstruction accuracy. Relaxing the tilt angle term can effectively reduce the uncertainty in the angle calculation, and by introducing the second constraint to limit the range of angle change, a tilt angle relaxation model is obtained. The tilt angle relaxation model helps to better handle angle data in the convex optimization solution process and avoid the problem of overfitting or error expansion. The reflective intensity relaxation model and the tilt angle relaxation model are substituted into the comprehensive loss function, and the initial convex optimization objective function is obtained by variable replacement. The objective function can combine the effects of reflection intensity and tilt angle, and consider their contribution to the overall 3D reconstruction accuracy. By introducing these two relaxation models into the objective function, it is ensured that the model can effectively process illumination and angle data during the optimization process, and accurate 3D reconstruction results are obtained through optimization. The initial convex optimization objective function is Taylor expanded, and the nonlinear terms in the objective function are approximated as linear terms, so that the originally complex nonlinear optimization problem is transformed into a linear problem that is easier to solve. Each nonlinear term is linearized to simplify the computational complexity and improve the convergence of the optimization process. Convex constraints are added to the linearized objective function to ensure that the optimization problem will not fall into a local optimal solution during iterative solution. In order to construct a solution framework for the optimization problem, the augmented Lagrangian model is adopted. By introducing Lagrangian multipliers and enhancing the objective function, the optimization problem can be efficiently solved under the premise of satisfying the constraints. Based on the augmented Lagrangian model, a sequential convex programming iterative format is constructed. In this format, by setting appropriate iteration steps and termination conditions, the optimization process is effectively guided to converge to the optimal solution. The sequential convex programming method gradually approaches the global optimal solution by continuously optimizing the objective function in each iteration. During the iteration process, each updated objective function will gradually reduce the error and eventually obtain a converged solution. During the optimization process, the optimization variables are updated by the alternating direction method of multipliers (ADMM). The ADMM method decomposes the original problem into multiple sub-problems and uses the local optimal solution of each sub-problem to gradually approach the overall optimal solution. In each iteration, the ADMM method ensures that the optimization process of the overall model can converge quickly and stably by coordinating the variables between different sub-problems. Substitute the converged solution into the augmented Lagrangian model, and restore the physical variables through reverse calculation.Through the restoration process, the actual measurement data of the bolt head, such as physical parameters such as reflection intensity and tilt angle, are recovered from the optimization solution to describe the spatial position, shape and illumination characteristics of the bolt head, and the measurement convex optimization model of the bolt head is obtained.

[0036] Step S5, performing edge detection on the bolt head profile feature data, and calculating the edge area depth value in combination with the bolt head depth feature data to obtain the bolt head detection data;

[0037] Specifically, the sub-pixel edge detection is performed on the contour feature data of the bolt head. The contour is refined by the grayscale centroid method to accurately identify the edge of the bolt head. The grayscale centroid method infers the edge position of the image to the sub-pixel level by calculating the grayscale value of each pixel and its coordinate relative to the center of the image, thereby obtaining more accurate contour data. The contour data is optimized by morphological processing. Morphological operations include opening and closing operations, which are used to remove small noise and fill holes in the contour respectively. The opening operation can remove small noise points in the image by corroding first and then dilating; while the closing operation can fill small holes in the contour caused by noise or other factors by dilating first and then corroding. Through morphological processing, more coherent and smooth contour optimization data is obtained. Based on the contour optimization data, connected domain analysis is performed to identify and analyze different areas in the bolt head contour. The independent areas in the contour are identified and separated to facilitate subsequent processing. In this process, the edge tracking algorithm is used to accurately track the edges in the image to ensure the closure of the contour, and the contour area is defined by the closed edge. After obtaining the closed contour data, the region is segmented. Different regions are divided according to the different gray values ​​and edge information in the image, and the corresponding depth segmentation data is obtained. The depth segmentation data is input into the depth extraction module for processing. In the depth extraction module, the depth data of each region is fitted by the surface fitting method to obtain the edge depth data. Surface fitting is a mathematical method that fits irregular data points into a smooth surface, which is used to accurately estimate the depth information of each region. Outlier removal is performed on the edge depth data. Data points that deviate from the normal range are identified and removed by statistical methods to ensure that the obtained depth data is more reliable. Outliers are caused by noise or abnormal measurements. By removing data points that do not conform to the convention, the accuracy and stability of the depth data are improved, and the depth optimization data is obtained. A depth distribution function is constructed based on the depth optimization data to describe the distribution of depth data in the entire bolt head area. By constructing the depth distribution function, the depth data is statistically analyzed to understand the distribution characteristics of different depth regions. The depth mapping data is based on the depth distribution function, and the depth data is mapped to the spatial coordinate system by mathematical methods to realize the spatialization of the depth data. The depth map data is transformed from the original image coordinate system to the actual three-dimensional space coordinate system to obtain the final bolt head detection data. Coordinate transformation includes operations such as translation, rotation, and scaling, which converts the two-dimensional data in the image into three-dimensional data corresponding to the physical space to obtain more intuitive detection results.

[0038] Step S6: Based on the bolt head detection data and the bolt head measurement convex optimization model, a feature fusion calculation is performed through a Gaussian mixture model to obtain a bolt head height detection result.

[0039] Specifically, principal component analysis is performed on the bolt head detection data and the bolt head measurement convex optimization model. The data is mapped from the high-dimensional space to the low-dimensional space through linear transformation, while retaining the original information of the data as much as possible to obtain the reduced-dimensional feature data. Principal component analysis can extract the most representative features in the bolt head detection data and the measurement model, making the subsequent processing more efficient and reducing the computational complexity. K-means clustering is performed on the reduced-dimensional feature data. K-means clustering is an unsupervised learning algorithm that divides data points into K clusters so that the data points within the cluster are as similar as possible, while the data points between clusters are quite different. This method finds the optimal clustering result by iteratively updating the cluster center. In the bolt head detection data, K-means clustering groups the feature data according to similarity to obtain the center sequence of the feature. The feature center sequence represents different groups or trends in the data. Based on the feature center sequence, a Gaussian distribution model group is constructed. The Gaussian distribution model can describe the distribution characteristics of the data and perform probabilistic modeling on the data. Through the Gaussian mixture model, the data of the detection data and the measurement model are respectively fitted into multiple Gaussian distributions, each of which corresponds to a subgroup or pattern in the data. The process of constructing the Gaussian distribution model group uses the EM algorithm (expectation maximization algorithm) to optimize the model parameters. The EM algorithm is an iterative optimization algorithm that gradually approaches the optimal parameters of the model by alternating the expectation step (E step) and the maximization step (M step). In the E step, the EM algorithm calculates the expected value of the latent variable based on the current model parameters, and in the M step, the model parameters are updated by maximizing the expected value until the model converges. The distribution parameter matrix of the Gaussian distribution model is optimized. Ensure that the parameters of the model are more suitable for the characteristics of the bolt head detection data and the measurement convex optimization model. During the optimization process, the elements of the parameter matrix are gradually adjusted through an iterative method to adapt to the changes and needs of the actual data. Through this process, the mixed weight coefficient is obtained, which represents the contribution of different Gaussian distributions in the overall model. The mixed weight coefficient is weighted and fused with the bolt head detection data to obtain the feature weight data. In the weighted fusion process, the mixed weight coefficient is used as a weighting factor to determine the importance of each feature in the overall calculation. The feature weight data is similar to the bolt head measurement convex optimization model. By calculating the similarity between the test data and the optimized model, the degree of their matching in high-dimensional space is evaluated. Similarity calculation methods include Euclidean distance, cosine similarity, etc., which can measure the similarity or difference between feature vectors. In the process of bolt head height detection, based on the similarity calculation results, the most matching features are screened out to obtain more accurate detection results. Based on the feature matching data, weighted averaging is performed to obtain a fused feature vector. By combining the advantages of different features, a comprehensive feature representation is obtained. The fused feature vector contains all the important information extracted from the test data, measurement model and Gaussian mixture model.The fused feature vector is spatially transformed from the original feature space to the actual three-dimensional space coordinate system to obtain the final bolt head height detection result.

[0040] In the embodiment of the present invention, by constructing a dual-channel encoder-decoder network structure, the parallel extraction of the bolt head contour feature and the depth feature is realized, wherein the first channel uses the BatchNormalization layer and the ReLU activation function to extract the contour feature, and the second channel introduces the jump connection structure and the LeakyReLU activation function to extract the depth feature, which effectively solves the problem of insufficient feature extraction. The structured light projection technology is combined with the phase unwrapping algorithm for three-dimensional reconstruction. By establishing the reflectivity parameter model and the attitude parameter model, the reflective characteristics and spatial attitude characteristics of the bolt surface are accurately described, and the problem of the difficulty of modeling complex surfaces in traditional methods is overcome. The relaxation variable is introduced to transform the reflective intensity term and the tilt angle term, and the sequential convex programming method is used to construct the objective function. The augmented Lagrangian model and the alternating direction multiplier method are used for optimization and solution, and the non-convex optimization problem is successfully converted into a convex optimization problem. Combined with sub-pixel edge detection and morphological processing technology, the accuracy of the contour feature is improved through opening and closing operations and connected domain analysis, and the depth data is optimized by using surface fitting and outlier removal methods, which improves the accuracy of the depth value calculation in the edge area.

[0041] In a specific embodiment, the process of executing step S1 may specifically include the following steps:

[0042] The head image of the high-lock bolt is divided into a contour feature extraction channel and a depth feature extraction channel;

[0043] A first encoder-decoder network is constructed according to the contour feature extraction channel. The encoder of the first encoder-decoder network includes 5 convolutional layers, each of which is connected to a BatchNormalization layer and a ReLU activation function. The decoder of the first encoder-decoder network includes 5 deconvolutional layers, each of which is connected to a BatchNormalization layer and a ReLU activation function.

[0044] The head image of the high-lock bolt is input into the encoder of the first encoder-decoder network, and the contour feature space data is obtained through 5-layer convolution operation. The convolution kernel size of the 5-layer convolution operation is 3×3, the step size is 2, and the number of channels is 64, 128, 256, 512, and 512 respectively;

[0045] Perform 5 layers of deconvolution operations on the contour feature space data. The convolution kernel size of the 5 layers of deconvolution operations is 3×3, the step size is 2, and the number of channels is 512, 256, 128, 64, and 1, respectively, to obtain the initial contour feature data.

[0046] A second encoder-decoder network is constructed according to the deep feature extraction channel. The encoder of the second encoder-decoder network includes 5 convolutional layers and a skip connection structure. Each convolutional layer is connected to a BatchNormalization layer and a LeakyReLU activation function. The decoder of the second encoder-decoder network includes 5 deconvolutional layers. Each deconvolutional layer is connected to a BatchNormalization layer and a LeakyReLU activation function.

[0047] The head image of the high-lock bolt is input into the encoder of the second encoder-decoder network, and the deep feature space data is obtained through 5 layers of convolution operations with skip connections. The convolution kernel size of the 5 layers of convolution operations is 3×3, the step size is 2, and the number of channels is 64, 128, 256, 512, and 512 respectively;

[0048] Perform 5 layers of deconvolution operations on the deep feature space data, and fuse the skip connection features of the corresponding coding layer in each layer of deconvolution operations. The convolution kernel size of the 5 layers of deconvolution operations is 3×3, the step size is 2, and the number of channels is 512, 256, 128, 64, and 1, respectively, to obtain the initial deep feature data;

[0049] The initial contour feature data and the initial depth feature data are normalized and feature aligned respectively to obtain the bolt head contour feature data and the bolt head depth feature data.

[0050] Specifically, for the head image of the high-lock bolt, it is divided into a contour feature extraction channel and a depth feature extraction channel. These two channels correspond to the shape contour information and depth (i.e., position in three-dimensional space) information of the image, respectively. In the image processing process, contour information helps to distinguish the shape, while depth information helps to determine the position of the object in three-dimensional space. The combination of the two helps to fully understand the geometric features and position of the bolt head. In order to extract contour features, the first encoder-decoder network is constructed. The encoder part of the network consists of five convolutional layers, each of which is followed by a BatchNormalization layer and a ReLU activation function. The purpose of the convolutional layer is to extract local features from the input image, and the BatchNormalization layer is used to standardize the output of each layer to make the network training more stable. The ReLU activation function is used to introduce nonlinearity and enhance the expression ability of the network. The formula for the convolution operation is expressed as:

[0051] ;

[0052] in, is the output of the convolutional layer, is the convolution kernel, is the input feature map, is the bias term, * represents the convolution operation, Represents the ReLU activation function. For each layer of convolution, the size of the convolution kernel is fixed to 3×3, the stride is 2, and the number of channels is 64, 128, 256, 512, and 512 respectively. A stride of 2 means that each convolution operation will reduce the size of the feature map by half, which helps to reduce the amount of calculation and extract more abstract features. The feature map after convolution enters the decoder part for deconvolution operation, which restores the low-dimensional feature map to a higher dimension in order to restore the contour features of the image. The deconvolution operation of the decoder uses 5 deconvolution layers, each of which is connected to a BatchNormalization layer and a ReLU activation function. The convolution kernel size of the deconvolution operation is also 3×3, the stride is 2, and the number of channels is 512, 256, 128, 64, and 1 respectively. The deconvolution operation gradually restores the spatial information by enlarging the input feature map, so that the low-dimensional features obtained by convolution can gradually approach the spatial resolution of the original image. During the deconvolution process, the weights of the convolution kernel are trained to accurately restore the contour feature data of the bolt head. For the extraction of deep features, a second encoder-decoder network is constructed. The encoder part of this network is similar to the first network, consisting of 5 convolutional layers, and each convolutional layer adopts a structure with jump connections. The jump connection directly transfers high-dimensional information from the encoding stage to the decoding stage to avoid information loss. Each convolutional layer is followed by a BatchNormalization layer and a LeakyReLU activation function. The LeakyReLU function allows a small slope on the negative semi-axis to avoid the "dead neuron" problem in ReLU. The formula for the convolution operation is also:

[0053] ;

[0054] The activation function of LeakyReLU is defined as:

[0055] ;

[0056] in, is a small constant, such as 0.01, which is used to avoid the dead neuron phenomenon when negative input is input. In the convolution process, the size of the convolution kernel is 3×3, the step size is 2, and the number of channels is 64, 128, 256, 512, and 512 respectively. This structure gradually extracts the deep feature information of the image and retains more low-level feature information through jump connections. The deep features are restored through deconvolution operations. The decoder part also contains 5 deconvolution layers, each of which is connected to the BatchNormalization layer and the LeakyReLU activation function. The difference is that in the deconvolution process, each deconvolution layer will be fused with the output of the corresponding encoder layer, and the jump connection method is used to enhance the recovery of deep features. After 5 layers of deconvolution, the initial deep feature data is obtained. The initial contour feature data and deep feature data are normalized and feature aligned. Normalization is to adjust the scale of the feature data so that the numerical range of different features is consistent, avoiding the unbalanced impact of features of different scales on model training. Normalization methods include Min-Max normalization or Z-score normalization. The normalized data enters the feature alignment step to align the feature data of different channels into the same coordinate system so that different feature information can be effectively combined in subsequent processing. After normalization and feature alignment, the profile feature data and depth feature data of the bolt head are obtained.

[0057] In a specific embodiment, the process of executing step S2 may specifically include the following steps:

[0058] The bolt head profile feature data is mapped to spatial coordinates to obtain the initial position data of the bolt head, and the stripe spot pattern generated by the structured light projector is projected onto the head surface of the high-locking bolt to obtain the bolt head structured light data;

[0059] The phase calculation and unwrapping processing are performed on the structured light data of the bolt head to obtain the depth parameter of the bolt head, and the reflection coefficient is calculated according to the initial position data of the bolt head to obtain the surface reflection matrix of the bolt head;

[0060] The angle between the surface normal vector and the light is calculated based on the bolt head surface reflection matrix to obtain the bolt head spatial posture data. The mapping relationship between light intensity and position is established based on the bolt head depth parameter and the bolt head spatial posture data to obtain the bolt head reconstruction function.

[0061] The bolt head reconstruction function is globally optimized to obtain three-dimensional reconstruction parameters, and the three-dimensional reconstruction parameters are input into the bolt head reconstruction function for spatial back-projection calculation to obtain the three-dimensional reconstruction data of the bolt head.

[0062] Specifically, the bolt head contour feature data is mapped to spatial coordinates to obtain the initial position of the bolt head. The contour feature data contains the shape information of the bolt head, but this information is two-dimensional. It is converted into three-dimensional coordinate data through spatial coordinate mapping. Through the camera internal and external parameters and calibration matrix, each pixel in the image is mapped to three-dimensional space. Assume that the pixel coordinates captured by the camera are , its corresponding three-dimensional space coordinates Calculated by the following formula:

[0063] ;

[0064] in, is the camera's internal parameter matrix, including parameters such as focal length and optical center, represents its inverse matrix, are the pixel coordinates on the image, and It is a three-dimensional coordinate obtained by spatial coordinate mapping. Through this process, the contour feature data of the bolt head obtains the spatial coordinate information to form the initial position data of the bolt head. The stripe spot pattern generated by the structured light projector is projected onto the head surface of the high-locking bolt. The shape of the stripes will be affected by the shape of the bolt surface, thereby reflecting the three-dimensional structure of the object. The structured light data of the bolt head is obtained by the spot pattern reflected from the head surface of the high-locking bolt. The projected stripe pattern will be distorted according to the change of surface curvature, and the displacement of the stripes reflects the depth information of the surface at different positions. Assume that the pixel position of the projected stripe pattern in the image is , the corresponding surface depth value It is obtained by calculating the phase change of the stripes. Phase calculation is to obtain accurate depth parameters by performing phase unwrapping processing on the stripe pattern. The interference stripes formed by the stripe pattern on the surface are used to obtain depth information by calculating the phase. Assuming the phase is , depth value It is expressed by the following formula:

[0065] ;

[0066] in, is the wavelength of light, is the phase information of the fringe, depth Represents the three-dimensional coordinates of the surface at the corresponding pixel position. Phase unwrapping is used to solve the depth value ambiguity problem caused by phase jump, and restore the real depth information through algorithms such as multiple sampling and phase difference method. After obtaining the depth parameter of the bolt head, the reflection coefficient is calculated. The reflection coefficient reflects the intensity of light reflected from the surface of the bolt head, which is affected by factors such as surface material, surface smoothness, and incident angle. Assume that the reflection coefficient is ,in is the angle between the incident light and the normal. According to the Fresnel equation, the reflection coefficient is related to the incident angle The relationship between is expressed by the following formula:

[0067] ;

[0068] in, is the refractive index of the material, the reflection coefficient It is a key parameter for the interaction between surface material and light. The bolt head surface reflection matrix obtained by calculating the reflection coefficient helps the system understand the nature of the interaction between light and surface, and provides the necessary information for three-dimensional reconstruction. Using the reflection matrix, the surface normal vector and the angle between the light are calculated to obtain the spatial posture data of the bolt head. The spatial posture data is represented by the normal vector and the rotation matrix, which is used to describe the direction of the object in three-dimensional space. Assume that the surface normal vector is , the light direction is , angle Calculated by the following formula:

[0069] ;

[0070] in, is the surface normal vector, is the direction vector of the incident light, · represents the dot product operation, Represents the modulus of a vector. The position and orientation of the object in three-dimensional space are inferred by calculating the spatial posture data. After obtaining the depth parameter and spatial posture data of the bolt head, the mapping relationship between light intensity and position is established. Assume that the light intensity is , the location is , then the mapping relationship is established through the following function:

[0071] ;

[0072] This function describes the relationship between the light intensity distribution in the image and the position of each point in the three-dimensional space. Through this function, the light intensity information in the image can be reversed to obtain the spatial position data required for three-dimensional reconstruction. The reconstruction function of the bolt head is globally optimized to obtain accurate three-dimensional reconstruction parameters. The global optimization method finds the best parameter configuration by minimizing the error function. The error function is expressed as:

[0073] ;

[0074] in, is the error, is the pixel position on the image, It is the light intensity at a three-dimensional position obtained through the mapping relationship. By minimizing the error function, the three-dimensional reconstruction parameters of the bolt head are optimized. The three-dimensional reconstruction parameters are input into the reconstruction function, and the spatial back-projection calculation is performed to reconstruct the three-dimensional space through the known relationship between light intensity and position. The spatial back-projection calculation reversely infers the known image information back to the three-dimensional space through a method similar to projection geometry to obtain the three-dimensional reconstruction data of the bolt head.

[0075] In a specific embodiment, the process of executing step S3 may specifically include the following steps:

[0076] The bolt head profile feature data and the standard profile feature data are normalized and converted into probability distribution to obtain profile feature probability data, and the profile feature probability data is negatively logarithmically calculated to obtain profile cross entropy loss data;

[0077] The depth feature data of the bolt head is subjected to difference calculation with the standard depth feature data to obtain the depth feature error data, and the depth feature error data is subjected to mean calculation to obtain the depth mean square error loss data;

[0078] A first loss weight coefficient is set according to the contour cross entropy loss data to calculate the contour loss weighted data, and a second loss weight coefficient is set according to the depth mean square error loss data to calculate the depth loss weighted data;

[0079] The contour loss weighted data and the depth loss weighted data are linearly combined to obtain a comprehensive loss function.

[0080] Specifically, the profile feature data of the bolt head is normalized with the standard profile feature data to eliminate the influence of different data ranges. Assume that the profile feature data of the bolt head is , the standard contour feature data is , which are represented as a -dimensional vector, where is the dimension of the data. Each dimension of data is standardized using the mean and standard deviation. The specific formula is as follows:

[0081] ;

[0082] ;

[0083] in, and They are the first dimension, and are the mean and standard deviation of the bolt head data, and is the mean and standard deviation of the standard contour data. The normalized contour feature data and the standard contour feature data are converted into probability distribution data. For example, assuming that Gaussian distribution is used to model these data, the probability density function obtained is and P are the probability distributions of contour feature data and standard data respectively. The obtained contour feature probability data is expressed as:

[0084] ;

[0085] ;

[0086] Through these probability distribution data, the negative logarithm is calculated to calculate the cross entropy loss. Cross entropy is a common indicator to measure the difference between two probability distributions. The calculation formula of cross entropy is:

[0087] ;

[0088] in, is the probability distribution of the contour feature The value of the data point, is the corresponding probability value of the standard contour feature data. The cross entropy loss measures the difference between the contour feature distribution predicted by the model and the standard contour feature distribution. is the probability distribution of the contour feature The value of the data point, is the corresponding probability value of the standard contour feature data. The cross entropy loss measures the difference between the contour feature distribution predicted by the model and the standard contour feature distribution. The depth feature data of the bolt head is calculated by difference with the standard depth feature data. Assume that the depth feature data of the bolt head is , the standard depth feature data is , the difference calculation formula is:

[0089] ;

[0090] in, It is The depth feature error data represents the difference between the predicted depth and the standard depth. To calculate the mean square error loss, all depth error values ​​are squared and their mean is calculated:

[0091] ;

[0092] in, is the dimension of the deep feature data, It is The square of the depth error. Set different weight coefficients for the contour cross entropy loss and the depth mean square error loss. Assume that the weight coefficient of the contour loss is , the weight coefficient of the depth loss is , the weighted contour loss and depth loss are obtained by the following formula:

[0093] ;

[0094] ;

[0095] in, and is the weight coefficient, reflecting the importance of contour loss and depth loss in the final loss function. These coefficients are adjusted according to actual needs and experimental results. The comprehensive loss function is a linear combination of these two weighted loss terms:

[0096] ;

[0097]

[0098] The comprehensive loss function includes the errors of the bolt head profile feature and the depth feature, and adjusts the weight coefficient and , balancing the contributions of the two in the loss function, and thus optimizing the performance of the model. By minimizing the comprehensive loss function, the model can simultaneously optimize the prediction accuracy of contour features and depth features, achieving more accurate bolt head height detection.

[0099] In a specific embodiment, the process of executing step S4 may specifically include the following steps:

[0100] The reflection intensity term in the 3D reconstruction data of the bolt head is replaced with a relaxation variable, and the first constraint condition is introduced to obtain a reflection intensity relaxation model;

[0101] The tilt angle term in the 3D reconstruction data of the bolt head is replaced with a relaxation variable, and a second constraint condition is introduced to obtain a tilt angle relaxation model.

[0102] Substitute the reflection intensity relaxation model and the tilt angle relaxation model into the comprehensive loss function, and obtain the initial convex optimization objective function through variable substitution;

[0103] The initial convex optimization objective function is Taylor expanded, and the nonlinear terms are linearized to obtain the linearized objective function. Convex constraints are added to the linearized objective function, and the augmented Lagrangian model is constructed by the interior point method.

[0104] Based on the augmented Lagrangian model, a sequential convex programming iteration format is constructed. By setting the iteration step size and termination condition, the iterative optimization rule is obtained. The convex optimization solution is solved according to the iterative optimization rule. The optimization variables are updated by the alternating direction multiplier method to obtain a converged solution.

[0105] The converged solution is substituted into the augmented Lagrangian model, and the physical variables are restored through reverse calculation to obtain the convex optimization model for bolt head measurement.

[0106] Specifically, when processing the reflection intensity item, it is assumed that the reflection intensity item in the 3D reconstruction data of the bolt head is , which exists in the observed data of the model and is not completely accurate due to noise or other factors. In order to introduce a slack variable, a new slack variable is used. Alternative Reflectivity Item , and the reflection intensity relaxation model is obtained. The introduction of relaxation variables helps to alleviate the uncertainty in the reflection intensity calculation and enables the subsequent optimization process to tolerate a certain range of errors. The reflection intensity relaxation model is expressed as:

[0107] ;

[0108] in, is the reflection intensity data after the slack variables are introduced, is the original reflection intensity data, is the first constraint, is the tolerance error of the constraint, which is determined by experiments or data analysis. Consider the tilt angle term in the 3D reconstruction data of the bolt head, assuming that the angle is Similarly, to deal with the error or uncertainty in the tilt angle, a new slack variable is introduced , which replaces the original tilt angle data , and the tilt angle relaxation model is obtained. The introduction of relaxation variables can eliminate certain errors and make the model more flexible in practical applications. The tilt angle relaxation model is expressed as:

[0109] ;

[0110] in, is the tilt angle data after the slack variable is introduced, is the original tilt angle data, is the second constraint, is the tolerance error of the constraint. Substituting the reflection intensity relaxation model and the tilt angle relaxation model into the comprehensive loss function, the obtained objective function will contain these two relaxation variables, and the initial convex optimization objective function is obtained by variable substitution. The comprehensive loss function contains multiple terms, which represent the reflection intensity, tilt angle and other features related to the three-dimensional reconstruction of the bolt head. The comprehensive loss function is expressed as:

[0111] ;

[0112] in, and are weight coefficients, which respectively control the importance of reflection intensity, tilt angle and other feature items in the comprehensive loss function. is other relevant feature data, is the standard data. Taylor expansion is performed on the initial convex optimization objective function, and the nonlinear terms are linearized. Taylor expansion is a linearization method that converts complex nonlinear problems into linear problems by approximating the behavior of the original function at a certain point. The nonlinear terms are approximated by Taylor expansion to obtain the linearized objective function:

[0113] ;

[0114] in, and are the initial values ​​of the current optimization variables, is the gradient of the objective function. In order to make the objective function more suitable for solving, a convex constraint is added to the linearized objective function to construct an augmented Lagrangian model. The augmented Lagrangian method is a method that integrates constraints into the objective function and can effectively solve constrained optimization problems. The form of the augmented Lagrangian model is:

[0115] ;

[0116] in, and is the Lagrange multiplier, and is the augmented parameter. Based on the augmented Lagrangian model, a sequential convex programming iterative format is constructed. Sequential convex programming gradually optimizes the objective function by recursively updating variables and constraints. In each step, the iterative optimization rule controls the optimization process by setting appropriate step sizes and termination conditions. The result of each iterative calculation will update the optimization variable until the convergence condition is met. The optimization variable is updated through the alternating direction multiplier method to obtain the optimal solution. The core idea of ​​the alternating direction multiplier method is to decompose the original problem into several simpler sub-problems and solve these sub-problems alternately to speed up the optimization process. The updated formula is as follows:

[0117] ;

[0118] ;

[0119] Through multiple iterations, a converged solution is obtained. The convergence condition is that when the change of the optimization variable is less than a certain set threshold, the optimization is considered to have converged. By substituting the converged solution into the augmented Lagrangian model and reversing the calculation to restore the physical variables, a convex optimization model for bolt head measurement is obtained. The physical variables include the three-dimensional position, rotation angle, and reflection intensity of the bolt. These physical quantities are derived and calculated through the parameters obtained by optimization.

[0120] In a specific embodiment, the process of executing step S5 may specifically include the following steps:

[0121] Sub-pixel edge detection is performed on the bolt head contour feature data, and refined contour data is obtained through the grayscale centroid method;

[0122] Morphological processing is performed based on the refined contour data, and contour optimization data is obtained through opening and closing operations;

[0123] Conduct connected domain analysis on the contour optimization data, obtain closed contour data through edge tracking algorithm, and perform regional segmentation based on the closed contour data to obtain depth segmentation data;

[0124] The depth segmentation data is input into the depth extraction module, and the edge depth data is obtained by surface fitting. The edge depth data is then removed for outliers to obtain depth optimization data.

[0125] A depth distribution function is constructed according to the depth optimization data to obtain depth mapping data, and the depth mapping data is subjected to coordinate transformation to obtain bolt head detection data.

[0126] Specifically, sub-pixel edge detection is performed on the contour feature data of the bolt head. Sub-pixel edge detection is a more detailed process based on traditional edge detection. It can locate the edge position through a more accurate algorithm and improve the fineness of the edge. Use basic edge detection methods such as the Canny edge detection algorithm or the Sobel operator, and then combine the grayscale centroid method to perform sub-pixel level refinement. The grayscale centroid method locates the edge by calculating the centroid position of the grayscale value of the image area. The formula is expressed as:

[0127] ;

[0128] in, is the position of the pixel, is the gray value at that position, and are the centroid positions of the grayscale values ​​of the region, respectively. This method can obtain contour data that is more refined than the traditional pixel-level edge. The refined contour data is morphologically processed to optimize the contour. Morphological operation is an image processing method based on set theory, which is used to remove noise and disconnected areas. Use open and closed operations to improve the quality of contour data. The open operation removes small noise points by first corroding and then dilating, while the closed operation is first dilating and then corroding, which helps to fill small holes and breaks in the contour. The open operation is expressed as:

[0129] ;

[0130] in, is the image to be processed, is a structural element, is the structural element at point The morphological transformation of . The formula for the closing operation is:

[0131] ;

[0132] Through these operations, small noises and breaks in the contour are effectively processed to obtain contour optimization data. The optimized contour data is subjected to connected domain analysis, and the integrity of the contour data is improved through the edge tracking algorithm. Connected domain analysis is used to identify connected areas in the image, especially in the case of large noise, and can effectively separate and extract target objects. The edge tracking algorithm is used to track the shape of the contour along the edge curve to ensure the closure of the contour. Edge tracking methods include gradient tracking and edge extension algorithms, which help the system extract continuous and complete closed contour data from preliminary contour data. For the closed part of the contour, the regional segmentation method is used to process it. By segmenting the closed contour area, multiple representative regions are obtained, and the depth features of each region are extracted separately. In the depth segmentation process, partitions are performed based on the differences in gradient and depth values ​​of the region. This method can effectively extract the depth features of different regions. The formula is expressed as:

[0133] ;

[0134] in, Indicates the depth value within the area. is the threshold of depth segmentation, ensuring that each segmented area is distinguished in depth. The depth segmentation data is input to the depth extraction module, which extracts accurate depth data from the segmented area based on the surface fitting method. Surface fitting uses the least squares method or other optimization algorithms to fit the depth data in the area to obtain the depth value of each area. Assume that the depth value of a certain area is obtained by a two-dimensional function To model the surface, the goal of surface fitting is to minimize the fitting error, which can be expressed as:

[0135] ;

[0136] in, are the pixel coordinates within the region, is the actual depth value of the location. By minimizing the error, a fitting surface is obtained, thereby extracting the depth data of each area. Outliers are removed from the edge depth data. Outliers are invalid data caused by measurement errors, noise, or inaccurate algorithms. Outlier removal is achieved through statistical methods, such as judging and removing data points that do not conform to statistical laws based on the Z-score method or the IQR method. For example, the conditions for using Z-score to remove outliers are:

[0137] ;

[0138] in, is the depth value Z-score, is the threshold value, usually set to 2 or 3. After removing outliers, more accurate depth optimization data is obtained. According to the depth optimization data, the depth distribution function is constructed and the coordinates are transformed to obtain the final inspection data of the bolt head. The depth distribution function is constructed by statistically analyzing the distribution of depth values ​​and fitting with Gaussian distribution or other appropriate distributions. When constructing the depth distribution function, the maximum likelihood estimation method is used to fit the data. The formula is:

[0139] ;

[0140] in, is the parameter of the distribution, It is the depth data The probability density function under the distribution. Through this method, accurate depth mapping data is obtained. The depth mapping data is transformed into the actual physical coordinate system of the bolt head, and finally the detection data of the bolt head is obtained. The depth data is matched with the image coordinates, and the coordinate transformation is realized through the projection matrix or other geometric transformation methods.

[0141] In a specific embodiment, the process of executing step S6 may specifically include the following steps:

[0142] Perform principal component analysis on the bolt head inspection data and the bolt head measurement convex optimization model to obtain dimension-reduced feature data;

[0143] Perform K-means clustering on the reduced-dimensional feature data to obtain the feature center sequence;

[0144] A Gaussian distribution model group is constructed based on the feature center sequence, and the distribution parameter matrix is ​​obtained through the EM algorithm;

[0145] The distribution parameter matrix is ​​optimized to obtain the hybrid weight coefficient, and the hybrid weight coefficient is weightedly fused with the bolt head detection data to obtain the feature weight data;

[0146] The feature weight data is similarly calculated with the bolt head measurement convex optimization model to obtain feature matching data;

[0147] Based on the weighted average of the feature matching data, a fused feature vector is obtained, and the fused feature vector is spatially transformed to obtain the bolt head height detection result.

[0148] Specifically, principal component analysis is performed on the bolt head inspection data and the bolt head measurement convex optimization model to extract the most representative features from the high-dimensional data. Principal component analysis is a dimensionality reduction technique that projects the data into a new coordinate system through orthogonal transformation to maximize the variance of the data. When performing principal component analysis, the data matrix (Assumption ,in is the sample size, is the number of features) is centered by subtracting the mean of each column:

[0149] ;

[0150] in, is the data matrix The mean vector of each column. Calculate the covariance matrix:

[0151] ;

[0152] Covariance matrix Describe the relationship between the various dimensions of the data. By calculating the eigenvalues ​​and eigenvectors of the covariance matrix, the principal components of the data are obtained, and the front principal components to reduce the dimension of the data and form a new low-dimensional feature matrix:

[0153] ;

[0154] in, Is included before The matrix of eigenvectors, It is the feature data after dimensionality reduction. The feature data after dimensionality reduction helps the system reduce the redundant information of the data while retaining most of the variability of the data. K-means clustering algorithm is applied to the feature data after dimensionality reduction in order to classify the feature data into different classes. K-means clustering is an unsupervised learning algorithm used to classify data into clusters, where the similarity between samples in each cluster is as high as possible, and the differences between samples in different clusters are as large as possible. In the K-means algorithm, randomly select The initial center point (i.e., the cluster center of the cluster) is obtained, and then the cluster center is updated iteratively. Each iteration includes two steps: assigning each sample to the cluster center closest to it and recalculating the center point of each cluster. clusters, and the center point of each cluster represents the characteristic center of the cluster. Assume It is The objective function of K-means clustering is expressed as:

[0155] ;

[0156] in, Indicates Clusters, Belongs to A sample of is the cluster center. By optimizing this objective function, we get the feature center sequence , and a Gaussian distribution model is constructed based on it. Based on the feature center sequence, a Gaussian distribution model group is constructed to describe the distribution characteristics of different clusters. Gaussian distribution is a probability distribution model. The feature data of each cluster is modeled with a Gaussian distribution. Assume that The data distribution of the clusters is Gaussian distribution ,in is the mean vector, is the covariance matrix. To estimate the parameters of these Gaussian distributions, the EM algorithm is used. The EM algorithm is an iterative optimization algorithm used to estimate model parameters in the presence of latent variables. For the Gaussian mixture model, the EM algorithm process consists of two steps: the expectation step (E step) and the maximization step (M step). In the E step, given the current model parameters and sample data , calculate the posterior probability of each data point belonging to each cluster, that is

[0157] ;

[0158] in, It is The mixing coefficient of the clusters, It is The Gaussian distribution probability density function of each cluster. The M step is to maximize the likelihood function and update the model parameters. The specific update formula is:

[0159] ;

[0160] By iteratively updating these parameters until the model converges, the distribution parameter matrix of each cluster is obtained and . Optimize the distribution parameter matrix to obtain the mixing weight coefficient. Mixing weight coefficient Represents the prior probability of each cluster, reflecting the importance of the cluster in the overall data. Optimizing these weight coefficients can improve the model's fit to the data. The optimization method uses the gradient descent method to solve the optimal and other parameters. The hybrid weight coefficient is weighted and fused with the bolt head detection data. Each feature vector is weighted according to the hybrid weight of its corresponding cluster to obtain the weighted feature weight data. Assume It is feature vectors, and the mixing weight coefficient is , the weighted feature weight data is expressed as:

[0161] ;

[0162] The weighted feature weight data is similarly calculated with the convex optimization model of the bolt head measurement. The degree of matching between the weighted features and the convex optimization model is verified by calculating the similarity between the weighted features and the convex optimization model. Similarity calculation methods include cosine similarity or Euclidean distance. For example, cosine similarity is expressed as:

[0163] ;

[0164] in, is the eigenvector of the convex optimization model for bolt head measurement, is the weighted feature vector. The fused feature vector is obtained by weighted averaging based on feature matching data. According to the contribution of different features, a comprehensive feature vector is obtained by combining all feature information:

[0165] ;

[0166] in, is the weight coefficient of each feature, which is obtained through model training or optimization. The fused feature vector is spatially transformed to finally obtain the height detection result of the bolt head. The spatial transformation is achieved through rotation matrix or other transformation methods to ensure that the final detection result matches the actual physical space.

[0167] The above describes the bolt head height detection method in the embodiment of the present invention. The following describes the bolt head height detection system in the embodiment of the present invention. Figure 2 , an embodiment of a bolt head height detection system in an embodiment of the present invention includes:

[0168] A feature extraction module is used to input the head image of the high-lock bolt into the encoder-decoder for feature extraction to obtain the bolt head contour feature data and the bolt head depth feature data;

[0169] The projection module is used to establish a three-dimensional reconstruction mathematical model according to the contour feature data of the bolt head, and perform structured light projection on the head of the high-locking bolt to obtain the three-dimensional reconstruction data of the bolt head;

[0170] A calculation module is used to perform cross entropy loss calculation on the bolt head profile feature data and mean square error loss calculation on the bolt head depth feature data to obtain a comprehensive loss function;

[0171] A construction module is used to perform slack variable conversion on the three-dimensional reconstruction data of the bolt head based on a comprehensive loss function, and to construct an objective function using a sequential convex programming method to obtain a convex optimization model for bolt head measurement;

[0172] The detection module is used to perform edge detection on the bolt head contour feature data, and calculate the edge area depth value in combination with the bolt head depth feature data to obtain the bolt head detection data;

[0173] The fusion module is used to perform feature fusion calculation based on the bolt head detection data and the bolt head measurement convex optimization model through the Gaussian mixture model to obtain the bolt head height detection result.

[0174] Through the cooperation of the above components, a dual-channel encoder-decoder network structure is constructed to realize the parallel extraction of bolt head contour features and depth features. The first channel uses BatchNormalization layer and ReLU activation function to extract contour features, and the second channel introduces jump connection structure and LeakyReLU activation function to extract depth features, which effectively solves the problem of insufficient feature extraction. Structured light projection technology is combined with phase unwrapping algorithm for three-dimensional reconstruction. By establishing reflectivity parameter model and attitude parameter model, the reflective characteristics and spatial attitude characteristics of the bolt surface are accurately described, overcoming the difficulty of traditional methods in modeling complex surfaces. Relaxation variables are introduced to transform the reflective intensity term and the tilt angle term, and the sequential convex programming method is used to construct the objective function. The augmented Lagrangian model and alternating direction multiplier method are used for optimization and solution, successfully transforming the non-convex optimization problem into a convex optimization problem. Combining sub-pixel edge detection and morphological processing technology, the accuracy of contour features is improved through opening and closing operations and connected domain analysis. At the same time, surface fitting and outlier removal methods are used to optimize depth data, which improves the accuracy of depth value calculation in edge areas.

[0175] A processor is also provided in an embodiment of the present invention, and the processor is used to implement the above method when executed.

[0176] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0177] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0178] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0179] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0180] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting the head height of a bolt, characterized in that: The method comprises: The head image of the high-lock bolt is input into the encoder-decoder for feature extraction to obtain the bolt head contour feature data and the bolt head depth feature data; A three-dimensional reconstruction mathematical model is established according to the bolt head contour feature data, and structured light projection is performed on the head of the high-locking bolt to obtain three-dimensional reconstruction data of the bolt head; Performing cross entropy loss calculation on the bolt head profile feature data, and performing mean square error loss calculation on the bolt head depth feature data, to obtain a comprehensive loss function; Based on the comprehensive loss function, the three-dimensional reconstruction data of the bolt head is transformed into slack variables, and the objective function is constructed using a sequential convex programming method to obtain a convex optimization model for bolt head measurement; specifically, the following steps are performed: the reflection intensity item in the three-dimensional reconstruction data of the bolt head is replaced with a slack variable, and a first constraint condition is introduced to obtain a reflection intensity relaxation model; the inclination angle item in the three-dimensional reconstruction data of the bolt head is replaced with a slack variable, and a second constraint condition is introduced to obtain a inclination angle relaxation model; the reflection intensity relaxation model and the inclination angle relaxation model are substituted into the comprehensive loss function, and an initial convex optimization model is obtained by variable replacement. Objective function; Taylor expansion is performed on the initial convex optimization objective function, and nonlinear terms are linearized to obtain a linearized objective function, convex constraints are added to the linearized objective function, and an augmented Lagrangian model is constructed by an interior point method; a sequential convex programming iteration format is constructed based on the augmented Lagrangian model, and an iterative optimization rule is obtained by setting an iteration step size and a termination condition, and a convex optimization solution is performed according to the iterative optimization rule, and the optimization variables are updated by an alternating direction multiplier method to obtain a converged solution; the converged solution is substituted into the augmented Lagrangian model, and the physical variables are restored by reverse calculation to obtain a convex optimization model for bolt head measurement; Performing edge detection on the bolt head contour feature data, and calculating the edge area depth value in combination with the bolt head depth feature data to obtain bolt head detection data; According to the bolt head detection data and the bolt head measurement convex optimization model, feature fusion calculation is performed through a Gaussian mixture model to obtain a bolt head height detection result.

2. The method for detecting the head height of a bolt according to claim 1, characterized in that: The head image of the high-lock bolt is input into the encoder-decoder for feature extraction to obtain the bolt head contour feature data and the bolt head depth feature data, including: The head image of the high-lock bolt is divided into a contour feature extraction channel and a depth feature extraction channel; Constructing a first encoder-decoder network according to the contour feature extraction channel, wherein the encoder of the first encoder-decoder network comprises 5 convolutional layers, each of which is connected to a BatchNormalization layer and a ReLU activation function, and the decoder of the first encoder-decoder network comprises 5 deconvolutional layers, each of which is connected to a BatchNormalization layer and a ReLU activation function; Inputting the head image of the high-lock bolt into the encoder of the first encoder-decoder network, and obtaining contour feature space data through 5-layer convolution operation, wherein the convolution kernel size of the 5-layer convolution operation is 3×3, the step size is 2, and the number of channels is 64, 128, 256, 512, and 512 respectively; Performing 5 layers of deconvolution operations according to the contour feature space data, wherein the convolution kernel sizes of the 5 layers of deconvolution operations are 3×3, the step size is 2, and the number of channels is 512, 256, 128, 64, and 1, respectively, to obtain initial contour feature data; Constructing a second encoder-decoder network according to the deep feature extraction channel, wherein the encoder of the second encoder-decoder network comprises 5 convolutional layers with skip connections, each convolutional layer is connected to a BatchNormalization layer and a LeakyReLU activation function, and the decoder of the second encoder-decoder network comprises 5 deconvolutional layers, each deconvolutional layer is connected to a BatchNormalization layer and a LeakyReLU activation function; Input the head image of the high-lock bolt into the encoder of the second encoder-decoder network, and obtain deep feature space data through 5 layers of convolution operations with skip connections, where the convolution kernel size of the 5 layers of convolution operations is 3×3, the step size is 2, and the number of channels is 64, 128, 256, 512, and 512 respectively; Performing 5 layers of deconvolution operations according to the deep feature space data, and fusing the jump connection features of the corresponding coding layer in each layer of deconvolution operations, the convolution kernel sizes of the 5 layers of deconvolution operations are 3×3, the step size is 2, and the number of channels is 512, 256, 128, 64, and 1, respectively, to obtain initial deep feature data; The initial contour feature data and the initial depth feature data are respectively normalized and feature aligned to obtain bolt head contour feature data and bolt head depth feature data.

3. The method for detecting the head height of a bolt according to claim 2, characterized in that: The three-dimensional reconstruction mathematical model is established according to the bolt head contour feature data, and structured light projection is performed on the head of the high-locking bolt to obtain the three-dimensional reconstruction data of the bolt head, including: Performing spatial coordinate mapping on the bolt head profile feature data to obtain the initial position data of the bolt head, and projecting the striped light spot pattern generated by the structured light projector onto the head surface of the high-locking bolt to obtain the bolt head structured light data; Performing phase calculation and unwrapping processing on the structured light data of the bolt head to obtain a depth parameter of the bolt head, and performing reflection coefficient calculation according to the initial position data of the bolt head to obtain a surface reflection matrix of the bolt head; Based on the bolt head surface reflection matrix, the surface normal vector and the angle between the light are calculated to obtain the bolt head spatial posture data, and according to the bolt head depth parameter and the bolt head spatial posture data, a mapping relationship between light intensity and position is established to obtain a bolt head reconstruction function; The bolt head reconstruction function is globally optimized to obtain three-dimensional reconstruction parameters, and the three-dimensional reconstruction parameters are input into the bolt head reconstruction function to perform spatial back-projection calculation to obtain three-dimensional reconstruction data of the bolt head.

4. The method for detecting the head height of a bolt according to claim 3, characterized in that: The cross entropy loss calculation is performed on the bolt head profile feature data, and the mean square error loss calculation is performed on the bolt head depth feature data to obtain a comprehensive loss function, including: The bolt head profile feature data and the standard profile feature data are normalized and converted into probability distribution to obtain profile feature probability data, and negative logarithm calculation is performed on the profile feature probability data to obtain profile cross entropy loss data; Performing difference calculation on the bolt head depth feature data and standard depth feature data to obtain depth feature error data, and performing mean calculation on the depth feature error data to obtain depth mean square error loss data; A first loss weight coefficient is set according to the contour cross entropy loss data to calculate contour loss weighted data, and a second loss weight coefficient is set according to the depth mean square error loss data to calculate depth loss weighted data; The contour loss weighted data and the depth loss weighted data are linearly combined to obtain a comprehensive loss function.

5. The method for detecting the head height of a bolt according to claim 1, characterized in that: The edge detection is performed on the bolt head contour feature data, and the edge area depth value is calculated in combination with the bolt head depth feature data to obtain the bolt head detection data, including: Performing sub-pixel edge detection on the bolt head contour feature data, and obtaining refined contour data by using a grayscale centroid method; Performing morphological processing according to the refined contour data, and obtaining contour optimization data through opening and closing operations; Performing a connected domain analysis on the contour optimization data, obtaining closed contour data through an edge tracking algorithm, and performing region segmentation according to the closed contour data to obtain depth segmentation data; The depth segmentation data is input into a depth extraction module, edge depth data is obtained by surface fitting, and outliers are removed from the edge depth data to obtain depth optimization data; A depth distribution function is constructed according to the depth optimization data to obtain depth mapping data, and coordinate transformation is performed on the depth mapping data to obtain bolt head detection data.

6. The method for detecting the head height of a bolt according to claim 5, characterized in that: The method of performing feature fusion calculation based on the bolt head detection data and the bolt head measurement convex optimization model through a Gaussian mixture model to obtain a bolt head height detection result includes: Performing principal component analysis on the bolt head detection data and the bolt head measurement convex optimization model to obtain dimension-reduced feature data; Performing K-means clustering on the dimension-reduced feature data to obtain a feature center sequence; Constructing a Gaussian distribution model group according to the characteristic center sequence, and obtaining a distribution parameter matrix by using an EM algorithm; Optimizing the distribution parameter matrix to obtain a mixed weight coefficient, and weightedly fusing the mixed weight coefficient with the bolt head detection data to obtain feature weight data; Performing similarity calculation on the feature weight data and the bolt head measurement convex optimization model to obtain feature matching data; A weighted average is performed based on the feature matching data to obtain a fused feature vector, and the fused feature vector is spatially transformed to obtain a bolt head height detection result.

7. A bolt head height detection system, characterized in that: Used to perform the bolt head height detection method according to any one of claims 1 to 6, the bolt head height detection system comprising: A feature extraction module is used to input the head image of the high-lock bolt into the encoder-decoder for feature extraction to obtain the bolt head contour feature data and the bolt head depth feature data; A projection module is used to establish a three-dimensional reconstruction mathematical model according to the bolt head contour feature data, and perform structured light projection on the head of the high-locking bolt to obtain three-dimensional reconstruction data of the bolt head; A calculation module, used to perform cross entropy loss calculation on the bolt head profile feature data, and perform mean square error loss calculation on the bolt head depth feature data, to obtain a comprehensive loss function; A construction module is used to perform slack variable conversion on the three-dimensional reconstruction data of the bolt head based on the comprehensive loss function, and to construct an objective function using a sequential convex programming method to obtain a convex optimization model for bolt head measurement; A detection module, used to perform edge detection on the bolt head contour feature data, and calculate the edge area depth value in combination with the bolt head depth feature data to obtain bolt head detection data; The fusion module is used to perform feature fusion calculation through a Gaussian mixture model based on the bolt head detection data and the bolt head measurement convex optimization model to obtain the bolt head height detection result.

8. A processor, characterized in that: The processor is used to implement the bolt head height detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the processor is enabled to execute the method for detecting the head height of a bolt according to any one of claims 1 to 6.

Images