Fuse tightening direction judgment method based on deep learning under industrial vision

Abstract

The invention discloses a fuse tightening direction judgment method based on deep learning under industrial vision. The method comprises the steps of 1, fuse data acquisition; step 2, image processing; 3, building a network model; and 4, the tightening direction is judged. Through the method provided by the invention, the fuse detection accuracy and tightening direction judgment can be effectively improved, the safety of industrial scene device installation and operation is improved, and the problems caused by lack of fuses or wrong directions are prevented.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method for determining the tightening direction of fuses based on deep learning under industrial vision. Background Technology

[0002] In industrial vision applications, fuses, as crucial fastening components, are typically made of two or more strands of twisted metal wire. Their main function is to prevent connections from loosening due to vibration or other external forces through mechanical binding. This component has two typical installation methods: one is to bridge two screws and wrap it around the target object; the other is to use a single-end fixing method, where the movable end is wrapped around the object's surface to form an anti-loosening structure. When the system lacks a fuse, under continuous vibration, the screw connections may gradually loosen due to the lack of sustained force, eventually leading to threaded connection failure or even fastener detachment. This protective mechanism plays a critical role in maintaining the long-term stable operation of industrial equipment.

[0003] The anti-loosening function of a fuse relies on its winding direction matching the tightening direction of the screw (usually clockwise). If the vision system fails to correctly identify the tightening direction, the fuse may be wound in the opposite direction (e.g., counterclockwise), making it ineffective at tightening screws under vibration, or even accelerating loosening. Automated production lines rely on vision systems for quality inspection; incorrect fuse winding direction can lead to assembly failure. For example, a reverse-wound fuse may come loose during equipment operation, causing safety hazards or equipment malfunctions.

[0004] In recent years, object detection technology has been increasingly widely applied in various fields. Its core function is to locate the position of the target in an image and perform category recognition, typically using bounding boxes as the annotation method. However, in industrial vision inspection scenarios, the placement orientation of fuses is random, with perfectly horizontal or vertical distributions being rare. This makes it difficult for traditional horizontal bounding boxes to achieve accurate calibration. Specifically, the orientation of the image capture and fuse installation may be arbitrary, and their spatial distribution may be tilted at any angle, introducing significant annotation errors into horizontal / vertical bounding boxes. Furthermore, due to the special shape during installation, the drawn annotation boxes are difficult to adapt to the true geometric features of the fuse object. At the same time, using bounding boxes will include too much irrelevant background area, diluting the effective features of the target and affecting the accuracy of the detection model. In addition, when a fuse is already present in the detected image, how to correctly and efficiently identify the fuse orientation to improve detection accuracy and recognition efficiency remains a major challenge. Summary of the Invention

[0005] The purpose of this invention is to solve the problems mentioned in the background art and to propose a method for determining the tightening direction of fuses based on deep learning under industrial vision.

[0006] To achieve the objective of this invention, this invention provides a method for determining the tightening direction of a fuse based on deep learning under industrial vision, the method comprising the following steps:

[0007] Step 1: Fuse data acquisition;

[0008] Step 2: Image processing;

[0009] Step 3: Network model construction;

[0010] Step 4: Determine the tightening direction.

[0011] Furthermore, step 1 specifically includes the following steps:

[0012] Step 1-1: Deploy multiple sensors (industrial cameras, endoscopes) on the production line to acquire clearly visible images;

[0013] Steps 1-2: Collect data from different batches of the same model at multiple time periods to expand the dataset size and enrich the image features of the fuse dataset.

[0014] Furthermore, step 2 specifically includes the following steps:

[0015] Step 2-1: Annotate the acquired image using a rotating bounding box, using a uniform annotation format: (x1, y1)(x2, y2)(x3, y3)(x4, y4), which represent the coordinates of the four vertices of the annotation box, respectively;

[0016] Step 2-2: Normalize, denoise, and align the data, and divide the training set, validation set, and test set into a 6:2:2 ratio.

[0017] Steps 2-3: Store the data in the corresponding path.

[0018] Furthermore, step 3 specifically includes the following steps:

[0019] Step 3-1: Constructing the YOLO11 network: Initially extracting image features using the backbone network;

[0020] Step 3-2: Upsample the features using the Neck network and fully fuse them with the original features to obtain the fused image features;

[0021] Step 3-3: Input the fused features into the OBB rotating bounding box detection head. The detection head will output the coordinate information of the four vertices of the detection box, as well as the rotation angle of the detection box.

[0022] Steps 3-4: Adjust hyperparameters, such as the number of training rounds, batch size, and learning rate, to improve detection accuracy.

[0023] Furthermore, step 4 specifically includes the following steps:

[0024] Step 4-1: Based on the output coordinate information, construct the corresponding vector to represent the relative position of the screw and the fuse;

[0025] Step 4-2: Determine the calculation method as the vector outer product, extend the vector on the two-dimensional coordinates to three-dimensional space, and calculate the result;

[0026] Step 4-3: Determine whether the fuse tightening direction meets the requirements based on the calculation results, and calculate the detection accuracy rate.

[0027] Compared with the prior art, the significant advancement of the present invention lies in the following: Compared with the prior art, the beneficial effects of the present invention are as follows:

[0028] This paper proposes a deep learning-based method for determining the tightening direction of fuses under industrial vision. This method combines the high accuracy of deep learning with the simplicity and convenience of traditional mathematical vector cross product methods. Specifically, the proposed method offers the following benefits:

[0029] (1) By using deep learning methods and combining them with rotating bounding box annotation, the shortcomings of traditional object detection methods, such as strong environmental sensitivity, poor pose adaptability, complex algorithm process, and poor robustness and generalization, are avoided.

[0030] (2) The introduction of a rotating frame reduces the impact of background noise and makes it less likely to cause recognition errors when the aspect ratio of the target changes significantly.

[0031] (3) In the backbone network, a variant of the BottleNeck module, the C3k2 module, is introduced. This module allows for custom kernel size adjustments and extracts features from the network by segmenting feature maps and applying smaller (3×3) convolutions. Compared to larger convolution kernels, it is faster and has lower computational cost. Furthermore, the Cross-Stage Local Spatial Attention (C2PSA) module enhances spatial attention in feature maps, improving the model's focus on important parts of the image. Through spatial pooling features, the model can more effectively focus on specific regions of interest.

[0032] (4) In the neck network, the feature maps obtained by upsampling and the backbone network are fully fused to obtain both low-level features such as color and texture and high-level feature information. By fusing low-level and high-level features, information from different features can be fully utilized, and feature maps with richer information can be output to downstream tasks to improve detection performance.

[0033] (5) In the Head network, the original detection head is modified into a new rotating target detection head. This detection head can predict the rotation angle of the object in addition to predicting the position and category of the object to be detected by the original detection head, thereby improving the detection accuracy.

[0034] (6) At the same time, when determining the position of screws and fuses, the use of vector outer product can ensure that the implementation is simple and efficient, and can also achieve good accuracy, avoiding the disadvantages of using deep learning methods that are cumbersome and time-consuming.

[0035] The above methods can effectively improve the detection accuracy of fuses and the determination of tightening direction. The detection mAP reaches 93.2%, and the direction determination accuracy reaches 99%, which improves the safety of device installation and operation in industrial scenarios and prevents problems caused by missing fuses or incorrect direction.

[0036] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0037] Figure 1 This is an overall flowchart of an embodiment of the present invention;

[0038] Figure 2 This is a flowchart of step 1 in an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of step 2 in an embodiment of the present invention;

[0040] Figure 4 This is a flowchart of step 3 in an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of step 4 in an embodiment of the present invention;

[0042] Figure 6 This is a schematic diagram of the YOLO11 backbone network + neck network model structure according to an embodiment of the present invention;

[0043] Figure 7 This is a schematic diagram of the YOLO11 rotation detection head module according to an embodiment of the present invention;

[0044] Figure 8 This is a schematic diagram of a single module of the C3K2 embodiment of the present invention;

[0045] Figure 9 This is a schematic diagram of a single module of the C2PSA embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Addressing the shortcomings of traditional fuse tightening identification methods: 1. High degree of arbitrariness in orientation, resulting in poor robustness and generalization. Fuses have unique morphological characteristics, generally being relatively thin and long with some curvature. Simultaneously, the screw installation position is relatively unpredictable, and variations in the shooting angle lead to high arbitrariness in the orientation of the resulting fuse image. This can prevent correct identification of fuses of the same type but with different orientations, causing poor robustness and generalization of the model. 2. Abundant background information and introduction of significant noise. Traditional rectangular bounding boxes are horizontal; however, the unique shape of fuses means that after using rectangular bounding boxes, the proportion of fuse features is relatively small, and a large amount of background features are introduced, significantly impacting subsequent detection accuracy. 3. Tightening direction determination. Current methods mostly lack a fixed method for determining the tightening direction, making them susceptible to image quality and other factors, resulting in low detection accuracy. This invention discloses a method for determining the tightening direction of fuses based on deep learning in industrial vision. It extracts local and global features through the YOLO11 model and uses C3k2 and C2PSA modules to significantly improve the accuracy of target detection and cross-scene adaptability. It also uses a rotating annotation box to unify the annotation format and reduce the noise impact of the background.

[0048] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments:

[0049] like Figure 1 As shown, the method for determining the tightening direction of fuses based on deep learning under industrial vision includes the following steps:

[0050] Step 1: Fuse data acquisition;

[0051] Step 2: Image processing;

[0052] Step 3: Network model construction;

[0053] Step 4: Determine the tightening direction.

[0054] like Figure 2 As shown, step 1 is the industrial data acquisition process:

[0055] Step 101: Deploy multiple sensors (industrial cameras, endoscopes) on the production line to acquire clearly visible images. The cameras must have at least 2 megapixels, adjustable focus, and the ability to rotate to adapt to different shooting positions.

[0056] Step 102: Multiple sensors acquire images at different times and under different lighting conditions, such as morning, noon, and evening. The images are from the same model at different times to enrich image features and acquire clear and visible images.

[0057] like Figure 3 As shown, step 2 is the image processing procedure:

[0058] Step 201: Annotate the acquired image using a rotated bounding box, using a uniform annotation format: class_indexx1 y1 x2 y2 x3 y3 x4 y4, where class_index represents the category of the object to be detected, and (x1,y1), (x2,y2), (x3,y3), and (x4,y4) represent the coordinates of the four vertices of the bounding box, respectively. The annotation box uses a rotated bounding box instead of a rectangular bounding box. This method reduces background noise caused by using rectangular bounding boxes, allows for full selection of fuse features, and improves detection performance.

[0059] Step 202: Normalize, denoise, and align the data, and divide the training set, validation set, and test set into a 6:2:2 ratio;

[0060] Step 203: Store the data in the corresponding path and save it according to the location of the training set, validation set, and test set.

[0061] like Figure 4 As shown, step 3 is the network model building process:

[0062] Step 301: Construct the YOLO11 network, which consists of a backbone network, a neck network, and an OBB detection head.

[0063] The backbone network is used to initially extract image features. The backbone network incorporates the C3k2 module, a variant of the bottleneck module, which allows for custom kernel size. It extracts features by segmenting the feature map and applying smaller 3×3 convolutions, resulting in faster speed and lower computational cost compared to larger convolution kernels. A cross-stage local spatial attention module (C2PSA) is used to enhance spatial attention in the feature map, improving the model's focus on important parts of the image. Spatial pooling features enable the model to more effectively focus on specific regions of interest. The backbone network's workflow is as follows:

[0064] Input an original image with the shape [640, 640, 3], representing the length, width, and number of channels of the image, respectively;

[0065] Two Conv convolutional layers were used to initially extract features, with output channels of 64 and 128 respectively. The resulting features are Backbone P1 and Backbone P2, and the downsampled feature sizes are 1 / 2 and 1 / 4 of the original image.

[0066] A C3k2 module is used and repeated twice, with 256 output channels.

[0067] Using a Conv convolutional layer with 256 output channels, the resulting feature is the backbone network layer feature map BackboneP3. The downsampled feature size is 1 / 8 of the original image.

[0068] A C3k2 layer is repeated twice, with 512 output channels;

[0069] Using a Conv convolutional layer with 512 output channels, the resulting feature is the backbone network layer feature map BackboneP4. The downsampled feature size is 1 / 16 of the original image.

[0070] A C3k2 layer is repeated twice, with 512 output channels and a shortcut jump connection;

[0071] Using a Conv convolutional layer with 1024 output channels, the resulting feature is the backbone network layer feature map Backbone P5. The downsampled feature size is 1 / 32 of the original image.

[0072] A C3k2 layer is repeated twice, with 1024 output channels and a shortcut jump connection;

[0073] Finally, a C2PSA module is used and repeated twice, with an output channel of 1024.

[0074] Step 302: The Neck network uses multiple upsampling layers to upsample the input image features, then fuses these features with features of the same size as the original features to obtain the fused features. Through this fusion method, the model can fully learn both local and global features. The Neck network workflow is as follows:

[0075] (1) An upsampling layer enlarges the original feature map by two times, fills it using the nearest neighbor method, and connects it with the Backbone P4 feature according to the channel;

[0076] (2) A C3k2 layer is repeated twice, with 512 output channels;

[0077] (3) An upsampling layer enlarges the original feature map by two times, fills it using the nearest neighbor method, and connects it with the Backbone P3 feature map of the backbone network layer according to the channel to obtain the Neck feature map of the third neck network, denoted as NeckP3;

[0078] (4) A C3k2 layer is repeated twice, with 256 output channels;

[0079] (5) A Conv convolutional layer with 256 output channels is connected to the Backbone P4 feature according to the channels to obtain the Neck feature map of the fourth neck network, denoted as Neck P4;

[0080] (6) A C3k2 layer is repeated twice, with 512 output channels;

[0081] (7) A Conv convolutional layer with 512 output channels is connected to the Backbone P5 feature according to the channels to obtain the Neck feature map of the fifth neck network, denoted as Neck P5;

[0082] (8) A C3k2 layer is repeated twice, with 1024 output channels;

[0083] Step 303: After image feature extraction and fusion of the backbone network and neck network, three feature maps are obtained: Neck P3, Neck P4, and Neck P5. The fused features are then input into the head network, which uses an OBB (Operated Box Bypass) object detection head. Figure 7 As shown; the detection head outputs the coordinate information of the four vertices of the detection frame, as well as the rotation angle of the detection frame;

[0084] Step 303 includes:

[0085] Features are extracted using three convolutions, and the regression loss and point coordinates are calculated using a regression loss function.

[0086] Features are extracted using two depthwise separable convolutions and one convolution, and the classification loss function is used to calculate the classification loss and the category of the object to be detected.

[0087] Features are extracted using three convolutions, and the rotation angle (angle) is calculated; simultaneously, the formula is used:

[0088] (angle.sigmoid()-0.25)*math.pi normalizes the angle to (0,1) using the sigmoid activation function, then subtracts 0.25 and multiplies it by π, converting the angle from an angle to radians, which is helpful for calculating the angle rotation loss later.

[0089] Step 304: Adjust hyperparameters, such as the number of training rounds, batch size, and learning rate, to improve detection accuracy.

[0090] like Figure 5 As shown, step 4 is the process of determining the tightening direction:

[0091] Step 401: Based on the output coordinate information, construct the corresponding vector to represent the relative positions of the screw and the fuse; with the center points of the two fuses as A and D, and the two endpoints of the fuses as B and C; Step 402: Determine the values ​​of the vectors respectively. and and The result of the cross product of vectors;

[0092] The calculation method is determined to be the vector outer product, which extends the vector in two-dimensional coordinates to three-dimensional space; assuming the coordinates of points A, B, and C are respectively: A(x... A ,y B ),B(x B ,y B ),C(x C ,y C If the vector is... and The outer product is: in Let be the basis vector, and be the unit vector in the z-direction of the three-dimensional coordinate system; let res = [(x B -x A )(y C -y B )-(y B -y A (x) C -x B ]], res represents the coefficient of the vector outer product calculation result. If res>0, it means exist The counterclockwise direction; if res < 0, then it means exist The clockwise direction; if res = 0, then it means and Collinear;

[0093] Step 403: Determine whether the fuse tightening direction meets the requirements based on the calculated result res; when res>0, it indicates... exist The direction is counterclockwise, which satisfies the requirement that the winding direction and the screw tightening direction are clockwise; the detection accuracy rate is obtained statistically.

[0094] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An industrial vision-based deep learning-based fuse tightening direction judgment method, characterized by, The method comprises: Step 1: fuse data acquisition; Step 2: image processing; Step 3: network model building; Step 4: tightening direction judgment.

2. The method of claim 1, wherein, Fuse data acquisition, comprising: Step 101: deploying multiple sensors on the production line for image acquisition; Step 102: multiple sensors respectively take pictures under different light at different time periods, and the images come from different time periods of the same model.

3. The method of claim 1, wherein, The image processing process comprises: Step 201: using a rotated box to label the collected images, using a unified labeling format: class_indexx1y1x2y2x3y3x4y4, wherein class_index represents the category of the object to be detected, (x1, y1), (x2, y2), (x3, y3), (x4, y4) respectively represent the coordinates of the four vertices of the labeling box; wherein the labeling box uses a rotated box; Step 202: normalize, denoise and align the data respectively, and divide them into training set, validation set and test set; Step 203: store the data in the corresponding path, and save them according to the positions of the training set, the validation set and the test set.

4. The method of claim 1, wherein, The network model building process comprises: Step 301: building a YOLO11 network, which includes a backbone network, a neck network and an OBB detection head in turn; Using the backbone network Backbone to preliminarily extract image features; wherein the backbone network Backbone part introduces a C3k2 module, the C3k2 module is a variant of the bottleneck module, which can customize the kernel size, extract the features in the network by dividing the feature map and applying a smaller 3x3 convolution, and set a cross-stage local spatial attention module C2PSA to enhance the spatial attention in the feature map; through spatial pooling features, the model can pay more attention to specific regions of interest; Step 302: the neck network Neck uses multiple up-sampling layers to up-sample the input image features, and then fuses them with the features of the same size as the original features to obtain the fused features; Step 303: after image feature extraction and fusion by the backbone network and the neck network, three feature maps Neck P3, Neck P4 and Neck P5 are obtained; the fused features are input into the head network Head, and the head network uses an OBB rotated box target detection head; the detection head outputs the coordinate information of the four vertices of the detection box and the rotation angle of the detection box; Step 304: adjusting the hyperparameters.

5. The method of claim 4, wherein, The workflow of the backbone network is as follows: Input an original image; Use two Conv convolutional layers to preliminarily extract features, and the output channels are 64 and 128 respectively, and the obtained features are backbone network layer feature maps Backbone P1 and Backbone P2, and the size of the down-sampled features is 1 / 2 and 1 / 4 of the original image; A C3k2 module is repeated twice, and the output channel is 256; Use a Conv convolutional layer, and the output channel is 256, and the obtained feature is a backbone network layer feature map Backbone P3, and the size of the down-sampled feature is 1 / 8 of the original image; One C3k2 layer and repeat twice, output channel is 512; Use one Conv convolution layer, output channel is 1024, get the feature of backbone network layer feature map Backbone P5, the feature size of the original picture after downsampling is 1 / 32; One C3k2 layer and repeat twice, output channel is 1024, and carry shortcut jump connection; Finally, one C2PSA module and repeat twice, output channel is 1024. Neck network workflow as follows: One up-sampling layer, the original feature map is enlarged twice, nearest neighbor method is used for padding, and is connected with backbone network layer Backbone P4 feature according to channel; 6. The method of claim 4, wherein, One C3k2 layer and repeat twice, output channel is 512; One up-sampling layer, the original feature map is enlarged twice, nearest neighbor method is used for padding, and is connected with backbone network layer Backbone P3 feature according to channel, get the third neck network Neck feature map, recorded as Neck P3; One C3k2 layer and repeat twice, output channel is 256; One Conv convolution layer, output channel is 256, and is connected with backbone network Backbone P4 feature according to channel, get the fourth neck network Neck feature map, recorded as Neck P4; One C3k2 layer and repeat twice, output channel is 512; One Conv convolution layer, output channel is 512, and is connected with backbone network Backbone P5 feature according to channel, get the fifth neck network Neck feature map, recorded as Neck P5; One C3k2 layer and repeat twice, output channel is 1024. Step 303 comprises: Use 3 convolution to extract features, and use regression loss function to calculate regression loss and point coordinates; 7. The method of claim 4, wherein, Use 2 depth separable convolution and 1 convolution to extract features, and use classification loss function to calculate classification loss and the category of the object to be detected; Use 3 convolution to extract features, calculate the rotation angle angle; At the same time, use the formula: (angle.sigmoid()-0.25)*math.pi to normalize angle using the activation function sigmoid to (0, 1), then subtract 0.25 and multiply by pi, convert angle from angle to radian. Tightening direction judgment comprises: ​ 8. The method of claim 1, wherein, ​ Step 401: according to the output coordinate information, construct the corresponding vector, for indicating the relative position of the screw and the fuse; take the center point of two fuses as A, D, and the two end points of the fuse as B, C; Step 402: respectively judge the results of the vector cross product of and and ​ The calculation method is vector cross product, the vector on two-dimensional coordinate is extended to three-dimensional space; assuming the coordinates of points A, B, C are respectively: A(x A ,y B ), B(x B ,y B ), C(x C ,y C ), the cross product of vector and is: Wherein is the base vector, which is the unit vector in the z direction of three-dimensional coordinate system; let res = [(x B -x A )(y C -y B )-(y B -y A )(x C -x B )], res represents the coefficient of the calculation result of vector cross product, if res>0, it indicates that is in the counterclockwise direction of ; if res<0, it indicates that is in the clockwise direction of ; if res=0, it indicates that and are collinear; Step 403: judging whether the fuse tightening direction meets the requirement according to the calculation result res; when res>0, it means that In the counterclockwise direction, at this time, the requirement that the winding direction and the tightening direction of the screw present clockwise is met; the detection accuracy rate is obtained by statistics.

Images