A method and apparatus for bolt inspection

Abstract

The application discloses a neural network, a method and a device for bolt detection, and relates to the technical field of image processing.The neural network comprises an area of interest pooling layer, a position regression module, a confidence calculation module, a fusion module, a classification prediction module, an anchor frame generation layer, a first convolution layer connection, a second convolution layer, a classification feature transformation layer, an activation function, a classification feature adjustment layer, a third convolution layer and a proposal frame, and further comprises a RESNET50-C network, a feature pyramid, a distance and focal loss function, a scaling convolution layer, a rotated intersection over union calculation module and a soft non-maximum suppression module.The method is based on the neural network, and a picture containing a bolt is input into the neural network to obtain a detection result.The device is a computer readable storage medium and stores a computer program, and the computer program is executed to realize the corresponding steps.The device has high correct detection rate for an inclined bolt through the scaling convolution layer, the rotated intersection over union calculation module and the like.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a neural network, method, and apparatus for bolt detection. Background Technology

[0002] Bolts are an important component in many industrial construction projects. Although their manufacturing process is not complex, their installation environments are often concealed and confined spaces, making the detection and identification of bolt positions quite difficult. Correct bolt installation positions directly affect product quality, performance, and safety; therefore, research on bolts is of great significance for improving the level and competitiveness of the manufacturing industry.

[0003] Traditional inspection methods mainly include manual inspection and image inspection techniques. However, these techniques generally suffer from several problems: slow inspection speed and long inspection cycle, resulting in low inspection efficiency; high inspection difficulty, making it impossible to effectively inspect areas inaccessible to humans; and cumbersome inspection processes, making automated identification difficult. Furthermore, with changing market demands and rapid industrial development, traditional inspection techniques can no longer meet the inspection requirements of bolt parts. Therefore, how to automate the inspection and identification of bolt parts has become a problem worthy of in-depth research in current engineering construction.

[0004] With the rapid development of deep learning and artificial intelligence, automatic identification of bolt parts has become a viable technological application. This technology has broad prospects and far-reaching impact in the manufacturing and engineering fields, bringing new solutions and opportunities for bolt installation position detection.

[0005] The automatic identification of bolt parts offers numerous advantages. First, it eliminates the subjectivity and tediousness of manual inspection, improving accuracy and consistency. Second, automatic identification technology enables real-time detection and big data processing, adapting to the demands of high-speed production lines and thus increasing production efficiency and capacity. Furthermore, automatic identification systems can continuously learn and optimize; over time, their performance will continuously improve, providing strong support for the sustainable development of the manufacturing industry.

[0006] Although many object detection algorithms have achieved remarkable results in various fields, there are still some challenges in the identification of bolt parts, and a widely accepted and truly feasible detection method has not yet emerged.

[0007] Existing technical issues and considerations:

[0008] How to solve the technical problem of high error rate in detecting tilted bolt targets. Summary of the Invention

[0009] This invention provides a neural network, method, and apparatus for bolt detection, solving the technical problem of high error rate in detecting inclined bolt targets.

[0010] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0011] A neural network for bolt detection includes a region-of-interest (ROI) pooling layer, a location regression module, a confidence calculation module, and a fusion module connected in sequence; a classification prediction module connected between the ROI pooling layer and the confidence calculation module; an anchor box generation layer, a first convolutional layer, a second convolutional layer, a classification feature transformation layer, an activation function and a classification feature adjustment layer, a third convolutional layer, and a proposal box connected in sequence; and a RESNET50-C network, a feature pyramid, a distance and focus loss function, a scaling convolutional layer, a rotation-intersection-union (ROI) calculation module, and a soft non-maximum suppression (NNMS) module. The RESNET50-C network is connected to the ROI pooling layer via the feature pyramid, and the feature pyramid is connected to the anchor box generation layer. The classification feature adjustment layer, scaling convolutional layer, rotation-intersection-union (ROI) calculation module, NNMS module, proposal box, distance-intersection-union (ROI) loss function, and ROI pooling layer are connected in sequence. The first convolutional layer is connected to the scaling convolutional layer via the third convolutional layer.

[0012] A further technical solution is as follows: Based on the improvement of the ResNet50 network, a ResNet50-C network is obtained. The ResNet50 network consists of five stages. In the zero stage, the 7×7 convolutional layer with a stride of 2 and 64 channels is decomposed into three 3×3 convolutional layers with strides of [32, 32, 64] and channels of [1, 2, 1]. In the fourth stage, in the downsampling module, the strides of the first two convolutional layers in the first path A are swapped, a 2×2 average pooling layer with a stride of 2 is added in the second path B, and the stride of the 1×1 convolutional layer is changed to 1. The outputs of the first path A and the second path B are assimilated, and the outputs of the two paths are added together as the overall output of the downsampling module.

[0013] A further technical solution is as follows: In the anchor frame generation layer, the angle value [-10°, -25°, -40°, -55°, -70°, -90°] is added, the scale value is changed to [4, 8, 16], and the aspect ratio is changed to [1:3, 1:5, 1:7, 1:9].

[0014] A further technical solution is that the first convolutional layer is a 3×3 convolutional layer.

[0015] A further technical solution is that the second convolutional layer is a 1×1 convolutional layer.

[0016] A further technical solution is that the third convolutional layer is a 1×1 convolutional layer.

[0017] A method for bolt detection, based on the aforementioned neural network for bolt detection, includes a detection step, which involves inputting an image containing bolts into the neural network to obtain a detection result.

[0018] A further technical solution is that the test results include the confidence level of the bolts in the testing process.

[0019] An apparatus for bolt detection includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the corresponding steps described above.

[0020] An apparatus for bolt inspection includes a computer-readable storage medium storing a computer program that, when executed by a processor, performs the aforementioned corresponding steps.

[0021] The beneficial effects of adopting the above technical solution are as follows:

[0022] First, a neural network for bolt detection includes, in sequence, a region-of-interest (ROI) pooling layer, a location regression module, a confidence calculation module, and a fusion module; a classification prediction module connected between the RIO pooling layer and the confidence calculation module; an anchor box generation layer, a first convolutional layer, a second convolutional layer, a classification feature transformation layer, an activation function and a classification feature adjustment layer, a third convolutional layer, and a proposal box; it also includes a RESNET50-C network, a feature pyramid, a distance and focus loss function, a scaling convolutional layer, a rotation-intersection-union (OUI) calculation module, and a soft non-maximum suppression (NNMS) module. The RESNET50-C network is connected to the RIO pooling layer via the feature pyramid, and the feature pyramid is connected to the anchor box generation layer. The classification feature adjustment layer, scaling convolutional layer, OUI calculation module, NNMS module, proposal box, distance and focus loss function, and RIO pooling layer are sequentially connected. The first convolutional layer is connected to the scaling convolutional layer via the third convolutional layer. This technical solution, through the scaling convolutional layer and the OUI calculation module, achieves high accuracy in detecting tilted bolt targets.

[0023] Second, a method for bolt detection, based on the aforementioned neural network for bolt detection, includes a detection step. The detection step involves inputting an image containing the bolt into the neural network to obtain a detection result. This technical solution, through scaling convolutional layers and rotational intersection-union calculation modules, achieves high accuracy in detecting tilted bolt targets.

[0024] Third, an apparatus for bolt detection includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the corresponding steps described above. This technical solution achieves high accuracy in detecting tilted bolt targets through scaling convolutional layers and rotational intersection-over-union (OI) calculation modules.

[0025] Fourth, an apparatus for bolt detection includes a computer-readable storage medium storing a computer program that, when executed by a processor, performs the aforementioned corresponding steps. This technical solution, through scaling convolutional layers and rotational intersection-over-union (OI) calculation modules, achieves high accuracy in detecting tilted bolt targets.

[0026] See the detailed implementation section for further description. Attached Figure Description

[0027] Figure 1 This is a block diagram of the improved Faster R-CNN network.

[0028] Figure 2 This is a flowchart of the improved neural network method;

[0029] Figure 3 This is the flowchart for R&D Example 1;

[0030] Figure 4 This is the flowchart for R&D Example 2;

[0031] Figure 5 This is the flowchart for R&D Example 3;

[0032] Figure 6 This is a block diagram of the existing Faster R-CNN network.

[0033] Figure 7 This is a block diagram of the ResNet50 network.

[0034] Figure 8 This is a block diagram of the ResNet50-C network.

[0035] Figure 9 This is a block diagram of the FPN principle;

[0036] Figure 10a This is a data diagram showing the aspect ratio settings of the anchor frame;

[0037] Figure 10b This is a data diagram showing the anchor frame dimensions.

[0038] Figure 10c This is a data chart showing the anchor frame angle settings;

[0039] Figure 11 This is the data flow graph of the scaling convolutional layer;

[0040] Figure 12 This is a data graph calculated using the traditional intersection-union ratio (IUU) method;

[0041] Figure 13a This is a data graph for calculating the rotational intersection-union ratio;

[0042] Figure 13b This is a data graph for calculating the rotational intersection-union ratio;

[0043] Figure 14 It is a dataset of bolts;

[0044] Figure 15a This is the original image;

[0045] Figure 15b This is the image after brightness adjustment;

[0046] Figure 15c This is the image after Gaussian noise processing;

[0047] Figure 15d It is an image that has been randomly rotated;

[0048] Figure 16a These are images with traditional annotations;

[0049] Figure 16b This is the image after rotation and annotation;

[0050] Figure 16c It is a screenshot of a traditional annotation file;

[0051] Figure 16d It is a screenshot of the rotated annotation file;

[0052] Figure 17 This is the mAP curve of the improved Faster R-CNN;

[0053] Figure 18 It is a graph showing the ablation comparison of the backbone network;

[0054] Figure 19 It is a graph comparing the ablation of the loss function;

[0055] Figure 20a This is a screenshot showing the first test result;

[0056] Figure 20b This is a screenshot showing the second test result;

[0057] Figure 20c This is a screenshot showing the third test result;

[0058] Figure 20d This is a screenshot showing the results of the fourth test. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this application or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0060] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0061] Example 1:

[0062] like Figure 1 As shown, this invention discloses a neural network for bolt detection, comprising a RESNET50-C network, a region proposal network, a feature pyramid, a region of interest pooling layer, distance and focus loss functions, a location regression module, a classification prediction module, a confidence calculation module, and a fusion module. The region proposal network includes an anchor box generation layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a classification feature transformation layer, an activation function, a classification feature adjustment layer, a scaling convolutional layer, a rotation-intersection-union (OI) calculation module, a soft nonmaximum suppression module, and proposal boxes. The RESNET50-C network... The C network, feature pyramid, region of interest pooling layer, location regression module, confidence calculation module, and fusion module are connected sequentially. The region of interest pooling layer is connected to the confidence calculation module via the classification prediction module. The feature pyramid, anchor box generation layer, first convolutional layer, second convolutional layer, classification feature transformation layer, activation function, classification feature adjustment layer, scaling convolutional layer, rotation intersection-over-union calculation module, soft nonmaximum suppression module, proposal box, distance and focus loss function, and region of interest pooling layer are connected sequentially. The first convolutional layer is connected to the scaling convolutional layer via the third convolutional layer.

[0063] The first convolutional layer is a 3×3 convolutional layer, the second convolutional layer is a 1×1 convolutional layer, and the third convolutional layer is a 1×1 convolutional layer.

[0064] The ResNet50-C network is obtained by improving the ResNet50 network. The ResNet50 network consists of five stages. In the zero stage, the 7×7 convolutional layer with a stride of 2 and 64 channels is decomposed into three 3×3 convolutional layers with strides of [32, 32, 64] and channels of [1, 2, 1]. In the fourth stage, in the downsampling module, the strides of the first two convolutional layers in the first path A are swapped, a 2×2 average pooling layer with a stride of 2 is added in the second path B, and the stride of the 1×1 convolutional layer is changed to 1. The outputs of the first path A and the second path B are assimilated, and the outputs of the two paths are added as the overall output of the downsampling module.

[0065] The anchor frame generation layer has added angle values ​​[-10°, -25°, -40°, -55°, -70°, -90°], changed the scale value to [4, 8, 16], and changed the aspect ratio to [1:3, 1:5, 1:7, 1:9].

[0066] Example 2:

[0067] This invention discloses a method for bolt detection, based on the neural network for bolt detection in Embodiment 1, including a detection step, which includes inputting an image containing bolts into the neural network to obtain a detection result containing bolt confidence.

[0068] Example 3:

[0069] The present invention discloses a device for bolt detection, which is an electronic device, and the electronic device includes the neural network of Embodiment 1.

[0070] Example 4:

[0071] The present invention discloses a device for bolt detection, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory and the processor form an electronic terminal, and the processor executes the computer program to implement the steps of Embodiment 2.

[0072] Example 5:

[0073] The present invention discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in Embodiment 2.

[0074] Compared to the above embodiments, the program module can also be a hardware module made using existing logic operation technology to implement the corresponding logic operation steps, communication steps and control steps, thereby realizing the above-mentioned corresponding steps. The logic operation unit is existing technology and will not be described in detail here.

[0075] Research and development process:

[0076] Key innovations: First, the ResNet50 network was improved to enhance its sensitivity to feature information. Multi-directional detection was applied to bolt part detection, and a custom dataset of bolt parts was constructed, filling a gap in multi-directional target detection for bolt parts. A scaling convolutional layer was introduced for the first time; after the feature map passes through this layer, the final bounding box fits the detection target more closely. To adapt to the detection of multi-directional bolt parts, the region proposal network was improved, enabling multi-directional detection of bolt parts.

[0077] 1. The most fundamental technical problem to be solved

[0078] This invention addresses the problem of poor manual inspection and identification capabilities in bolt detection, resolves false positives and negatives caused by excessive tilt in bolt component target detection, solves the problem of discontinuous angles in rotating target detection, and enhances the detection accuracy for small and tilted targets.

[0079] 2. Core Technology Solution

[0080] like Figure 1 The diagram shown is a schematic of the improved Faster R-CNN network.

[0081] The improved Faster R-CNN network includes a RESNET50-C network, a region proposal network, a feature pyramid, region of interest pooling layers, distance and focus loss functions, a location regression module, a classification prediction module, a confidence calculation module, and a fusion module. The region proposal network includes an anchor box generation layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a classification feature transformation layer, an activation function, a classification feature adjustment layer, a scaling convolutional layer, a rotation-intersection-union (OI) calculation module, a soft nonmaximum suppression module, and proposal boxes. The first convolutional layer is a 3×3 convolutional layer, the second convolutional layer is a 1×1 convolutional layer, and the third convolutional layer is a 1×1 convolutional layer. The ET50-C network is connected to the feature pyramid and the region of interest pooling layer. The feature pyramid, anchor box generation layer, first convolutional layer, second convolutional layer, classification feature transformation layer, activation function, classification feature adjustment layer, scaling convolutional layer, rotation intersection-over-union (OCI) calculation module, soft non-maximum suppression module, proposal box, distance and focus loss function and region of interest pooling layer are connected in sequence. The first convolutional layer, third convolutional layer and scaling convolutional layer are connected in sequence. The region of interest pooling layer, location regression module, confidence calculation module and fusion module are connected in sequence. The region of interest pooling layer is connected to the classification prediction module and the confidence calculation module.

[0082] like Figure 2 The diagram shown is a flowchart of the improved neural network method.

[0083] This invention improves upon Faster R-CNN, which consists of a VGG16 backbone, a region proposal network, a region of interest pooling layer, a location regression module, and a classification prediction module. This invention replaces the original VGG16 network with a ResNet50 backbone and modifies it. The ResNet50 network consists of five stages, with the main adjustments made in stages 0 and 4. In stage 0, the 7×7 convolutional layer with a stride of 2 and 64 channels is decomposed into three 3×3 convolutional layers with strides of [32, 32, 64] and channels of [1, 2, 1], enhancing the network's sensitivity to information. In stage 4, paths A and B in the downsampling module are improved. The strides of the first two convolutional layers in path A are swapped, allowing the first convolutional layer in path A to obtain more feature information and reduce the loss of feature information. A 2×2 average pooling layer with a stride of 2 is added to path B, and the stride of the 1×1 convolutional layer is changed to 1. The outputs of paths A and B are assimilated. Finally, the outputs of the two paths are added together as the overall output of the downsampling block. The adjusted ResNet50 network is called the ResNet50-C network. Then, the Region Proposal Network (RPN) was improved. In the anchor frame generation layer of the RPN, the scale values ​​and aspect ratio parameters used in traditional anchor frame generation are insufficient for rotation detection. The improved RPN adds angle values ​​[-10°, -25°, -40°, -55°, -70°, -90°] when generating anchor frames, and changes the scale values ​​to [4, 8, 16] and the aspect ratios to [1:3, 1:5, 1:7, 1:9] based on the shape of the bolt. The improved RPN also adds scaling convolutional layers, which can adjust the generated anchor frames to make the final generated detection boxes fit the detection target better. Then, the Intersection over Union (IoU) ratio is calculated. When calculating the IoU of adjacent boxes in rotation detection, polygonal overlapping areas will appear. Obviously, the horizontal IoU calculation cannot fully meet the requirements of rotation detection. Therefore, this invention uses rotational IoU calculation. By dividing the polygonal overlapping parts into multiple triangles, the area of ​​the triangles is calculated and summed to complete the rotational IoU calculation. Furthermore, this invention replaces the nonmaximum suppression module with a soft nonmaximum suppression module, employing a more flexible and refined approach to handle overlapping bounding boxes. Finally, the total loss function was replaced with a Gaussian Wasserstein distance and focus loss function, which is applied via backpropagation to the region proposal network, location regression module, and classification prediction module, thus mitigating the problem of angular discontinuities in rotation detection.

[0084] Key technical features: Stage adjustments to the ResNet50 network, improved region proposal network, and scaling convolutional layers.

[0085] 3. Beneficial technical effects

[0086] This invention provides an improved multi-directional bolt detection method for Faster R-CNN. It modifies the sensitivity of the backbone network enhancement model to feature information and utilizes a multi-scale fusion module to strengthen the model's detection of small targets. In the proposal generation stage, the region proposal network is improved to obtain the optimal proposal box. To address the issue of angular discontinuities in multi-directional detection, Gaussian Wasserstein distance and focus loss are used as loss functions instead of the traditional Smooth L1 loss function. Experimental results on bolt recognition show that the improved Faster R-CNN model achieves an mAP value of 87.4%, which is 7.6% higher than that of the horizontal detection-based Faster R-CNN. Compared with other multi-directional detection models, including R2CNN, RRPN, R2PN, ROI-trans, RSDET, RTDDe, etc., the model proposed in this paper has higher accuracy and stronger robustness. Specifically, the AP of the proposed model is 13.2% higher than R2CNN, 8.7% higher than RRPN, 7.9% higher than R2PN, 2.9% higher than RTMDet, 0.8% higher than ROI-trans, and 0.2% higher than RSDET.

[0087] 4. Inventive concept

[0088] The inventive concept of this application is based on existing technology and focuses on solving the problem of automatic identification and positioning of bolt parts, thereby making up for the shortcomings of existing technology in this field. In the current technological context, although various target detection algorithms have been successfully applied in different fields, there is still a lack of a truly feasible method for the specific challenge of bolt parts.

[0089] To overcome this problem, this invention first proposes an improved backbone network, ResNet50-C, designed to enhance the network's ability to perceive the feature information of bolt parts. Simultaneously, a Feature Pyramid Network (FPN) is introduced, which, through multi-scale information fusion, makes the network more adaptable to the diverse sizes and shapes of bolt parts. An improved region proposal network is designed to flexibly adjust the angle and scale of the anchor frame, further improving the detection accuracy and stability of bolt parts. To better handle the special characteristics of bolt parts, scaling convolutional layers are also designed to optimize the anchor frame, thereby further improving the accuracy of target localization.

[0090] In designing the loss function, this invention employs the Gaussian Wasserstein distance and focus loss functions as the total loss function, effectively alleviating the problem of discontinuous angle loss in rotating target detection and improving the overall recognition effect and stability. Through this innovative loss function design, this invention can better adapt to the characteristics of bolted parts, thereby enhancing the system's reliability.

[0091] R&D Example 1:

[0092] like Figure 3 As shown, the inventors proposed a multi-directional bolt detection method based on an improved Faster RCNN.

[0093] like Figure 6 As shown, the Faster R-CNN network includes a VGG16 network, a region proposal network, a region of interest (ROI) pooling layer, a Smooth L1 Loss function, a location regression module, a classification prediction module, a confidence calculation module, and a fusion module. The ROI network includes an anchor box generation layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a classification feature transformation layer, an activation function, a classification feature adjustment layer, an intersection-over-union (IoU) calculation module, a non-maximum suppression (NMS) module, and proposal boxes. The first convolutional layer is a 3×3 convolutional layer, the second convolutional layer is a 1×1 convolutional layer, and the third convolutional layer is a 1×1 convolutional layer. The VGG16 network is connected to the ROI pooling layer. The VGG16 network, anchor box generation layer, first convolutional layer, second convolutional layer, classification feature transformation layer, activation function, classification feature adjustment layer, IoU calculation module, NMS module, proposal boxes, and ROI pooling layer are connected sequentially. The first convolutional layer, third convolutional layer, and IoU calculation module are connected sequentially. The proposal boxes are then processed by a Smooth L1 Loss function. The loss function is connected to the region of interest pooling layer. The region of interest pooling layer, the location regression module, the confidence calculation module, and the fusion module are connected in sequence. The region of interest pooling layer is connected to the confidence calculation module through the classification prediction module.

[0094] like Figure 6 As shown, Faster R-CNN first extracts features from the input image using VGG16, then inputs the feature maps into the RPN to generate anchor boxes. Non-maximum suppression is then used to select proposal boxes. The proposal boxes and the feature maps output from the backbone network are then fed into the region of interest (ROI) pooling layer to complete bounding box regression and classification. Finally, confidence scores are calculated and fused with the original image for output. The Smooth L1 Loss function is applied through backpropagation to the region proposal network, the location regression module, and the classification prediction module.

[0095] The inventors replaced the original VGG16 network with a ResNet50-C network as the backbone network for training the model.

[0096] like Figure 7 and Figure 8 As shown, the ResNet50-C network is an improvement upon the ResNet50 network. The ResNet50 network consists of five stages, and the adjustments to the ResNet50-C network are mainly reflected in stages 0 and 4. In stage 0, the 7×7 convolutional layer with a stride of 2 and 64 channels is decomposed into three 3×3 convolutional layers with strides of [32, 32, 64] and channels of [1, 2, 1], enhancing the network's sensitivity to information. In stage 4, paths A and B in the downsampling module are improved. The strides of the first two convolutional layers in path A are swapped, allowing the first convolutional layer of path A to obtain more feature information and reducing feature loss. A 2×2 average pooling layer with a stride of 2 is added to path B, and the stride of the 1×1 convolutional layer is changed to 1, assimilating the outputs of paths A and B. Finally, the outputs of the two paths are added together as the overall output of the downsampling block.

[0097] like Figure 9 As shown, the image features extracted by the network are combined with the feature pyramid network to output five feature maps at different scales. The FPN convolves the feature map of the top layer in the bottom-up path, and the resulting feature map is used as the starting point of the top-down path. The bottom-up structure can be combined with the forward propagation process of the ResNest50-C network, and the features of the last layer of the ResNest50-C network in each stage are used as the base feature map {N2,N3,N4,N5}. In the top-down process, the network upsamples the high-level features to obtain features with the same spatial resolution as the low-level features, and then uses lateral connections to fuse the upsampled features with the base features {N2,N3,N4,N5}, and finally generates the FPN feature map {P2,P3,P4,P5}. The relationship between {P2,P3,P4,P5} and {N2,N3,N4,N5} is shown in Equation (1).

[0098]

[0099] In equation (1), P i To generate the final retrieval value for the FPN feature map, N i For the retrieval value of the basic feature, Conv i×i This indicates an i×i convolutional layer, and upsample represents upsampling.

[0100] Simultaneously, max pooling is performed on P5 to obtain feature P6. The specific relationship between P5 and P6 is shown in equation (2).

[0101] P6 = maxpooling(P5)(2)

[0102] In equation (2), maxpooling represents max pooling.

[0103] Because the image resolution varies across multiple feature layers, when performing multi-scale feature mapping, the feature layer of the most suitable size is selected to extract feature blocks for region prediction boxes at different scales. The level P of the feature pyramid... k The calculation is determined by calculation, and the calculation formula is shown in equation (3).

[0104]

[0105] In equation (3), k0 is the baseline value, w is the width of the generated prediction box, h is the length of the generated prediction box, and 640 is the size of the input image. The feature pyramid is used to compare and analyze the feature information of each layer, and then a feature map containing rich feature information is obtained, thereby gradually improving the network's ability to detect bolt parts of different scales.

[0106] The output feature map is then fed into the improved region proposal network.

[0107] like Figure 10a The diagram shown illustrates the aspect ratio setting of the anchor frame.

[0108] like Figure 10b The diagram shown is a schematic of the anchor frame dimensions.

[0109] like Figure 10c The diagram shown illustrates the setting of the anchor frame angle value.

[0110] In the anchor box generation layer of the Region Proposal Network (RPN), compared to the original RPN, the improved RPN adds angle values ​​[-10°, -25°, -40°, -55°, -70°, -90°] when sampling anchor boxes, and sets special scale values ​​[4, 8, 16] and aspect ratios [1:3, 1:5, 1:7, 1:9] according to the characteristics of bolt parts. The improved RPN first generates 72 rotated anchor boxes (R-anchors) at each anchor point on the feature map with different scales, aspect ratios, and angle values. The corresponding regression layer generates 360 output values, and the classification layer generates 144 output values. For a feature map of size H×W, it can generate H×W×72 rotated anchor boxes.

[0111] The improved region proposal network adds a scaling convolutional layer.

[0112] like Figure 11As shown, the scaling convolutional layer first passes the generated feature map through a 3×3 convolutional layer, and then performs weight compensation and angle assignment. The weight compensation can be used to adjust the anchor box size appropriately based on the feature map content and the pre-labeled ground truth boxes. The angle assignment can be used to fine-tune the rotation angle of the anchor box based on the pre-labeled ground truth boxes, thus optimizing the rotation of the anchor box. This makes the final detection box fit the rotation target better. This can be expressed by formula (4).

[0113] y(F0)=Conv 3×3 [x(F n )]·x(F0+F q +F θ (4)

[0114] In equation (4), y(F0) represents the feature after scaling the convolutional layer, and Conv 3×3 It is a 3×3 convolutional layer, x(F n F is the feature at the nth position in the feature map, F0 is the initial position coordinate, and F q Weight compensation value, F θ To assign an angle value to the shape of the anchor frame corresponding to this position.

[0115] After adjusting the anchor boxes, the improved rotation region proposal network determines positive and negative samples based on the overlap between each anchor box and the actual rotated bounding box, i.e., the Intersection over Union (IoU). Then, it assigns a corresponding rotated bounding box regression target to each positive sample anchor box.

[0116] Positive samples are identified by the Intersection over Union (IoU) ratio (IoU) above a certain threshold. These candidate boxes are labeled as positive samples, indicating good overlap with the ground truth bounding boxes and are likely regions containing the target. Negative samples are identified by the IoU ratio (IoU) below a certain threshold. These candidate boxes have little or no overlap with the ground truth bounding boxes and therefore do not contain the target. The purpose of using IoU to determine positive and negative samples is to help the object detection algorithm learn more effectively. By labeling positive and negative samples, the network can focus more on regions containing the target, thereby improving the accuracy and efficiency of the detection algorithm. Simultaneously, this also helps reduce computational cost, as only a small portion of the candidate boxes are used for training, while other regions unrelated to the real target are excluded.

[0117] However, the traditional cross-union ratio is calculated based on a rectangular frame.

[0118] like Figure 12 As shown, X and Y are two adjacent bounding boxes. In traditional object detection, since adjacent bounding boxes are all horizontal rectangles, it is only necessary to calculate the ratio of the area of ​​the intersection region to the area of ​​the union region of the two bounding boxes.

[0119] The formula for calculating the intersection-to-union ratio is shown in equation (5).

[0120]

[0121] In equation (5), X and Y represent two adjacent bounding boxes, area(A∩B) represents the area of ​​the overlap between the two bounding boxes, and area(A∪B) represents the total area covered by the two bounding boxes.

[0122] However, when calculating the intersection-union ratio (CURBR) for multi-directional object detection, the rotated bounding boxes contain angular information, while traditional CURBR methods only consider the overlap between rectangles in the horizontal and vertical directions, neglecting the impact of rotation angles. With large rotation angles, using traditional CURBR methods may increase errors. Therefore, this invention proposes a rotational CURBR calculation method based on multi-directional object detection.

[0123] like Figure 13a and Figure 13b As shown, the rotational intersection-union ratio (CIU) calculation method demonstrates two cases of bounding boxes when calculating the CIU in rotational detection. Given two partially overlapping rotated bounding boxes S... ABCD and S EFGH .

[0124] like Figure 13a As shown, the polygon of the overlapping part is defined as... polygonal region Let I be the vertex of ILKJ. Connecting all other vertices with I as the computational vertex divides the polygonal region into multiple triangles, i.e., △ ILK and △ IKJ Then calculate the polygon The area is calculated using the formula shown in equation (6).

[0125]

[0126] In equation (6), Let Area(△) be the area of ​​the polygon. ILK +△ IKJ ) is △ ILK and △ IKJ The area and sum of.

[0127] Converted to a general formula:

[0128]

[0129] In equation (7), Let n be the area of ​​the polygon, n be the number of triangles, and i be the index of the triangle. Area(△) iLet be the area of ​​the i-th triangle. Combining equations (6) and (7), we can obtain the formula (8) for calculating the rotation intersection-union ratio.

[0130]

[0131] Right now:

[0132] In equation (8), Let Area(S) be the area of ​​the polygon. all ) represents the sum of the areas of all rectangles.

[0133] This invention uses Soft-NMS, or Soft Non-Maximum Suppression algorithm, to filter multiple overlapping candidate boxes near the target based on the intersection-union ratio (IU) to find the most likely target box, thus eliminating redundant detection results. Compared to traditional NMS, it is more flexible and precise in handling overlapping targets. Its formula is as follows:

[0134]

[0135] In equation (9), s i This represents the confidence level of the i-th prediction box; The prediction box with the highest confidence; b i Represents the i-th pair There are overlapping prediction boxes; The cross-union ratio between predicted bounding boxes; N t The set confidence threshold for the prediction box.

[0136] The rotating rectangle of this invention is represented by five parameters (cx, cy, w, h, θ), and its regression equation is as follows:

[0137]

[0138] In equation (10), cx, cy, w, h, and θ represent the center coordinates, width, height, and angle of the frame, respectively. The variables θ and θ' are... a θ *These are used for ground truth boxes, anchor boxes, and predicted boxes, respectively, and also for cx, cy, w, and h. Since the traditional Smooth L1 loss function only considers horizontal detection, it suffers from angular discontinuities when used for multi-directional detection. In contrast, the Gaussian Wasserstein distance takes into account both the model's prediction and target distributions. By calculating the Gaussian Wasserstein distance between these two distributions, their degree of difference can be measured. The Gaussian Wasserstein distance maps these two probability distributions to a high-dimensional space and calculates the minimum transmission cost between them, thus considering differences in overall structure and shape. By minimizing the Gaussian Wasserstein distance, we can encourage the model to generate a smoother prediction distribution, especially at boundaries, thereby mitigating the angular discontinuity problem.

[0139] The problem of discontinuous loss is generally caused by a sharp jump in angle loss when the angle value reaches the boundary of the specified angle range.

[0140] This invention uses Gaussian Wasserstein distance and focus loss function to enhance the model's learning ability and attention to key samples. The improved model can learn and predict more accurately when dealing with more diverse regression forms. Specifically, as shown in Equation (11).

[0141]

[0142] In equation (11), N represents the number of anchor frames, and obj n It is a binary value, obj n =1 indicates the foreground, obj n =0 indicates background; background is not subject to regression processing. n Let gt represent the nth predicted bounding box. n This is the true value of the nth target. t n p represents the label of the nth object. n Let λn represent the distribution of the probability of the nth objective. λ1 and λ2 are parameters controlling the trade-offs; λ1 defaults to 2, and λ2 defaults to 1. Classification loss Ln cls Set as focus loss.

[0143] R&D Example 2:

[0144] like Figure 4 As shown, the inventors proposed a multi-directional bolt detection method based on an improved Faster RCNN. Images of bolts in construction sites and conventional environments are acquired using both manual labor and image acquisition equipment.

[0145] like Figure 14 As shown, the constructed bolt dataset mainly consists of bolts in construction environments and normal environments.

[0146] The collected bolt part images were cropped and augmented, with the original images cropped to 640×640 pixels. Then, the images were rotated and annotated to construct a bolt part image dataset. Image processing techniques, including random rotation, random brightness adjustment, and Gaussian noise, were used to augment the dataset, increasing its diversity, enhancing bolt target information, and improving detection performance. The dataset was randomly split into training, testing, and validation sets, with 70% allocated to training and 15% to validation.

[0147] like Figure 15a The image shown is an original image from the dataset.

[0148] like Figure 15b The image shown is the result after increasing the brightness of the original image.

[0149] like Figure 15c The image shown is the result after adding Gaussian noise to the original image.

[0150] like Figure 15d As shown, this is the result of randomly rotating the original image.

[0151] This invention uses `roLabelIng` to annotate the dataset. Unlike traditional horizontal annotation, the annotation boxes in this invention are all rotated, which allows for better recognition and angle localization of bolt images during training. Bolt parts are represented using the "bolt" tag, and a corresponding XML file is generated after annotation.

[0152] like Figure 16a The image shown is a schematic diagram of a traditional labelImg horizontal box annotation.

[0153] like Figure 16b The diagram shown is a schematic of the rolabelImg rotating frame labeling used in this invention.

[0154] like Figure 16c As shown, this is the content of the XML file corresponding to the traditional labelImg annotation. The horizontal annotation file includes the four coordinate values ​​of the horizontal bounding box.

[0155] like Figure 16d As shown, this is the content of the XML file corresponding to the rolabelImg annotation used in this invention. The rotation annotation file contains information about the length, height, rotation center, and rotation angle of the rotation frame.

[0156] The GPU used for training in this invention is a Tesla V100-SXM2 with 32GB of video memory. The model is based on the PyTorch 1.7.0 deep learning framework, and the model construction and network training were completed using CUDA 10.1 and CUDNN 7.6.3 in GPU mode. The training parameters are shown in Table 1.

[0157] Table 1: Experimental Parameter Settings

[0158]

[0159] In this context, epochs represents the number of iterations, indicating the total number of times the entire training dataset is used for training. Increasing the number of iterations allows the model to learn data features better, but too many iterations can lead to overfitting. Batch Size represents the batch size, i.e., the number of samples used for training in each iteration. A larger batch size can improve training efficiency but may consume more memory. A smaller batch size may lead to training instability. Learning Rate controls the step size of each parameter update. A larger learning rate can speed up training but may lead to unstable training. A smaller learning rate can improve stability but may slow down training. Mosaic is a data augmentation technique that blends multiple images into a single large image, helping the model learn targets at different scales and locations. It can improve the model's generalization ability and robustness. Weight Decay is a regularization technique that prevents the model from overfitting by adding the square of the weights as a penalty term to the loss function. It helps control the model's complexity. Img Size represents the image size used as input to the model during training. A larger image size may improve the model's accuracy but will increase computational cost and memory usage. Smaller image sizes may reduce accuracy, but training is faster.

[0160] The evaluation metrics of this invention are Average Precision (AP) and mean Average Precision (mAP), which comprehensively consider both precision and recall. AP reflects the accuracy of the algorithm; a higher value indicates better algorithm performance. The accuracy is calculated as shown in equation (12).

[0161]

[0162] In equation (12), TP represents the number of predicted boxes with an intersection-union ratio (IU) greater than a certain threshold, and FP represents the number of predicted boxes with an IU less than a certain threshold or the number of redundant predicted boxes. The recall rate is calculated as shown in equation (13).

[0163]

[0164] In equation (13), FN represents the number of undetected ground truth boxes.

[0165] Plot a precision-recall (PR) curve with precision as the ordinate and recall as the abscissa. The area under the curve is the average precision (AP) for a certain category. AP is calculated as shown in equation (14).

[0166]

[0167] In equation (14), R is the recall rate and P is the precision rate.

[0168] mAP is the average AP value for all categories, calculated as shown in Equation (15).

[0169]

[0170] In equation (15), mAP is the average AP value for all categories.

[0171] To demonstrate the effectiveness of the improvements made to the Faster R-CNN network in this invention, ablation experiments were conducted to compare the ablation results of some improved modules. PR curves were used to compare the results of the ablation experiments; a larger area enclosed by the PR curve and the coordinate axes indicates better detection performance. The AP value was used as a measure of the detection accuracy of the object detection model.

[0172] like Figure 17 As shown, the mAP value of the model changes with the number of iterations during training. The mAP value increases sharply in the first 30 iterations, increases slowly in the 30-100 iterations, and then stabilizes after 100 iterations, fluctuating around 87.4.

[0173] like Figure 18 As shown, the PR curves of the ResNet50 network before and after the improvement change during training. The improved ResNet50 network achieves an AP value of 87.4%, a 0.2% improvement compared to the original ResNet50 network. It can be seen that the PR curve of ResNet50-C has a larger area enclosed by the coordinate axes than the PR curve of ResNet50. This confirms that the improved ResNet50 network can obtain more feature information from the feature map, and three 3×3 convolutional layers are more sensitive to feature information than a single 7×7 convolutional layer. This indicates that the improved ResNet50 network is more suitable for multi-directional bolt detection than the original ResNet50 network.

[0174] like Figure 19 As shown, the PR curves of the two loss functions change during training. The loss function used in this invention improves the AP value by 3.2% compared to the Smooth L1 loss function. It can be seen that the PR curve of the loss function used in this invention has a larger area enclosed by the coordinate axes compared to the PR curve of the Smooth L1 Loss function. This confirms that the loss function used in this paper can significantly improve the accuracy of bounding box regression and has better performance. This is because the loss function used in this paper can solve the problem of loss discontinuity caused by bounding box regression in multi-directional object detection by transforming the rotated bounding box into a two-dimensional Gaussian distribution.

[0175] R&D Example 3:

[0176] like Figure 5 As shown, the dataset obtained in R&D Example 2 is input into the improved Faster RCNN network model for training, resulting in two trained models. Then, the bolt part image is input into the trained model for prediction, which is the result of the model training.

[0177] like Figure 20a As shown in Figure d, there are four representative test results.

[0178] like Figure 20a The image shows the detection results for bolts when the target bolt is occluded and for small target bolts. It can be seen that the confidence scores for all eight bolts in the image are above 0.89, from left to right: 0.94, 0.92, 0.98, 0.99, 0.98, 0.99, 0.89, 0.98. Each detection box fits the target bolt perfectly. The five small bolts also showed good detection results, with the occluded small bolt (confidence score 0.98) being detected very well. This demonstrates the model's excellent detection capabilities when processing images. Whether the bolt is occluded or a small target bolt, the model can accurately detect the target and provide reliable results.

[0179] like Figure 20b The image shows the detection results of bolts in a complex background during construction. It demonstrates the model's detection results in a practical application of identifying flanges in overhead contact line support foundations. This application scenario presents complex target shapes and background interference; however, the model is still able to accurately identify the target object, with a confidence score of 0.98 for the bolt target, showcasing the model's excellent robustness and environmental adaptability.

[0180] like Figure 20cThe image shows the bolt detection results when the bolt has a small, undefined rotation angle, which can easily lead to angular discontinuities. It presents the detection results when two bolt target images are located within a small angle range not defined by the improved region proposal network, highlighting the model's handling of loss discontinuities. The proposed method, by introducing a loss function based on Gaussian-Wasserstein distance, effectively addresses this problem and demonstrates excellent performance in this scenario. As can be seen, when the bolt target image is within a small angle range not defined by the improved region proposal network, the bolt confidence score reaches above 0.98, and the bounding box area is also optimal. This indicates that the proposed model has the ability to handle situations involving small angles and angular loss discontinuities.

[0181] like Figure 20d The image shows the detection results for bolts with large rotation angles. It demonstrates the model's detection performance when processing target images with large angle ranges. Even when the target image has a large rotation angle, the model can still accurately locate and identify the target, showing good adaptability to large angle changes.

Claims

1. An apparatus for bolt detection, comprising a region of interest pooling layer, a position regression module, a confidence calculation module and a fusion module connected in sequence, a classification prediction module connected between the region of interest pooling layer and the confidence calculation module, an anchor box generation layer, a first convolution layer connection, a second convolution layer, a classification feature transformation layer, an activation function and a classification feature adjustment layer connected in sequence, a third convolution layer and a proposal box, characterized in that: The RESNET50-C network, a feature pyramid, a distance and focus loss function, a dilated convolution layer, a rotated intersection over union calculation module and a soft non-maximum suppression module are further included, the RESNET50-C network is connected with a region of interest pooling layer through the feature pyramid, the feature pyramid is connected with an anchor box generation layer, a classification feature adjustment layer, the dilated convolution layer, the rotated intersection over union calculation module, the soft non-maximum suppression module, a proposal box, the distance and focus loss function and the region of interest pooling layer are sequentially connected, and the first convolution layer is connected with the dilated convolution layer through the third convolution layer. ​ The RESNET50-C network is obtained based on the improved ResNet50 network, the ResNet50 network is composed of five stages, in the zeroth stage, a 7*7 convolution layer with a step of 2 and a channel number of 64 is decomposed into three 3*3 convolution layers with steps of [32, 32, 64] and channel numbers of [1, 2, 1] respectively; in the fourth stage, in the downsampling module, the steps of the first two convolution layers in the first path A are exchanged, a 2*2 average pooling layer with a step of 2 is added in the second path B, and the step size of the 1*1 convolution layer is changed to 1, the outputs of the first path A and the second path B are assimilated, and the outputs of the two paths are added as the overall output of the downsampling module; The dilated convolution layer first passes the generated feature map through a 3*3 convolution layer, then performs weight compensation and angle assignment, the weight compensation adjusts the anchor box size according to the feature map content and the pre-labeled real box, and the angle assignment fine-tunes the rotation angle of the anchor box according to the pre-labeled real box, thereby optimizing the rotated anchor box; The device for bolt detection is inputted with a picture containing a bolt to obtain a detection result.

2. A device for bolt detection according to claim 1, characterized in that: The anchor box generation layer increases the setting of angle values [-10°, -25°, -40°, -55°, -70°, -90°], changes the scale values to [4, 8, 16], and changes the aspect ratio to [1:3, 1:5, 1:7, 1:9].

3. The device for bolt detection according to claim 1, characterized in that: The first convolution layer is a 3*3 convolution layer.

4. The apparatus for bolt detection of claim 1, wherein: The second convolution layer is a 1*1 convolution layer.

5. The apparatus for bolt detection of claim 1, wherein: The third convolution layer is a 1*1 convolution layer.

6. A method for bolt detection, characterized by: The device for bolt detection according to any one of claims 1 to 5 comprises a detection step, and the detection step comprises inputting a picture containing a bolt into the device for bolt detection to obtain a detection result.

7. A method for bolt detection according to claim 6, characterized in that: In the detection step, the detection result comprises a confidence of the bolt.

8. An apparatus for bolt detection, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the computer program to implement the corresponding steps in claims 6 or 7.

9. An apparatus for bolt detection, comprising a computer readable storage medium having stored thereon a computer program, the computer program comprising instructions for causing a computer to perform the steps of: The computer program is executed by the processor to implement the corresponding steps in claims 6 or 7.

Images