Image recognition system suitable for pole climbing robot

Abstract

The embodiment of the application provides an image recognition system suitable for a pole climbing robot, comprising a basic unit, which is obtained by modifying an image classification network based on a deep convolution network to realize extraction of feature information of an input image; a neck unit, which is used for expanding a receptive field of a model and performing feature fusion on feature maps of different sizes; and a head unit, which is used for classification and bounding box regression according to final features extracted by the two network modules and obtaining a final prediction result. By recording a working process of a worker, real-time detection on safety violation behaviors of the worker is realized, and timely warning is given.

Description

Technical Field

[0001] This application relates to the field of image recognition, and more particularly to an image recognition system suitable for pole climbing robots. Background Technology

[0002] With the continuous advancement of robotics technology, its applications in industries such as medical devices, power grid inspection, and agricultural production are becoming increasingly widespread. Pole-climbing robots play a crucial role in the power industry's distribution network automation process. Unlike mobile robots, pole-climbing robots need to climb object surfaces to complete their movements. During the pole-climbing process, video images are captured and processed to generate judgment commands. However, the comprehensive detection, classification, and localization of violations in the acquired images present a significant challenge. Currently, there is relevant research both domestically and internationally, but the research on robots and violation recognition is relatively independent. On the one hand, integrating these systems and improving them based on the characteristics of pole-climbing robots still requires further in-depth exploration. On the other hand, videos or images captured in actual engineering projects are affected by on-site factors, and postures constantly change during the climbing and operation process. Therefore, different technologies are needed to handle different application scenarios, thereby reducing the impact of factors such as light, angle, and weather, while simultaneously capturing human posture information in real time under different conditions. Therefore, it is necessary to develop appropriate image recognition and processing methods based on the pole-climbing characteristics of pole-climbing robots.

[0003] Traditional manual pole climbing operations involve scattered work locations and random times, making it difficult to effectively supervise workers on-site to ensure the standardization and normalization of the work process. Summary of the Invention

[0004] This application proposes an image recognition system suitable for pole climbing robots. It records the operation process of workers through an intelligent vision terminal, realizes real-time detection of workers' safety violations, and provides timely warnings.

[0005] The image recognition system for pole-climbing robots proposed in this application includes:

[0006] The basic unit is obtained by modifying a deep convolutional network-based image classification network to extract feature information from the input image.

[0007] The neck unit is used to expand the receptive field of the model and to fuse features from feature maps of different sizes.

[0008] The head unit is used to perform classification and bounding box regression based on the final features extracted by the two network modules mentioned above, and to obtain the final prediction result.

[0009] Optionally, the basic unit includes:

[0010] The BottleneckCSP module, representing a cross-stage local bottleneck, is used as the basic module for the basic unit.

[0011] C1 and C2 represent the number of channels in the input feature map and the final output feature map of this module, respectively; r is the channel dilation coefficient, which defaults to 0.5 to reduce the number of parameters; n is the number of times the Bottleneck submodules are stacked in this module; [kxk,C] represents C kxk convolutional kernels, with a default stride of 1; Represents element-wise addition, "concat" represents the concatenation operation of two feature maps along the channel dimension; BN represents batch normalization; Hardswish and LeakyReLU are two common activation functions, and their expressions are shown below;

[0012]

[0013] LeakyReLU(x)=max(0,x)+α*min(0,x);

[0014] The default value is 0.1.

[0015] Optionally, the basic unit includes:

[0016] The BottleneckCSP module, which addresses cross-stage local bottlenecks, separates the input into two branches.

[0017] One branch changes the number of channels in the output feature map through a 1x1 convolution, which is used for subsequent feature concatenation and increases the gradient backpropagation path during model training.

[0018] Another branch operates through n Bottleneck submodules, merging the two branches through splicing and CBM operations.

[0019] Optionally, the basic unit includes:

[0020] The Focus downsampling module is used to obtain a feature map with a width and height of 1 / 2 of the input feature map and 4C1 channels through equal-interval sampling operations. Then, after 1x1 convolution, BN and Hardswish operations, a feature map with C2 channels is output.

[0021] Optionally, the neck unit includes a spatial pyramid pooling module, specifically used for:

[0022] Reduce channel dimension and computational cost by using 1x1 convolution;

[0023] Three feature maps with different receptive fields are obtained by performing three max pooling operations of different sizes, and then concatenated with the feature map of the original input. This enhances the model's multi-scale detection capability while preserving the feature information of the original input.

[0024] After 1x1 convolution, BN and Hardswish operations, features from different receptive fields are further combined to obtain a feature map that integrates rich receptive fields, and a feature map with C2 channels can be output as needed.

[0025] Optionally, the head unit includes:

[0026] The overall network structure of the pole climbing worker detection model based on YOLOv5 is constructed by adopting the sparse prediction method of YOLOv3.

[0027] By fully fusing feature maps from multiple levels, a feature map is obtained for the final prediction.

[0028] Optionally, the image recognition system further includes a loss function construction unit, which specifically consists of three units: classification loss, confidence loss, and coordinate regression loss.

[0029] Based on the sparse prediction method of the head unit, both the classification loss and the confidence loss adopt the binary cross-entropy loss. The Sigmoid function is used to restrict the prediction results to the range [0,1] to achieve multi-label classification. The specific formula is as follows:

[0030] BCELoss(x)=-ω[p·ylogσ(x)+(1-y)log(1-σ(x)];

[0031]

[0032] Where ω represents the scaling factor, which is usually set to 1.0 by default; p represents the positive sample weighting factor, which can be fine-tuned according to actual training needs. Generally speaking, p>1 increases the model's recall, and p<1 increases the model's precision; y represents whether it is a positive or negative sample, y=1 represents a positive sample, and y=0 represents a negative sample.

[0033] Optionally, the image recognition system further includes a training and optimization unit, specifically comprising:

[0034] The pre-trained subunits have positive sample weighting coefficients p of 1.0 for both classification loss and confidence loss, and λ... obj , λ cls , λ boxThe values ​​were set to 0.05, 0.5, and 1.0 respectively. The training cycle of the network was set to 300 times. The parameters were updated using stochastic gradient descent. The learning rate was adjusted using a warm-up method. The initial learning rate was set to 0.01 and the momentum term was set to 0.937.

[0035] During the fine-tuning phase, the weights of the basic units in the pole climber detection model are frozen, and only the weight parameters of the neck and head units are updated to accelerate model training. The positive sample weighting coefficients p for BCELoss in the classification loss and confidence loss are set to 0.631 and 0.911, respectively, and λ... obj , λ cls , λ box The values ​​were set to 0.0296, 0.243, and 0.301 respectively. The training cycle of the network was set to 100 times. The parameters were updated using stochastic gradient descent and the learning rate was adjusted using a warm-up method. The initial learning rate was set to 0.0032 and the momentum term was set to 0.843.

[0036] Beneficial effects:

[0037] A deep neural network model based on YOLOv5, HRNet, ResNet18, and SE channel attention mechanism is trained primarily to detect risks and violations during pole climbing operations. The system records the workers' work process using an intelligent visual terminal, and, in conjunction with an intelligent image recognition system, detects and promptly alerts workers to any safety violations. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the structure of an image recognition system for pole-climbing robots proposed in an embodiment of this application;

[0040] Figure 2 This is a schematic diagram of the Bottleneck CSP module structure of the basic network proposed in the embodiments of this application;

[0041] Figure 3 This is a schematic diagram of the Focus downsampling module structure of the basic network proposed in the embodiments of this application;

[0042] Figure 4 This is a schematic diagram of the spatial pyramid pooling module structure of the basic network proposed in the embodiments of this application;

[0043] Figure 5 This is a schematic diagram of the overall network structure of the YOLOv5-based pole climber detection model proposed in this application embodiment. Detailed Implementation

[0044] To make the structure and advantages of this application clearer, the structure of this application will be further described below with reference to the accompanying drawings.

[0045] To achieve intelligent safety supervision at the pole climbing site, this project employs a deep neural network model based on YOLOv5, HRNet, ResNet18, and SE channel attention mechanism, focusing on training for monitoring risks and violations during pole climbing operations. The project uses intelligent visual terminals to record the workers' work process, and in conjunction with an intelligent image recognition system, it performs real-time detection and timely alerts for any safety violations by workers.

[0046] From the perspective of the objective, detecting violations such as not wearing a safety helmet is essentially a target detection task, and the main input information is image information. Therefore, in order to better utilize all the features of the image information, a convolutional neural network model is selected. The strengths of the YOLOv5 model are well-suited to the needs of this task, and the smaller model size means that YOLOv5 can ensure accuracy while also taking into account real-time performance. Therefore, the YOLOv5 model is selected for violation recognition.

[0047] The image recognition system for pole-climbing robots proposed in this application embodiment, such as Figure 1 As shown, it includes:

[0048] The basic unit is obtained by modifying a deep convolutional network-based image classification network to extract feature information from the input image.

[0049] The neck unit is used to expand the receptive field of the model and to fuse features from feature maps of different sizes.

[0050] The head unit is used to perform classification and bounding box regression based on the final features extracted by the two network modules mentioned above, and to obtain the final prediction result.

[0051] In implementation, the BottleneckCSP module, a cross-stage local bottleneck module, is used as the basic module of the basic unit, such as... Figure 2 As shown, C1 and C2 represent the number of channels in the input feature map and the final output feature map of this module, respectively; r is the channel dilation coefficient, which defaults to 0.5 to reduce the number of parameters; n is the number of times the Bottleneck submodule is stacked in this module; [kxk,C] represents C kxk convolutional kernels, with a default stride of 1; Represents element-wise addition, "concat" represents the concatenation operation of two feature maps along the channel dimension; BN represents batch normalization; Hardswish and LeakyReLU are two common activation functions, and their expressions are shown below;

[0052]

[0053] LeakyReLU(x)=max(0,x)+α*min(0,x);

[0054] The default value is 0.1.

[0055] The BottleneckCSP module, which addresses local bottlenecks across stages, splits the input into two branches. One branch uses a 1x1 convolution to change the number of channels in the output feature map, used for subsequent feature concatenation, increasing the gradient backpropagation path during model training. The other branch operates through n Bottleneck sub-modules, merging the two branches through concatenation and CBM operations. This reduces computational load and accelerates inference speed without compromising the feature extraction capabilities of the base network. Finally, the two branches are merged through concatenation and CBM operations. Furthermore, the Hardswish activation function is an improvement on the Swish function, replacing the time-consuming sigmoid function with ReLU6, reducing the difficulty of deploying the model on embedded devices without sacrificing model performance, thus meeting the needs of lightweight, miniaturized pole-climbing robot embedded devices.

[0056] The basic unit includes:

[0057] Adopting such Figure 3 The Focus downsampling module shown obtains a feature map with a width and height of 1 / 2 of the input feature map and 4C1 channels through equal-interval sampling operations. Then, it outputs a feature map with 4C1 channels after 1x1 convolution, BN and Hardswish operations.

[0058] Basic neural networks typically downsample feature maps using convolutional operations with a stride of 2 or max pooling to reduce the computational cost and number of parameters. However, this initial downsampling causes the model to lose a significant amount of image information, which is detrimental to subsequent classification and regression tasks. Considering the project's requirements for recognizing multiple types of violations, this method is clearly unsuitable for the specific circumstances.

[0059] Therefore, to meet actual needs, the project uses a Focus downsampling module. Through equal-interval sampling, it reduces the width and height of the feature map while preserving the original image information to the greatest extent possible. The specific structure is as follows: Figure 3As shown, the input feature map has C1 channels. After four equally spaced samplings, a feature map with a width and height of 1 / 2 of the input feature map and 4C1 channels is obtained. Then, after 1x1 convolution, BN and Hardswish operations, a feature map with C2 channels is output.

[0060] The neck unit includes, for example: Figure 4 The spatial pyramid pooling module shown is specifically used for:

[0061] Reduce channel dimension and computational cost by using 1x1 convolution;

[0062] Three feature maps with different receptive fields are obtained by performing three max pooling operations of different sizes, and then concatenated with the feature map of the original input. This enhances the model's multi-scale detection capability while preserving the feature information of the original input.

[0063] After 1x1 convolution, BN and Hardswish operations, features from different receptive fields are further combined to obtain a feature map that integrates rich receptive fields, and a feature map with C2 channels can be output as needed.

[0064] In practice, the input images for object detection models are generally of a fixed, single size. However, the size of the targets to be detected in the images varies, a phenomenon particularly prominent during pole climbing and other operational processes. Therefore, improving the receptive field of the base network while considering both global and local receptive fields is crucial for enhancing model performance.

[0065] To address this issue, the Spatial Pyramid Pooling (SPP) module effectively expands the model's receptive field and improves the performance of multi-scale object detection through max-pooling operations of varying sizes. The specific structure is as follows: Figure 4 As shown, "Maxpool, k" represents a kxk max pooling operation. The input feature map is first reduced in channel dimension through a 1x1 convolution to decrease computation. Next, three max pooling operations of different sizes are used to obtain feature maps with three different receptive fields, which are then concatenated with the original input feature map. This enhances the model's multi-scale detection capability while preserving the original input feature information. Finally, through 1x1 convolution, Batch Normalization (BN), and Hardswish operations, the features from different receptive fields are further combined to obtain a feature map with a rich receptive field. A feature map with C2 channels can be output as needed.

[0066] Optionally, the head unit includes:

[0067] The overall network structure of the pole climbing worker detection model based on YOLOv5 is constructed by adopting the sparse prediction method of YOLOv3.

[0068] By fully fusing feature maps from multiple levels, a feature map is obtained for the final prediction.

[0069] In implementation, based on the aforementioned units and drawing inspiration from the DarkNet53 network structure, the basic network structure of the pole-climbing worker detection model is shown in Table 1. In the last row of the table, all Bottleneck modules in the BottleneckCSP operator have no skip connections.

[0070] Table 1. Basic Network Structure of the Pole Climbing Worker Detection Model

[0071]

[0072] To address the need for real-time performance, the sparse prediction method of YOLOv3 is used as the head part, resulting in the overall network structure of the YOLOv5-based pole-climbing worker detection model as follows: Figure 5 As shown. C2-C5 represent feature map sets of the same size, and kC is the original... Figure 1 The feature map is of size / 2k; "x2" represents nearest neighbor upsampling with a stride of 2; " / 2" represents the stride of the first 3x3 convolutional kernel in the corresponding module being 2, i.e., a 1 / 2 downsampling operation is performed on the feature map. Through the full fusion of feature maps at multiple levels, feature maps of three levels, N3-N5, are obtained for the final prediction.

[0073] Optionally, the image recognition system further includes a loss function construction unit, which specifically consists of three units: classification loss, confidence loss, and coordinate regression loss.

[0074] Based on the sparse prediction method of the head unit, both the classification loss and the confidence loss adopt the binary cross-entropy loss. The Sigmoid function is used to restrict the prediction results to the range [0,1] to achieve multi-label classification. The specific formula is as follows:

[0075] BCELoss(x)=-ω[p·ylogσ(x)+(1-y)log(1-σ(x)];

[0076]

[0077] Where ω represents the scaling factor, which is usually set to 1.0 by default; p represents the positive sample weighting factor, which can be fine-tuned according to actual training needs. Generally speaking, p>1 increases the model's recall, and p<1 increases the model's precision; y represents whether it is a positive or negative sample, y=1 represents a positive sample, and y=0 represents a negative sample.

[0078] Optionally, the image recognition system further includes a training and optimization unit, specifically comprising:

[0079] The pre-trained subunits have positive sample weighting coefficients p of 1.0 for both classification loss and confidence loss, and λ... obj , λ cls , λ box The values ​​were set to 0.05, 0.5, and 1.0 respectively. The training cycle of the network was set to 300 times. The parameters were updated using stochastic gradient descent. The learning rate was adjusted using a warm-up method. The initial learning rate was set to 0.01 and the momentum term was set to 0.937.

[0080] During the fine-tuning phase, the weights of the basic units in the pole climber detection model are frozen, and only the weight parameters of the neck and head units are updated to accelerate model training. The positive sample weighting coefficients p for BCELoss in the classification loss and confidence loss are set to 0.631 and 0.911, respectively, and λ... obj , λ cls , λ box The values ​​were set to 0.0296, 0.243, and 0.301 respectively. The training cycle of the network was set to 100 times. The parameters were updated using stochastic gradient descent and the learning rate was adjusted using warmup. The initial learning rate was set to 0.0032 and the momentum term was set to 0.843.

[0081] In implementing the summary, considering that the pole climber detection model is the foundation for subsequent keypoint detection and violation identification tasks, the YOLOv5x model was adopted for pole climber detection in the project to achieve better accuracy. The entire training process was divided into two sub-units: pre-training and fine-tuning training.

[0082] In the pre-trained subunits, the positive sample weighting coefficient p in both the classification loss and confidence loss of BCELoss is set to 1.0, and λ... obj , λ cls、 λ box The values ​​are set to 0.05, 0.5, and 1.0 respectively. The network training epcoh is set to 300 times, the parameters are updated using stochastic gradient descent, and the learning rate is adjusted using a warm-up method. The initial learning rate is set to 0.01, and the momentum term is set to 0.937.

[0083] The fine-tuning subunit, following the transfer learning approach, freezes the weights of the basic network part of the pole climber detection model, updating only the weight parameters of the neck and head regions to accelerate model training. The positive sample weighting coefficients p for BCELoss in the classification loss and confidence loss are set to 0.631 and 0.911 respectively, with λ... obj , λ cls , λ boxThe values ​​were set to 0.0296, 0.243, and 0.301, respectively. The network training cycle was set to 100 times, and the parameters were updated using stochastic gradient descent and the learning rate was adjusted using a warmup method. The initial learning rate was set to 0.0032, and the momentum term was set to 0.843.

[0084] The above description is merely an embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image recognition system suitable for pole-climbing robots, characterized in that, The system includes: The basic unit is used to modify the image classification network based on deep convolutional networks to extract feature information from the input image. The basic unit includes: a cross-stage local bottleneck (BottleneckCSP) module that separates the input into two branches; one branch changes the number of channels of the output feature map through a 1x1 convolution, which is used for subsequent feature concatenation and increases the gradient backpropagation path during model training; the other branch is operated by n Bottleneck sub-modules, and the two branches are merged through concatenation and CBM operations; the basic unit also includes: a Focus downsampling module that obtains a feature map with a width and height of 1 / 2 of the input feature map and 4C1 channels through equal-interval sampling operations, and then outputs a feature map with 4C2 channels through 1x1 convolution, BN and Hardswish operations. The neck unit is used to expand the receptive field of the model and perform feature fusion on feature maps of different sizes. The neck unit includes a spatial pyramid pooling module, which is specifically used to: reduce the channel dimension through 1x1 convolution to reduce computation; obtain feature maps with three different receptive fields through three max pooling operations of different sizes, and concatenate them with the feature map of the original input to enhance the model's multi-scale detection capability while preserving the feature information of the original input; and further combine the features of different receptive fields through 1x1 convolution, BN, and Hardswish operations to obtain a feature map with a rich receptive field, and can output a feature map with C2 channels as needed. The head unit is used to perform classification and bounding box regression based on the final features extracted by the two network modules mentioned above, and to obtain the final prediction result.

2. The image recognition system for pole-climbing robots according to claim 1, characterized in that, The basic unit includes: The BottleneckCSP module, representing a cross-stage local bottleneck, is used as the basic module for the basic unit. C1 and C2 represent the number of channels in the input and output feature maps of this module, respectively; r is the channel dilation coefficient, which defaults to 0.5 to reduce the number of parameters; n is the number of stacking operations of the Bottleneck submodules in this module; [kxk,C] represents C kxk convolutional kernels, with a default stride of 1. "Represents element-wise addition," "concat" represents the concatenation operation of two feature maps along the channel dimension; BN represents batch normalization; Hardswish and LeakyReLU are two common activation functions, and their expressions are shown below; ； ； The default value is 0.

1.

3. The image recognition system for pole-climbing robots according to claim 1, characterized in that, The head unit includes: The overall network structure of the pole climbing worker detection model based on YOLOv5 is constructed by adopting the sparse prediction method of YOLOv3. By fully fusing feature maps from multiple levels, a feature map is obtained for the final prediction.

4. The image recognition system for pole-climbing robots according to claim 1, characterized in that, The image recognition system also includes a loss function construction unit, which is specifically composed of three units: classification loss, confidence loss, and coordinate regression loss. Based on the sparse prediction method of the head unit, both the classification loss and the confidence loss adopt the binary cross-entropy loss. The Sigmoid function is used to restrict the prediction results to the range [0,1] to achieve multi-label classification. The specific formula is as follows: ； ； in, The scale factor is represented by 1.0, which is usually set to 1.0 by default. p represents the weighting factor for positive samples, which can be fine-tuned according to actual training needs. Generally speaking, p>1 increases the recall of the model, and p<1 increases the precision of the model. y represents whether it is a positive or negative sample, y=1 represents a positive sample, and y=0 represents a negative sample.

5. The image recognition system for pole-climbing robots according to claim 1, characterized in that, The image recognition system further includes a training and optimization unit, specifically comprising: In the pre-trained sub-units, the positive sample weighting coefficient p of BCELoss in both the classification loss and confidence loss is set to 1.

0. , , The values ​​were set to 0.05, 0.5, and 1.0 respectively. The training cycle of the network was set to 300 times. The parameters were updated using stochastic gradient descent. The learning rate was adjusted using a warm-up method. The initial learning rate was set to 0.01 and the momentum term was set to 0.

937. Fine-tuning the sub-units involves freezing the weights of the basic units in the pole climber detection model and updating only the weight parameters of the neck and head units to accelerate model training. The positive sample weighting coefficients p for BCELoss in the classification loss and confidence loss are set to 0.631 and 0.911, respectively. , , The values ​​were set to 0.0296, 0.243, and 0.301 respectively. The training cycle of the network was set to 100 times. The parameters were updated using stochastic gradient descent and the learning rate was adjusted using a warm-up method. The initial learning rate was set to 0.0032 and the momentum term was set to 0.843.

Images