Bridge crane anti-collision early warning method based on binocular vision system

Abstract

The invention belongs to the technical field of target detection, and discloses a bridge crane anti-collision early warning method based on a binocular vision system, which comprises the following steps: (1) using two binocular cameras to obtain 360-degree monitoring videos with a lifting hook as a center; (2) extracting image data for preprocessing; (3) inputting the pre-processed image into an algorithm of a small-MobileNetv3 backbone network, and inputting the pre-processed image into the algorithm of the small-MobileNetv3 backbone network; an exchange attention mechanism Shuffle Attention module is constructed in the backbone network; (4) judging whether the detection results of the left and right images have a corresponding relation, if so, judging a target distance, otherwise, continuing detection; (5) judging whether a human-shaped living body exists within a range of 3 meters, if so, performing active collision prevention, namely emergency stop, and otherwise, continuing detection; and (6) judging whether a human-shaped living body exists within a range of 4 meters, if so, carrying out dangerous state early warning, otherwise, continuing detection.

Description

Technical Field

[0001] The invention belongs to the technical field of target detection, and in particular relates to an anti-collision warning method for a bridge crane based on a binocular vision system. Background Art

[0002] Bridge cranes usually operate in large warehouses, factories, ports and other environments, which are usually very crowded with many obstacles and human activities. Due to the huge weight and size of bridge cranes, once a collision occurs, it may cause serious property damage or even casualties. Operators may have difficulty accurately judging the surrounding environment due to poor vision. Traditional anti-collision systems usually rely on sensor technologies such as ultrasonic sensors, lidar or radar systems, which can provide certain collision warnings, but have limited accuracy in complex environments. CCTV is used for monitoring, but due to the lack of intelligent analysis capabilities, operators need to focus on the screen for a long time, which is prone to fatigue and negligence.

[0003] Related reference CN115180522A discloses a safety monitoring method for a lifting device construction site, which tracks specific human objects in real time and extracts features in the current image through a backbone network. The backbone network is a Ghostnet structure, which fuses features through the neck layer and cannot combine channel and spatial attention. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a bridge crane anti-collision warning method based on a binocular vision system that can allow the system to quickly adjust the warning strategy according to changes in the current environment based on real-time data processing capabilities and better combine channel and spatial attention.

[0005] In order to solve the above technical problems, the present invention provides a bridge crane anti-collision warning method based on a binocular vision system, comprising the following steps: (1) Use two binocular cameras to obtain 360° surveillance video centered on the hook; (2) Extract image data for preprocessing; (3) Input the preprocessed image into the algorithm of the small-MobileNetv3 backbone network; Construct the Shuffle Attention module in the backbone network; (4) Determine whether there is a corresponding relationship between the detection results of the left and right images. If so, determine the target distance; otherwise, continue detection; (5) Determine whether there is a living human figure within a 3-meter range. If so, perform active collision avoidance, i.e. emergency stop; otherwise, continue detection; (6) Determine whether there is a living human figure within a 4-meter range. If so, issue a dangerous status warning; otherwise, continue detection.

[0006] Preferably, step (1) includes the following sub-steps: (1.1) Two binocular cameras are installed on both sides of the top of the hook of the bridge crane to obtain 360° monitoring video centered on the hook; (1.2) Use Zhang Zhengyou calibration method to calibrate the binocular camera and Bouguet algorithm to perform stereo correction on the binocular camera to obtain the distortion parameters, intrinsic parameter matrix and rotation and translation matrices of the left and right cameras as well as de-distortion processing.

[0007] Preferably, step (2) includes the following sub-steps: (2.1) First, extract frames from the real-time video of the bridge crane, and then scale each frame to make the original image size match the model; (2.2) The image is then normalized to scale the pixel values ​​to the range required by the model; (2.3) Use data augmentation strategies to randomly flip, crop, and rotate images to increase the generalization ability of the model.

[0008] Preferably, step (3) includes the following sub-steps: (3.1) In order to better combine channel and spatial attention, a Shuffle Attention (SA) module is constructed in the backbone network.

[0009] (3.2) The feature map output by the backbone network will pass through the SPP (Spatial Pyramid Pooling) module. In order to solve the inconsistency of pyramid feature representation and improve the detection ability of multi-scale targets, a multi-scale feature fusion method Adaptively Spatial Feature Fusion (ASFF) is constructed. The output of the SPP module will pass through the ASFF module.

[0010] (3.3) The feature map will enter the head part of the backbone network, and the bounding box, category confidence and object confidence output by the model will go through the post-processing steps: SPP module, ASFF module. Non-maximum suppression (NMS) is used to eliminate overlapping bounding boxes. The bounding box, category and confidence score after post-processing constitute the final detection result.

[0011] Preferably, step (4) includes the following sub-steps: (4.1) Based on the detection results output by the left and right views, determine whether there is a corresponding relationship.

[0012] (4.2) If there is a corresponding relationship, the target distance information is calculated based on the disparity map and camera parameters.

[0013] Preferably, step (5) includes the following sub-steps: (5.1) Based on the output target distance information, determine whether there is a living humanoid within a 3-meter range centered on the hook.

[0014] (5.2) If yes, perform active collision avoidance operation; if no, continue detection.

[0015] Preferably, step (6) includes the following sub-steps: (6.1) Based on the output target distance information, determine whether there is a living humanoid within a 4-meter range centered on the hook.

[0016] (6.2) If present, issue a dangerous status warning; if not, continue testing.

[0017] Preferably, the SPP module is fused with the ASFF module via Concat.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention adopts a six-step binocular vision system-based bridge crane anti-collision warning method to obtain real-time target distance and monitor objects in the surrounding environment in real time. Operators can detect potential collision risks and issue warnings early without paying attention to the screen for a long time, and actively prevent collisions, better combine channel and spatial attention, and provide operators with decision support based on data analysis, helping them make better judgments in complex situations and improve the safety and efficiency of bridge crane operations. By lightweighting the target detection backbone network, the depth and complexity of the backbone network are greatly reduced, and the reasoning speed is improved while ensuring detection accuracy.

[0019] 2. The present invention provides a method for anti-collision warning of bridge cranes based on a binocular vision system. Through data enhancement strategies, the model's ability to detect human figures under different postures, angles and lighting conditions is improved. Through real-time monitoring and detection, the system can provide early warning when people approach dangerous areas, providing operators with enough time to take preventive measures.

[0020] 3. Adaptive fusion: Use Concat connection to fuse the SPP and ASFF modules by splicing. The number of channels of each detection branch can be adjusted as needed, further improving the detection capability of multi-scale targets.

[0021] 4. Through the combination of SPP and ASFF modules, the system can effectively handle targets of different sizes and improve detection accuracy and speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of the method of the present invention; Figure 2 The structure diagram of the Conv and Bneck modules in step (3.1) of the present invention; DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0024] like Figure 1 As shown in the figure, an anti-collision warning method for a bridge crane based on a binocular vision system provided by an embodiment includes the following steps: (1) Use two binocular cameras to obtain 360° surveillance video centered on the hook; This step (1) includes the following sub-steps: (1.1) Two binocular cameras are installed on both sides of the top of the hook of the bridge crane to obtain a 360° monitoring video centered on the hook; (1.2) Use Zhang Zhengyou calibration method to calibrate the binocular camera and Bouguet algorithm to perform stereo correction on the binocular camera to obtain the distortion parameters, intrinsic parameter matrix and rotation and translation matrices of the left and right cameras as well as de-distortion processing.

[0025] (2) Extract image data for preprocessing; This step (2) includes the following sub-steps: (2.1) First, extract frames of the real-time video of the suspended concrete bridge-building machine at a certain frame rate, and then scale each frame to make the original image size match the model; (2.2) The image is then normalized to scale the pixel values ​​to the range required by the model; (2.3) Use data augmentation strategies (existing technology) to randomly flip, crop, and rotate images to increase the generalization ability of the model and improve the model's ability to detect human figures under different postures, angles, and lighting conditions.

[0026] (3) Input the preprocessed image into the algorithm of the small-MobileNetv3 backbone network; This step (3) includes the following sub-steps: (3.1) The input image enters the small-MobileNetv3 backbone network, which extracts multi-scale features of the image through a series of convolutional layers, batch normalization layers, activation functions, and pooling layers. By lightweighting the target detection backbone network small-MobileNetv3, the depth and complexity of the backbone network are greatly reduced, which improves the inference speed while ensuring the detection accuracy.

[0027] like Figure 2 As shown in the figure, the lightweight backbone network is mainly composed of 3x3 Conv, 3x3 Bneck and 5x5 Bneck. 3x3Conv is composed of standard convolution with kernel size of 3 and stride of 2, batch normalization layer and h-swish activation function; the kxkBneck structure divides the input into two branches. The first branch passes through the standard convolution with kernel size of 1 to the first batch normalization BN layer and nonlinear activation function layer, and then passes through the standard convolution layer with kernel size of k and stride of s. Finally, it passes through the SE attention structure and the second standard convolution with kernel size of 1, batch normalization BN layer and nonlinear activation function layer and outputs. The other branch does not perform any operation, but is directly spliced ​​with the first branch by Concat and then output.

[0028] In the 1st, 3rd, 5th, 8th and 15th layers of the small-MobileNetv3 backbone network structure, standard convolution modules with a stride of 2 are used to halve the length and width of the input feature map. The stride of the standard convolution modules in other layers is 1. The SE attention module is introduced in the Bneck modules in the 5th to 7th and 12th to 16th layers, and no other Bneck modules are added. The k value of the Bneck modules in the 5th to 7th and 14th to 16th layers is 5, and the k value of other Bneck modules is 3. The nonlinear activation function used in the 2nd to 7th layers is ReLU, while the other layers use h-swish. The feature map sizes of the backbone output are 52x52, 26x26, and 13x13.

[0029] In order to better combine channel and spatial attention, a Shuffle Attention (SA) module is built in the backbone network. The Shuffle Attention module is an attention mechanism model for processing sequence data. Through multiple iterations, the order of sequence data is randomly disrupted, and then the attention weights are assigned to elements of different channel groups in each iteration.

[0030] For the input feature map , where C, H and W represent the number of channels, length and height of the feature map respectively. SA will divide X into G groups according to the number of channels C. , , for one of the sub-feature graphs The input will continue to be divided into two groups according to the number of channels C, namely X K1 and X K2 , .X K1 It will be used to generate channel attention maps, aggregate global feature information through global average pooling, and generate , the formula is as follows:

[0031] Then s will pass through a simple adaptive parameter selection module to obtain the output value of the channel attention, the formula is as follows: in: and represent the scaling factor and bias respectively, Represents a nonlinear activation function .

[0032] Another branch X K2 It will be used to generate the spatial attention map. First, group normalization (GN) is performed to adjust its spatial information distribution, and then Fc (·) is used to enhance X K2 The representation ability of , and finally the output value of spatial attention is obtained, the formula is as follows: in, and represent the scaling factor and bias respectively.

[0033] Then X' K1 , X' K2 and X K1 , X K2 An element-by-element multiplication operation is performed to achieve attention fusion, and the outputs of the two branches are concatenated to form a new sub-feature map, which is then aggregated for channel exchange operation to achieve cross-group information flow in the channel dimension.

[0034] (3.2) The feature map output by the backbone network will pass through the SPP (Spatial Pyramid Pooling) module. In order to solve the inconsistency of pyramid feature representation and improve the detection ability of multi-scale targets, a multi-scale feature fusion method Adaptively Spatial Feature Fusion (ASFF) is constructed. The output of the SPP module will further pass through the ASFF module.

[0035] ASFF mainly includes rescaling and adaptive fusion of feature maps. Rescaling of feature maps: When upsampling, first use a 1x1 standard convolution layer to compress the number of feature map channels, and then use the nearest neighbor interpolation method to improve the resolution. Depending on the level, the scaling factor is set to 2 or 4. For 1 / 2 ratio downsampling, use a depthwise separable convolution with a kernel size of 3 and a stride of 2 to adjust the number of channels and resolution. For 1 / 4 ratio downsampling, two depthwise separable convolutions with a kernel size of 3 and a stride of 2 are cascaded. Adaptive fusion: Use Concat connection to fuse SPP and ASFF modules by splicing, which can adjust the number of channels of each detection branch as needed, further improving the detection capability of multi-scale targets. Through the combination of SPP and ASFF modules, the system can effectively handle targets of different sizes and improve detection accuracy and speed.

[0036] (3.3) The feature map will enter the head part of the backbone network, and the bounding box, category confidence and object confidence output by the model will go through the post-processing steps: SPP module, ASFF module. Non-maximum suppression (NMS) is used to eliminate overlapping bounding boxes. The bounding box, category and confidence score after post-processing constitute the final detection result.

[0037] (4) Determine whether there is a corresponding relationship between the detection results of the left and right images. If so, determine the target distance; otherwise, continue detection.

[0038] This step (4) includes the following sub-steps: (4.1) Based on the detection results output by the left and right views, determine whether there is a corresponding relationship.

[0039] (4.2) If there is a corresponding relationship, calculate the target distance information based on the disparity map and camera parameters; otherwise continue with the detection.

[0040] (5) Determine whether there is a living human figure within a 3-meter range. If so, perform active collision avoidance, i.e., emergency stop. Otherwise, continue detection.

[0041] This step (5) includes the following sub-steps: (5.1) Based on the output detection results, determine whether there is a living humanoid within a 3-meter range centered on the hook.

[0042] (5.2) If yes, perform active collision avoidance operation; if no, continue detection.

[0043] (6) Determine whether there is a living human figure within a range of 4 meters. If so, issue a warning of a dangerous state. Otherwise, continue to detect and obtain the real-time target distance. Monitor objects in the surrounding environment in real time. Operators can detect potential collision risks early and issue warnings without having to pay attention to the screen for a long time. Actively avoid collisions and provide decision support based on data analysis for operators, helping them make better judgments in complex situations and improving the safety and efficiency of bridge crane operations.

[0044] This step (6) includes the following sub-steps: (6.1) Based on the output detection results, determine whether there is a living humanoid within a 4-meter range centered on the hook.

[0045] (6.2) If present, issue a dangerous status warning; if not, continue testing.

[0046] Through real-time monitoring and detection, the system can provide early warning when people approach dangerous areas, providing operators with enough time to take preventive measures. It can also easily adapt to new application scenarios and technological advances by updating training data sets and model parameters.

[0047] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A bridge crane anti-collision warning method based on a binocular vision system, characterized in that: The steps include: (1) Use two binocular cameras to obtain 360° surveillance video centered on the hook; (2) Extract image data for preprocessing; (3) Input the preprocessed image into the algorithm of the small-MobileNetv3 backbone network; Construct the Shuffle Attention module in the backbone network; (4) Determine whether there is a corresponding relationship between the detection results of the left and right images. If so, determine the target distance; otherwise, continue detection; (5) Determine whether there is a living human figure within a 3-meter range. If so, perform active collision avoidance, i.e. emergency stop; otherwise, continue detection; (6) Determine whether there is a living human figure within a 4-meter range. If so, issue a dangerous status warning; otherwise, continue detection.

2. The anti-collision warning method for a bridge crane based on a binocular vision system according to claim 1, wherein step (1) comprises the following sub-steps: (1.1) Two binocular cameras are installed on both sides of the top of the hook of the bridge crane to obtain 360° monitoring video centered on the hook; (1.2) Use Zhang Zhengyou calibration method to calibrate the binocular camera and Bouguet algorithm to perform stereo correction on the binocular camera to obtain the distortion parameters, intrinsic parameter matrix and rotation and translation matrices of the left and right cameras as well as de-distortion processing.

3. The anti-collision warning method for a bridge crane based on a binocular vision system according to claim 1, wherein step (2) comprises the following sub-steps: (2.1) First, extract frames from the real-time video of the bridge crane, and then scale each frame to make the original image size match the model; (2.2) The image is then normalized to scale the pixel values ​​to the range required by the model; (2.3) Use data augmentation strategies to randomly flip, crop, and rotate images to increase the generalization ability of the model.

4. The anti-collision warning method for a bridge crane based on a binocular vision system according to claim 1, wherein step (3) comprises the following sub-steps: (3.2) The feature map output by the backbone network will pass through the SPP (Spatial Pyramid Pooling) module; in order to solve the inconsistency of pyramid feature representation and improve the detection ability of multi-scale targets, a multi-scale feature fusion method adaptive spatial feature fusion (ASFF) is constructed; the output of the SPP module will pass through the ASFF module; (3.3) The feature map will enter the Head part of the backbone network. The bounding box, category confidence and object confidence output by the model will go through post-processing steps: SPP module, ASFF module; non-maximum suppression (NMS) is used to eliminate overlapping bounding boxes. The bounding box, category and confidence score after post-processing constitute the final detection result.

5. The anti-collision warning method for a bridge crane based on a binocular vision system according to claim 1, wherein step (4) comprises the following sub-steps: (4.1) Determine whether there is a corresponding relationship based on the detection results output by the left and right views; (4.2) If there is a corresponding relationship, the target distance information is calculated based on the disparity map and camera parameters.

6. The anti-collision warning method for a bridge crane based on a binocular vision system according to claim 1, wherein the step (5) comprises the following sub-steps: (5.1) Based on the output target distance information, determine whether there is a living humanoid within a 3-meter range centered on the hook; (5.2) If yes, perform active collision avoidance operation; if no, continue detection.

7. The anti-collision warning method for a bridge crane based on a binocular vision system according to claim 1, wherein the step (6) comprises the following sub-steps: (6.1) Based on the output target distance information, determine whether there is a living humanoid within a 4-meter range centered on the hook; (6.2) If present, issue a dangerous status warning; if not, continue testing.

8. The anti-collision warning method for bridge crane based on binocular vision system as described in claim 4, wherein the SPP module is fused through Concat and ASFF modules.

Images