Human, machine and cargo safety identification method based on rotation detection mechanism of self-adaptive target object of port smart brain

Abstract

The invention discloses a human, machine and cargo safety identification method based on a port intelligent brain self-adaptive target object rotation detection mechanism, and the method comprises the following steps: 1, carrying out the labeling of sample images of various cargo hangers and persons, obtaining the labels of each cargo hanger and person, and constructing a data set containing the labels of the hangers and persons; step 2, training by using the sample set improved yolov8 model to obtain a trained improved yolov8 model as a target detection model, introducing a classification detection head into a DWConv optimization detection head by the improved yolov8 model, improving the recognition rate, and performing post-processing on result output; and step 3, inputting a to-be-predicted picture into the improved target detection model, and outputting a horizontal frame coordinate and an angle value as a final result. According to the method, an automatic identification and detection system is adopted to replace manual observation, the perimeter below the lifting appliance is automatically detected through an algorithm, and manual fence drawing is not needed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for safely identifying people, machines and cargoes based on a rotation detection mechanism of an adaptive target object based on a port smart brain. Background Art

[0002] The port operation environment is complex, involving a large number of heavy machinery and vehicles, as well as busy logistics activities. Traditional manual monitoring and management methods are difficult to fully and real-time grasp the safety status of the operation site. The present invention integrates advanced visual perception technology and intelligent analysis algorithms to achieve real-time monitoring of port operations and early warning of safety hazards, effectively reducing accident risks and ensuring the safety of personnel and property.

[0003] During the operation of large machinery in the port, a large number of hooks and slings are used, and there are many loading and unloading links that require human-machine cooperation, such as unhooking, unhooking, mechanical cargo stacking in the cabin, and hand command. Therefore, there are many safety hazards in the process of human-machine interaction. There is an urgent need for a safety detection system to transform from human defense to technical defense, and from passive to active warning. We proposed an AI safety warning system for gantry cranes. A vertical monitoring device is deployed at the trunk of the gantry crane. The algorithm uses an edge AI computing terminal and is equipped with a sound device and deployed in the cab. The algorithm monitors the sling and the cargo under the sling in real time and automatically identifies its boundaries to form a virtual dynamic electronic fence. When a person or machine enters the electronic fence below, an alarm is automatically generated and a sound alarm is output to remind drivers and operators to pay attention to safety. Summary of the invention

[0004] The purpose of the present invention is to provide a method for safely identifying people, machines and cargoes based on a rotation detection mechanism of an adaptive target object based on a port smart brain, so as to solve the current market problems raised by the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for safely identifying people, machines and cargoes based on a rotation detection mechanism of a port smart brain that can adapt to target objects, comprising the following steps:

[0006] Step 1: Label sample images of various cargo spreaders and people, obtain labels for each cargo spreader and person, and construct a data set containing labels of spreaders, cargo, and people;

[0007] Step 2: Use the above sample set to improve the yolov8 model for training, and obtain the trained improved yolov8 model as the target detection model. The improved yolov8 model introduces DWConv (depth separable convolution) into the classification detection head to optimize the detection head, improve the recognition rate, and output the result for post-processing;

[0008] Step 3: Input the image to be predicted into the improved target detection model, and output the horizontal box coordinates and angle values, which are the final results.

[0009] Preferably, the labels of all sample images in step 2 are (c, x, y, w, h, Q), where c is the target category, (x, y) is the center point, w and h are the width and height of the horizontal box, and Q is the angle of rotation of the horizontal box.

[0010] Preferably, the improved yolov8 model described in step 2 includes an image input part, a backbone network extraction part, a feature fusion part, and a model detection head.

[0011] Preferably, in the image input part, the picture is enhanced through Mosaic data, supports multiple input sizes, can automatically adapt to model training and reasoning of different resolutions, flip, crop, scale, change brightness, saturation, and hue of each picture, initialize the anchor scale through K-means, and perform cluster analysis on the real bounding boxes of all annotated data.

[0012] Preferably, the backbone network extraction part is composed of 5 CBS structures, 4 C2F structures, and 1 SPPF structure. The CBS structure is composed of a 2D convolution, a BatchNorm2d, and a SiLU activation function. The C2F structure is composed of a 1x1 convolution channel number reduced to 1 / 2 of the original, multiple 3x3 convolution kernels for feature extraction, residual links, and a 1x1 convolution kernel for restoring the number of channels of the feature map; the SPPF structure is fused through three 5×5 maximum pooling operations.

[0013] Preferably, the feature fusion part is composed of FPN and PAN, and the outputs P3, P4, and P5 of the feature extraction backbone network are input into the PAN-FPN network structure to fuse feature maps of multiple scales; P5 is fused with P4 after upsampling to obtain F1, F1 is fused with P3 after a C2f layer and an upsampling once to obtain T1, T1 is fused with F1 after a convolution layer to obtain F2, F2 is passed through a C2f layer to obtain T2, T2 is fused with P5 after a convolution layer to obtain F3, F3 is passed through a C2f layer to obtain T3, and finally the feature maps of T1, T2, and T3 detection heads are obtained.

[0014] Preferably, the detection head part, each detection head consists of three parts, frame prediction (two 3x3 convolutions, one 1x1 convolution), category prediction (two DWConv convolutions, three 1x1 convolutions), angle prediction (newly added angle prediction, consisting of two DWConv convolutions, three 1x1 convolutions).

[0015] Preferably, the angle loss function is calculated by calculating the Gaussian distribution of the box through Σ, the formula is Σ(h,w,Q-π / 2), and then calculating the Gaussian distance between the boxes instead of the IOU loss function, the formula is: d 2 =lm1-m2l22+Tr(Σ1+Σ2-2(Σ1 / 21Σ2Σ1 / 21) 1 / 2 ;

[0016] Box loss function calculation, FL(p,q)=-q(qlog(p)+(1-q)log(1-p))q>0or-αp β (log(1-p)q=0;

[0017] Classification loss function, M is the number of categories, y if the true category of sample i is c, y = 1 otherwise y = 0, p is the probability of sample i, the formula is:

[0018]

[0019] Gaussian distance calculation angle loss is used to predict the rotation angle:

[0020] Convert the two rotated rectangles into Gaussian distributions; introduce the center point loss to bring the two distributions closer together so that they have the same center; obtain the Gaussian distribution of the intersection area by multiplying the two Gaussians; convert the three Gaussian distributions back into rotated rectangles, calculate the approximate IOU loss, and replace the original IOU loss function.

[0021] Preferably, the use of depthwise separable convolution to replace the original 3x3 convolution makes the angle prediction and category detection heads lighter and improves the accuracy; the depthwise separable convolution performs channel-by-channel convolution and point-by-point convolution, and then uses 1x1 convolution to merge features;

[0022] Ordinary convolution complexity: C OUT ×K 2 ×H×W

[0023] Depthwise separable convolution complexity: C in ×K 2 ×H×W+C OUT ×1 2 ×H×W

[0024] Cin is the number of input channels, Cout is the number of output channels, and k is the number of convolution kernels.

[0025] Compared with the prior art, the beneficial effects of the present invention are: the rotation detection mechanism based on the port smart brain that can adapt to the target object is used in the method for safely identifying people, machines and cargoes:

[0026] 1. The traditional early warning method is to rely on supervisors to observe. The present invention uses an automatic identification and detection system to replace manual observation.

[0027] 2. The safe operation area of ​​the spreader is constantly changing, and the dangerous area is also changing. If a fixed electronic fence is used, it cannot meet the dynamic needs. The present invention automatically detects the perimeter under the spreader through an algorithm, and there is no need to draw a fence manually;

[0028] 3. The detection method adopts a rotating target detection method. The present invention solves the problem of circumference changes during the continuous rotation of the sling, but at the same time faces complex backgrounds and a variety of slings. At the same time, the rotating target detection method still has challenges in rotating objects with large aspect ratios, dense distribution, and extremely unbalanced categories. The present invention adopts Gaussian distance loss to solve the problems of angle discontinuity and non-regression, interval discontinuity, and square regression.

[0029] 4. The system has high requirements for real-time performance. The present invention optimizes the detection head based on the original architecture and replaces the original ordinary convolution with separable convolution, which greatly reduces the amount of calculation, and the amount of calculation is reduced by 20% BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the detection framework of the present invention;

[0031] Figure 2 It is a schematic diagram of the steps of the method of the present invention;

[0032] Figure 3 It is a schematic diagram of the structure of the detection head of the present invention;

[0033] Figure 4 Schematic diagram of the Gaussian distance loss calculation process of the present invention;

[0034] Figure 5 For the final effect intention of the present invention; DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] See also Figure 1-5 The present invention provides a technical solution: a method for safely identifying people, machines and cargoes based on a rotation detection mechanism of a port smart brain that can adapt to target objects, characterized in that it includes the following steps:

[0037] Step 1: Label sample images of various cargo spreaders and people, obtain labels for each cargo spreader and person, and construct a data set containing labels of spreaders, cargo, and people;

[0038] Step 2: Use the above sample set to improve the yolov8 model for training, and obtain the trained improved yolov8 model as the target detection model. The improved yolov8 model introduces DWConv (depth separable convolution) into the classification detection head to optimize the detection head, improve the recognition rate, and output the result for post-processing;

[0039] Step 3: Input the image to be predicted into the improved target detection model, and output the horizontal box coordinates and angle values, which are the final results.

[0040] The labels of all sample images in step 2 are (c, x, y, w, h, Q), where c is the target category, (x, y) is the center point, w and h are the width and height of the horizontal box, and Q is the angle of rotation of the horizontal box.

[0041] The improved yolov8 model described in step 2 includes an image input part, a backbone network extraction part, a feature fusion part, and a model detection head.

[0042] In the image input part, the image is enhanced through Mosaic data, which supports multiple input sizes. It can automatically adapt to model training and inference at different resolutions, flip, crop, scale, change brightness, saturation, and hue of each image, initialize the anchor scale through K-means, and perform cluster analysis on the real bounding boxes of all annotated data.

[0043] The backbone network extraction part consists of 5 CBS structures, 4 C2F structures, and 1 SPPF structure. The CBS structure consists of a 2D convolution, a BatchNorm2d, and a SiLU activation function. The C2F structure consists of a 1x1 convolution channel number reduced to 1 / 2 of the original, multiple 3x3 convolution kernels to extract features, residual links, and a 1x1 convolution kernel to restore the number of channels of the feature map; the SPPF structure is fused through three 5×5 maximum pooling operations.

[0044] The feature fusion part is composed of FPN and PAN. The outputs P3, P4, and P5 of the feature extraction backbone network are input into the PAN-FPN network structure to fuse feature maps of multiple scales. P5 is fused with P4 after upsampling to obtain F1, F1 is fused with P3 after a C2f layer and upsampling once to obtain T1, T1 is fused with F1 after a convolution layer to obtain F2, F2 is passed through a C2f layer to obtain T2, T2 is fused with P5 after a convolution layer to obtain F3, F3 is passed through a C2f layer to obtain T3, and finally the feature maps of T1, T2, and T3 detection heads are obtained.

[0045] In the detection head part, each detection head consists of three parts: box prediction (two 3x3 convolutions and one 1x1 convolution), category prediction (two DWConv convolutions and three 1x1 convolutions), and angle prediction (newly added angle prediction, consisting of two DWConv convolutions and three 1x1 convolutions).

[0046] The angle loss function is calculated by calculating the Gaussian distribution of the box through Σ, the formula is Σ(h,w,Q-π / 2), and then calculating the Gaussian distance between the boxes to replace the IOU loss function, the formula is: d 2 =lm1-m2l2 2+Tr(Σ1+Σ2 -2(Σ1 / 2 1Σ2Σ1 / 2 1) 1 / 2 ;

[0047] Box loss function calculation, FL(p,q)=-q(qlog(p)+(1-q)log(1-p))q>0or-αp β (log(1-p) q=0;

[0048] Classification loss function, M is the number of categories, y if the true category of sample i is c, y = 1 otherwise y = 0, p is the probability of sample i, the formula is:

[0049]

[0050] Gaussian distance calculation angle loss is used to predict the rotation angle:

[0051] Convert the two rotated rectangles into Gaussian distributions; introduce the center point loss to bring the two distributions closer together so that they have the same center; obtain the Gaussian distribution of the intersection area by multiplying the two Gaussians; convert the three Gaussian distributions back into rotated rectangles, calculate the approximate IOU loss, and replace the original IOU loss function.

[0052] Using depth-wise separable convolution to replace the original 3x3 convolution makes the angle prediction and category detection heads lighter and more accurate. Depth-wise separable convolution performs channel-by-channel convolution and point-by-point convolution, and then uses 1x1 convolution to merge features.

[0053] Ordinary convolution complexity: C OUT ×K 2 ×H×W

[0054] Depthwise separable convolution complexity: C in ×K 2 ×H×W+C OUT ×1 2 ×H×W

[0055] Cin is the number of input channels, Cout is the number of output channels, and k is the number of convolution kernels.

[0056] Working principle: When using the rotation detection mechanism based on the port smart brain that can adapt to the target object in the human, machine and cargo safety identification method, first, a series of tasks are completed. The content not described in detail in this specification belongs to the existing technology known to professional and technical personnel in this field.

[0057] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for safely identifying people, machines and cargoes based on a rotation detection mechanism of a port smart brain that can adapt to the target, characterized in that: The following steps are involved: Step 1: Label sample images of various cargo spreaders and people, obtain labels for each cargo spreader and person, and construct a data set containing labels of spreaders, cargo, and people; Step 2: Use the above sample set to improve the yolov8 model for training, and obtain the trained improved yolov8 model as the target detection model. The improved yolov8 model introduces the classification detection head into the DWConv optimization detection head to improve the recognition rate, and outputs the result for post-processing; Step 3: Input the image to be predicted into the improved target detection model, and output the horizontal box coordinates and angle values, which are the final results.

2. According to claim 1, a method for safely identifying people, machines and cargoes based on the rotation detection mechanism of the port smart brain that can adapt to the target, characterized in that: The labels of all sample images in step 2 are (c, x, y, w, h, Q), where c is the target category, (x, y) is the center point, w and h are the width and height of the horizontal box, and Q is the angle of rotation of the horizontal box.

3. According to claim 1, a method for safely identifying people, machines and cargoes based on a rotation detection mechanism of a port smart brain that can adapt to targets, characterized in that: The improved yolov8 model described in step 2 includes an image input part, a backbone network extraction part, a feature fusion part, and a model detection head.

4. According to claim 3, a method for safely identifying people, machines and cargoes based on a rotation detection mechanism of a port smart brain that can adapt to targets, characterized in that: In the image input part, the image is enhanced through Mosaic data, supports multiple input sizes, can automatically adapt to model training and reasoning at different resolutions, flip, crop, scale, change brightness, saturation, and hue for each image, initialize the anchor scale through K-means, and perform cluster analysis on the real bounding boxes of all annotated data.

5. According to claim 3, a method for safely identifying people, machines and cargoes based on a rotation detection mechanism of a port smart brain that can adapt to targets, characterized in that: The backbone network extraction part is composed of 5 CBS structures, 4 C2F structures, and 1 SPPF structure. The CBS structure is composed of a 2D convolution, a BatchNorm2d, and a SiLU activation function. The C2F structure is composed of a 1x1 convolution channel number reduced to 1 / 2 of the original, multiple 3x3 convolution kernels to extract features, residual links, and a 1x1 convolution kernel to restore the number of channels of the feature map; the SPPF structure is fused through 3 5×5 maximum pooling operations.

6. According to claim 3, a method for safely identifying people, machines and cargoes based on a rotation detection mechanism of a port smart brain that can adapt to targets, characterized in that: The feature fusion part is composed of FPN and PAN. The outputs P3, P4, and P5 of the feature extraction backbone network are input into the PAN-FPN network structure to fuse feature maps of multiple scales. P5 is fused with P4 after upsampling to obtain F1, F1 is fused with P3 after a C2f layer and an upsampling to obtain T1, T1 is fused with F1 after a convolution layer to obtain F2, F2 is fused with a C2f layer to obtain T2, T2 is fused with P5 after a convolution layer to obtain F3, F3 is fused with a C2f layer to obtain T3, and finally the feature maps of T1, T2, and T3 detection heads are obtained.

7. According to claim 3, a method for safely identifying people, machines and cargoes based on a rotation detection mechanism of a port smart brain that can adapt to targets, characterized in that: The detection head part, each detection head consists of three parts: box prediction, category prediction, and angle prediction.

8. According to claim 3, a method for safely identifying people, machines and cargoes based on a rotation detection mechanism of a port smart brain that can adapt to targets, characterized in that: The angle loss function is calculated by calculating the Gaussian distribution of the box through Σ, the formula is Σ(h,w,Q-π / 2), and then calculating the Gaussian distance between the boxes to replace the IOU loss function, the formula is: d 2 =lm1-m2l2 2+Tr(Σ1+Σ2-2(Σ1 / 2 1Σ2Σ1 / 2 1) 1 / 2 ; Box loss function calculation, FL(p,q)=-q(qlog(p)+(1-q)log(1-p))q>0or-αp β (log(1-p)q=0; Classification loss function, M is the number of categories, y if the true category of sample i is c, y = 1 otherwise y = 0, p is the probability of sample i, the formula is: Gaussian distance calculation angle loss is used to predict the rotation angle: Convert the two rotated rectangles into Gaussian distributions; introduce the center point loss to bring the two distributions closer together so that they have the same center; obtain the Gaussian distribution of the intersection area by multiplying the two Gaussians; convert the three Gaussian distributions back into rotated rectangles, calculate the approximate IOU loss, and replace the original IOU loss function.

9. According to claim 3, a method for safely identifying people, machines and cargoes based on a rotation detection mechanism of a port smart brain that can adapt to targets, characterized in that: The use of depthwise separable convolution to replace the original 3x3 convolution makes the angle prediction and category detection heads lighter and improves the accuracy; the depthwise separable convolution performs channel-by-channel convolution and point-by-point convolution, and then uses 1x1 convolution to merge features; Ordinary convolution complexity: C OUT ×K 2 ×H×W Depthwise Separable Convolution Complexity: C in ×K 2 ×H×W+C OUT ×1 2 ×H×W Cin is the number of input channels, Cout is the number of output channels, and k is the number of convolution kernels.

Images