A method for detecting and warning people trapped under a crane

Abstract

The present invention belongs to the field of image recognition application technology, and specifically discloses a method for detecting and warning of personnel stranded under a crane, including: using a camera to obtain a picture of a crane scene to be detected, calibrating the camera and the picture, and obtaining a projection transformation module between the picture plane and the ground; preprocessing the picture to obtain a preprocessed picture; inputting the preprocessed picture into a pre-trained deep neural network model, outputting the position of the personnel, the position of the crane, and the projection point of the center of the crane on the ground; inputting the projection point into the projection transformation module to obtain the range of the dangerous area under the crane; judging whether the position of the personnel is in the dangerous area and issuing an alarm. The beneficial effects of the present invention are: solving safety problems in engineering construction sites or large machinery and equipment factories, so as to realize all-weather and real-time detection of personnel stranded in dangerous areas under cranes, and completing accurate, stable and timely monitoring and early warning operation processing to realize safe production operations.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition application technology, in particular to the field of target key point detection in the field of image recognition, and specifically relates to a method for detecting and warning of personnel stranded under a crane. Background Art

[0002] In construction sites or factories with large machinery and equipment, cranes and hoists are common mechanical equipment.

[0003] Since large objects need to be lifted frequently, the area below them is a dangerous area in the safety management system. People staying or walking in the dangerous area is a huge safety hazard during the operation, and this safety hazard needs to be monitored or inspected in real time.

[0004] The previous manual inspection method, on the one hand, was inefficient and could not meet the needs of modern enterprise production and development; on the other hand, safety hazards posed a threat to the personal safety of inspectors.

[0005] There is an urgent need for an easy-to-deploy camera-based detection method that can automatically detect the dangerous area under the crane and the people in the area in real time to realize automatic supervision and alarm functions.

[0006] Therefore, based on the above-mentioned problem, the present invention provides a method for detecting and giving an alarm when a person is stranded under a crane. Summary of the invention

[0007] Purpose of the invention: The purpose of the present invention is to provide a method for detecting and warning of personnel stranded under a crane, so as to realize all-weather and real-time detection of personnel stranded in dangerous areas under the crane.

[0008] Technical solution: A method for detecting and giving an alarm for people being stranded under a crane of the present invention comprises: using a camera to obtain a picture of a crane scene to be detected, calibrating the camera and the picture, and obtaining a projection transformation module between the picture plane and the ground; preprocessing the picture to obtain a preprocessed picture; inputting the preprocessed picture into a pre-trained deep neural network model, outputting the position of the person, the position of the crane and the projection point of the center of the crane on the ground; inputting the projection point into the projection transformation module to obtain the range of the dangerous area under the crane; judging whether the person is in the dangerous area and giving an alarm.

[0009] The technical solution of the present invention uses a camera to obtain a picture of a crane scene to be detected, including: obtaining a video stream captured by a network camera through a preset interface, converting the video stream into a picture in a 3-channel RGB matrix format, and using the converted picture as the picture of the crane scene to be detected.

[0010] The technical solution of the present invention calibrates the camera and the picture to obtain the projection transformation module between the picture plane and the ground, including: using a calibration plate to calibrate the camera internal parameters to obtain 10 internal parameters, including 5 internal matrix parameters and 5 distortion parameters; calibrating the ground projection transformation of the crane scene picture, first de-distorting the picture, then selecting four fixed points in the scene to obtain the world coordinates and picture coordinates of the four points, and then solving the overdetermined equations based on least squares or singular value decomposition methods to obtain the projection transformation matrix; according to the projection transformation matrix and the internal parameters of the camera, the projection transformation module between the picture plane and the ground plane is obtained.

[0011] In the technical solution, the picture is preprocessed, including: based on a preset area of ​​interest, cropping the area of ​​interest portion of the picture; scaling the cropped picture to a preset width and height; and normalizing the scaled picture.

[0012] In the technical solution, the deep neural network model includes: a slice layer connected in sequence, a convolution unit, a CSP1 unit, a convolution unit, a CSP3 unit, the first branch point, a convolution unit, a CSP3 unit, the second branch point, a convolution unit, an SPP module, a CSP2 unit, a convolution unit, and the third branch point. The third branch point is spliced ​​with the output of the second branch point after the upsampling unit, and then connected to a CSP2 unit, a convolution unit, the fourth branch point, after upsampling, is spliced ​​with the output of the first branch point, and then connected to a CSP2 unit, the fifth branch point, a convolution layer, and the first output layer; the fifth branch point is connected to a convolution unit and then spliced ​​with the fourth branch point, and then connected to a CSP2 unit, the sixth branch point, a convolution layer, and the second output layer; the sixth branch point is connected to a convolution unit and then spliced ​​with the third branch point, and then connected to a CSP2 unit, a convolution layer, and the third output layer.

[0013] In the technical solution, the slicing layer includes: separating and slicing the original 3-channel image into quarters, splicing it into a 12-channel feature map and then connecting it to a convolution unit. The convolution unit includes: a convolution layer, a BN layer, and a Leaky ReLU layer connected in sequence. The CSP1 unit includes: the input is divided into two branches, one of which is a convolution unit, a residual unit, and a convolution layer, wherein the residual unit is a residual unit whose output is added to the input after the input is connected to two convolution units; the other branch is the input connected to only one convolution layer. After the splicing operation, the two branches are connected to the BN layer, the Leaky ReLU layer, and a convolution unit. The CSP3 unit is similar to the CSP1 unit, except that the residual unit is changed from 1 to 3. The SPP module includes: after the input is connected to a convolution unit, the splicing operation is performed after connecting 4 branches, and then a convolution unit is connected. The 4 branches are 1×1, 5×5, 9×9, and 13×13 maximum pooling layers respectively. The CSP2 unit is similar to the CSP1 unit, except that the original residual unit is changed into two sequentially connected convolution units.

[0014] In the technical solution, the projection point is input into the projection transformation module to obtain the range of the dangerous area under the crane, including: dedistorting the coordinates of the projection point to obtain the coordinates without distortion, and then subjecting the coordinates without distortion to projection transformation to obtain the world coordinates of the projection point; according to the preset side length of the square area under the crane, the world coordinates of the four corner points of the square area are obtained; and then the world coordinates of the four corner points are projected and transformed into the distortion-free image coordinates, and then the distortion is added to obtain the coordinates of the four points under the original camera image, that is, the range of the dangerous area under the crane is obtained.

[0015] In the present technical solution, the method of determining whether a person's position is in a dangerous area and issuing an alarm includes: obtaining the center point of the person on the ground, using a positional relationship formula between a straight line and a point to determine whether the center point of the person is within the dangerous area below the crane, and if so, issuing an alarm.

[0016] Compared with the prior art, the beneficial effect of the method for detecting and warning of personnel stranded under a crane of the present invention is that it solves the production safety problem in engineering construction sites or factories with large-scale machinery and equipment, so as to realize all-weather, real-time detection of personnel stranded in dangerous areas under cranes, and can complete accurate, stable and timely monitoring and early warning operation processing to achieve safe production operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a method for detecting and giving an alarm for personnel staying under a crane according to the present invention;

[0018] Figure 2It is a structural schematic diagram of a deep convolutional neural network model of a method for detecting and giving an alarm for personnel trapped under a crane according to the present invention. DETAILED DESCRIPTION

[0019] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments.

[0020] like Figure 1 The flowchart of a method for detecting and giving an alarm for a person trapped under a crane is shown in FIG. 1 , which can be executed by a network camera and an electronic device for running a deep learning algorithm. The method specifically includes the following steps:

[0021] Obtain a crane scene image to be detected. The crane scene image to be detected obtained by the electronic device may be a scene image stored locally in the electronic device, or may be a scene image acquired in real time by an image acquisition module that communicates with the electronic device. The electronic device may include but is not limited to a mobile phone, a tablet computer, a computer, a smart camera, and a server.

[0022] Among them, the method for collecting pictures includes: obtaining the video stream collected by the network camera through a preset interface, converting the video stream into an RGB picture in a 3-channel matrix format, and using the converted picture as the crane scene picture to be detected.

[0023] Among them, the IP Camera (IPC) is a new generation of cameras that is a combination of traditional cameras and network technology. IPC can be used to capture the monitoring area, generate a digital video stream containing the crane scene (such as a video stream in the format of Moving Picture Experts Group (MPEG)), and transmit the video stream containing the crane scene to the electronic device through a preset wired network interface or a wireless network interface. After the electronic device obtains the digital video stream containing the crane scene, it can convert the stream format data sent by the IPC into data in the image matrix format supported by opencv by calling the built-in module of opencv, thereby realizing the acquisition of the image of the crane scene to be detected.

[0024] The camera is calibrated to obtain the projection transformation module between the image plane and the ground. Optionally, the camera can be calibrated with a checkerboard calibration plate to obtain 10 internal parameters, including 5 internal matrix parameters and 5 distortion parameters.

[0025] Camera calibration is a relatively mature problem. Calibration software includes but is not limited to opencv, matlab, etc., which provide a relatively comprehensive calibration, correction and other function interfaces. After obtaining 10 internal parameters, the original camera image can be distorted and corrected based on the theoretical basis of the camera's pinhole imaging model and distortion parameter model, thereby removing the distortion error.

[0026] Calibrate the ground projection transformation of the crane scene image. First, remove the distortion of the image, then select four fixed points in the scene, obtain the world coordinates and image coordinates of the four points, and then solve the overdetermined equations based on least squares or singular value decomposition (SVD) to obtain the projection transformation matrix.

[0027] According to the projection transformation matrix and the internal parameters of the camera, the projection transformation module between the image plane and the ground plane is obtained, so that the coordinates of the points on the image can be matched one by one with the coordinates of the ground plane.

[0028] The RGB image obtained in the first step is preprocessed, including: based on a preset region of interest, cropping the region of interest portion of the image; scaling the cropped image to a preset width and height; and normalizing the scaled image.

[0029] Cropping an image can filter out useless image areas, making the target to be detected occupy a larger proportion of the image, reducing the difficulty of the subsequent deep learning algorithm to identify the target to be detected. The cropped image is scaled to a preset width and height and normalized to adapt to the input width and height of the subsequent algorithm and the input normalization requirements of the algorithm to achieve the best detection accuracy of the algorithm.

[0030] The pre-processed image obtained in step 3 is input into the pre-trained deep convolutional neural network model to output the position of personnel, the position of the crane and the projection point of the center of the crane on the ground. Among them, the deep neural network (DNN) model is a technology in the field of machine learning. It is an algorithm that uses artificial neural networks as the architecture to represent and learn data (such as images, sounds, text, etc.). The convolutional neural network (CNN) is a neural network architecture that is particularly good at extracting image features. It is widely used in image recognition, target detection and other fields and has brought significant breakthroughs.

[0031] The structure of the pre-trained deep neural network model is as follows Figure 2As shown. The input is a preprocessed image, and the output is the position of the personnel, the position of the crane, and the coordinates of the projection point of the center of the crane on the ground. It is an end-to-end target detection neural network with key points, including: a slice layer connected in sequence, a convolution unit, a CSP1 (CrossStage Partial 1) unit, a convolution unit, a CSP3 unit, the first branch point, a convolution unit, a CSP3 unit, the second branch point, a convolution unit, an SPP (Spatial Pyramid Pooling) module, a CSP2 unit, a convolution unit, and the third branch point. The third branch point is spliced ​​with the output of the second branch point through the upsampling unit, and then connected to a CSP2 unit, a convolution unit, the fourth branch point, after upsampling, is spliced ​​with the output of the first branch point, and then connected to a CSP2 unit, the fifth branch point, a convolution layer, connected to the first output layer; the fifth branch point is connected to a convolution unit and then spliced ​​with the fourth branch point, and then connected to a CSP2 unit, the sixth branch point, a convolution layer, connected to the second output layer; the sixth branch point is connected to a convolution unit and then spliced ​​with the third branch point, and then connected to a CSP2 unit, a convolution layer, and connected to the third output layer. The results of each output layer are then processed by non-maximum suppression to obtain the final result, that is, the position of the personnel, the position of the crane, and the projection point of the center of the crane on the ground. The role of non-maximum suppression post-processing is to remove duplicate detection results, extract target detection frames with higher confidence, and improve the accuracy of model output.

[0032] The slicing layer includes: separating and slicing the original 3-channel image into quarters, splicing them into 12-channel feature maps and then connecting a convolution unit. Slicing reduces the length and width to half of the original, reducing the amount of calculation of subsequent convolution layers.

[0033] The convolution unit includes: a convolution layer, a batch normalization layer (BN), and a leaky ReLU layer connected in sequence. After the features are normalized by the BN layer, the features with smaller norms will obtain larger gradients, reducing the risk of gradient explosion. At the same time, the convergence speed of the deep neural network as the network depth increases is guaranteed, and the learning rate of the feature extraction model is accelerated. The leaky ReLU layer can introduce nonlinear factors into the neural network as an excitation function, and can also alleviate the problem of gradient disappearance.

[0034] The CSP1 unit includes: the input is divided into two branches, one of which is a convolution unit, a residual unit, and a convolution layer, wherein the residual unit is a residual unit whose output is added to the input after the input is connected to two convolution units; the other branch is an input connected to only one convolution layer. After the splicing operation, the two branches are connected to the BN layer, the Leaky ReLU layer, and a convolution unit.

[0035] The CSP3 unit is similar to the CSP1 unit, except that the number of residual units is changed from 1 to 3.

[0036] The SPP module includes: after the input is connected to a convolution unit, the four branches are connected and then spliced, and then connected to another convolution unit. The four branches are 1×1, 5×5, 9×9, and 13×13 maximum pooling layers. The maximum pooling layer can retain more feature information than the mean pooling layer. The use of the SPP module is more effective than simply using a maximum pooling layer to increase the reception range of the trunk features, significantly separate important context features, and improve the final accuracy of the neural network at the cost of a smaller amount of calculation.

[0037] The CSP2 unit is similar to the CSP1 unit, except that the original residual unit is changed into two sequentially connected convolution units.

[0038] The splicing of the branch points is based on the PAN (Path Aggregation Network) structure, which can simultaneously convey strong semantic features from top to bottom and strong positioning features from bottom to top, further improving the feature extraction capability.

[0039] The neural network is pre-trained with a preset crane scene dataset, so as to learn the ability to extract features of personnel and cranes. The preset crane scene dataset can be obtained by, but is not limited to, the following methods: collecting images on site, and manually annotating the positions of personnel, crane positions, and projection points of the crane center on the ground, and then dividing the images and annotations into a training set, a validation set, and a test set according to a preset ratio.

[0040] The projection point of the crane center on the ground obtained in step 4 is input into the projection transformation module obtained in step 2 to obtain the scope of the dangerous area under the crane, including: based on the camera intrinsic parameters obtained in step 2, the projection point coordinates are dedistorted to obtain the coordinates without distortion, and then the coordinates without distortion are transformed through the projection transformation matrix to obtain the world coordinates of the projection point; according to the preset side length of the square area under the crane, the world coordinates of the four corner points of the square area are obtained; and then the world coordinates of the four corner points are projected and transformed into the undistorted image coordinates, and the distortion is added to obtain the coordinates of the four points under the original camera image, that is, the scope of the dangerous area under the crane is obtained.

[0041] 6. Determine whether the personnel position is in the danger zone and issue an alarm, including: according to the personnel position obtained in step 4, obtain the coordinates of the center point of the personnel on the ground in the picture, use the position relationship formula of the straight line and the point, and according to the danger zone range obtained in step 5, determine whether the center point of the personnel is within the danger zone range under the crane. If so, issue an alarm.

[0042] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements without departing from the principle of the present invention. These improvements should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for detecting and warning of personnel retention under a crane, characterized in that: The following steps are included: Step 1: Use a camera to obtain a picture of the crane scene to be detected, calibrate the camera and the picture, and obtain a projection transformation module between the picture plane and the ground; Step 2: preprocessing the image to obtain a preprocessed image; Step 3: Input the pre-processed image into a pre-trained deep neural network model, and output the position of the personnel, the position of the crane, and the projection point of the center of the crane on the ground; Step 4: input the projection point into the projection transformation module to obtain the range of the dangerous area under the crane; Step 5: Determine whether the person is in a dangerous area and issue an alarm; The deep neural network model comprises: A slice layer connected in sequence: a convolution unit, a CSP1 unit, a convolution unit, a CSP3 unit, the first branch point, a convolution unit, a CSP3 unit, the second branch point, a convolution unit, an SPP module, a CSP2 unit, a convolution unit, and the third branch point; The third branch point is spliced ​​with the output of the second branch point through the upsampling unit, and then connected to a CSP2 unit, a convolution unit, and the fourth branch point. After upsampling, it is spliced ​​with the output of the first branch point, and then connected to a CSP2 unit, the fifth branch point, a convolution layer, and the first output layer; The fifth branch point is connected to a convolution unit and then spliced ​​with the fourth branch point, and then connected to a CSP2 unit. The sixth branch point, a convolution layer, is connected to the second output layer. The sixth branch point is connected to a convolution unit and then spliced ​​with the third branch point, and then connected to a CSP2 unit, a convolution layer, and the third output layer.

2. A method for detecting and giving an alarm when a person is stranded under a crane according to claim 1, characterized in that: The method of using a camera to obtain a crane scene picture to be detected includes: The video stream collected by the network camera is obtained through a preset interface, the video stream is converted into a picture in a 3-channel RGB matrix format, and the converted picture is used as the crane scene picture to be detected.

3. A method for detecting and warning of personnel retention under a crane according to claim 1, characterized in that: The camera and the image are calibrated to obtain a projection transformation module between the image plane and the ground, including: Use the calibration board to calibrate the camera internal parameters and obtain 10 internal parameters, including 5 internal matrix parameters and 5 distortion parameters; Calibrate the ground projection transformation of the crane scene image. First, dedistort the image, then select four fixed points in the scene, obtain the world coordinates and image coordinates of the four points, and then solve the overdetermined equations based on least squares or singular value decomposition methods to obtain the projection transformation matrix. According to the projection transformation matrix and the internal parameters of the camera, the projection transformation module between the image plane and the ground plane is obtained.

4. A method for detecting and warning of personnel retention under a crane according to claim 1, characterized in that: The preprocessing of the image includes: Based on a preset region of interest, cropping the region of interest portion of the image; Scale the cropped image to the preset width and height; Normalize the scaled image.

5. A method for detecting and warning of personnel retention under a crane according to claim 1, characterized in that: The slicing layer includes: separating and slicing the original 3-channel image into quarters, splicing them into 12-channel feature maps and then connecting them to a convolution unit; The convolution unit includes: a convolution layer, a BN layer, and a Leaky ReLU layer connected in sequence; The CSP1 unit includes: the input is divided into two branches, one of which is a convolution unit, a residual unit, and a convolution layer, wherein the residual unit is a residual unit whose output is added to the input after the input is connected to two convolution units; the other branch is an input connected to only one convolution layer, and the two branches are connected to a BN layer, a Leaky ReLU layer, and a convolution unit after a splicing operation; The CSP3 unit includes: the input is divided into two branches, one of which is a convolution unit, 3 residual units, and a convolution layer, wherein the residual unit is a residual unit whose output is added to the input after the input is connected to two convolution units; the other branch is the input connected to only one convolution layer, and the two branches are connected to the BN layer, the Leaky ReLU layer, and a convolution unit after the splicing operation; The SPP module includes: after the input is connected to a convolution unit, four branches are connected and then spliced, and then another convolution unit is connected, and the four branches are 1×1, 5×5, 9×9, and 13×13 maximum pooling layers respectively; The CSP2 unit includes: the input is divided into two branches, one branch is a convolution unit, a residual unit, and two convolution units connected in sequence, wherein the residual unit is a residual unit whose output is added to the input after the input is connected to two convolution units; the other branch is the input connected to only one convolution layer, and the two branches are connected to the BN layer, the Leaky ReLU layer, and a convolution unit after the splicing operation.

6. A method for detecting and giving an alarm when a person is stranded under a crane according to claim 1, characterized in that: The step of inputting the projection point into the projection transformation module to obtain the range of the dangerous area below the crane includes: The coordinates of the projection point are dedistorted to obtain the coordinates without distortion, and then the coordinates without distortion are projected and transformed to obtain the world coordinates of the projection point; according to the preset side length of the square area under the crane, the world coordinates of the four corner points of the square area are obtained; then the world coordinates of the four corner points are projected and transformed to the undistorted image coordinates, and then the distortion is added to obtain the coordinates of the four points under the original camera image, that is, the scope of the dangerous area under the crane is obtained.

7. A method for detecting and warning of personnel retention under a crane according to claim 1, characterized in that: The determining whether the person is in a dangerous area and issuing an alarm includes: The center point of the person on the ground is obtained, and the position relationship formula of the straight line and the point is used to determine whether the center point of the person is within the dangerous area below the crane. If so, an alarm operation is performed.

Images