Steel structure hoisting equipment and method based on posture image recognition and monitoring

Abstract

The invention provides a steel structure hoisting device and method based on posture image recognition and monitoring. The hoisting device comprises a chassis and a hoisting arm. A camera of the image acquisition unit is used for acquiring a real-time video of a chassis attitude and a real-time video of a suspension arm attitude; the image processing unit comprises a key frame extractor, a memory and a hoisting posture detector, the key frame extractor is set as an image recognition model, the memory is set as key frame storage, the hoisting posture detector is set as a hoisting instance segmentation module, a distortion removal module and a hoisting posture recognition module, and whether hoisting equipment exceeds a set range or not is judged; and the alarm unit is set to send out an alarm signal when the chassis posture or the suspension arm posture exceeds a set range. The posture of the hoisting equipment can be monitored in real time, the problem of overturning caused by the fact that the posture of the hoisting equipment exceeds a reasonable range is avoided, and the hoisting construction level and the safety level are improved.

Description

Technical Field

[0001] The invention relates to the technical field of building construction, and in particular to a steel structure hoisting device and method based on posture image recognition monitoring. Background Art

[0002] The steel structure of large buildings is divided into frame structure and large-span truss structure. The main components include steel columns, steel beams, and roof trusses. The span of large buildings is often more than 100 meters. After the overall lifting of the frame structure is completed, truss sliding construction is required.

[0003] The current hoisting and sliding construction mostly uses crawler hoists. Crawler hoists use chassis walking, and the boom swings and retracts independently. The boom does not have a separate power source, and obtains power from the main pump of the main engine through a hydraulic oil pipe. The chassis and boom drive can be linked or single-acting, and can be freely switched in different operating modes. During the construction process, the crawler hoist can be installed while walking, which is highly efficient. However, in the existing steel structure installation process, the driver of the hoisting equipment judges the status of the hoisting equipment based on experience. If there is a slight mistake, once the posture of the boom and hoisting equipment exceeds the reasonable range, safety accidents are likely to occur. In the face of the development of automation and intelligence of devices, designing a hoisting posture recognition and control device and method for large buildings can effectively increase the level of automation and intelligence of hoisting, the convenience of construction organization and management, and safety. Summary of the invention

[0004] In view of the problem that during the existing steel structure installation process, the driver judges the status of the lifting equipment based on experience, and the boom and the lifting equipment are prone to exceed the reasonable range and overturning accidents are prone to occur, the present invention provides a steel structure lifting equipment and method based on posture image recognition monitoring, so as to achieve real-time monitoring of the posture of the lifting equipment, so as to avoid the posture of the lifting equipment exceeding the reasonable range and not being discovered in time, causing collision and overturning problems.

[0005] The technical solution adopted by the present invention is a steel structure hoisting method based on posture image recognition monitoring, which intends to use a trained neural network model as a posture detector and judge the normal situation of the posture of the boom and hoisting equipment within the monitoring range through image processing technology, including:

[0006] Step 1: Use the hoisting camera to collect videos of the hoisting construction site and mark the postures of the crane arm and hoisting equipment in the video key frames.

[0007] Step 1.1, use three sets of cameras to monitor the hoisted chassis and boom to obtain real-time video;

[0008] Step 1.2, divide the real-time video into fixed frequencies;

[0009] Step 1.3, select 1 to 2 frames of clear boom and chassis images in each group of divided real-time videos and mark them.

[0010] Step 1.4, manually segment the postures of the boom and chassis in the marked image, and mark whether the postures are normal or not.

[0011] Step 2: Use the labeled dataset to train the video key frame separation model and output the key frames in the video.

[0012] Step 2.1, use the backbone network 1ResNet50 to extract video features;

[0013] Step 2.2, use the classification head to classify the key frames in the video;

[0014] Step 2.3, output the image classified as a key frame.

[0015] Step 3: Input the key frame image into the hoisting posture detection model. The hoisting segmentation module of the hoisting posture detection model performs instance segmentation on the boom and hoisting equipment, and outputs the regional segmentation results of the boom and hoisting equipment.

[0016] Step 3.1, use the backbone network 2ResNet50 to extract features from the key frame image;

[0017] Step 3.2: Use the graph reasoning attention module to locate the position of the boom and chassis;

[0018] Step 3.3, use the decoder to perform pixel-level segmentation on the detailed positions of the boom and chassis;

[0019] In step 3.4, the error generated by the segmentation is back-propagated to adjust the parameters of the graph inference attention and decoder.

[0020] Step 4: Use the camera internal parameters to dedistort the segmented area, input the dedistorted image into the hoisting posture detection module, and output the detection result.

[0021] Step 4.1, multiply the segmentation result with the original image to obtain an image containing the boom and chassis.

[0022] Step 4.2, combined with the camera's internal parameters, use the dedistortion algorithm to operate on the segmented image and output a normal viewing angle image.

[0023] Step 5: Compare the annotation results with the detection results and adjust the parameters of the lifting posture detection module.

[0024] Step 5.1, use the input backbone network 3ResNet50 to extract features from the dedistorted image;

[0025] Step 5.2, use the classification head to classify the normal posture of the image;

[0026] In step 5.3, the loss caused by misclassification is back-propagated to adjust the parameters of the classification head.

[0027] Step 6: The parameters of the lifting posture detection model are frozen immediately after training.

[0028] Step 6.1, after training on the manually annotated dataset, the parameters of the model are fixed;

[0029] Step 6.2, deploy the fixed parameters to the applied platform.

[0030] Step 7: After loading the model parameters, use the hoisting posture detection model to analyze the real-time video of the camera and repeat steps 2, 3, and 4.

[0031] In order to achieve the above-mentioned object, in the first aspect, the present invention provides a steel structure hoisting equipment based on posture image recognition monitoring. The hoisting equipment includes a chassis and a boom, and also includes an image acquisition unit, an image processing unit and an alarm unit, wherein the image acquisition unit is connected to the input end of the image processing unit, and the alarm unit is connected to the output end of the image processing unit; the image acquisition unit includes a camera, and the camera is configured to acquire a real-time video of the chassis posture and a real-time video of the boom posture; the image processing unit includes a key frame identifier, a memory and a hoisting posture detector, wherein the key frame identifier is configured to be a neural network model ResNet38 for performing key frame selection on the video acquired by the image acquisition unit, the memory is configured to store the key frame image of the chassis posture and the key frame image of the boom posture, the hoisting posture detector is configured to receive the image in the memory, and use the hoisting segmentation module GRBN to separate the boom and the chassis, and after dedistortion, the hoisting posture detection module VGG determines whether the chassis and the boom are beyond the set range; the alarm unit includes an alarm, and the alarm unit is configured to send an alarm signal when the chassis posture or the boom posture exceeds the set range.

[0032] Optionally, the camera includes a first camera located at the top of the boom and a second camera located on the middle outer side of the boom, and the first camera and the second camera are used to obtain real-time video of the boom posture, and the boom posture includes the rotation angle and height increase and decrease of the boom.

[0033] Optionally, the camera further includes a third camera arranged on the chassis, and the third camera is used to obtain a real-time video of the chassis posture, wherein the chassis posture includes the position of the chassis.

[0034] Optionally, in the image processing unit, a key frame identifier selects the videos of the three cameras indiscriminately, selects one key frame from each camera per unit time, and the time interval needs to be set manually, which is 0.5s by default.

[0035] Optionally, the recognition result of the image processing unit is stored in a memory.

[0036] Optionally, the key frame images of the chassis and the boom stored in the memory are automatically stored after the lifting posture detector has finished reading them.

[0037] Optionally, the steel structure lifting equipment based on posture image recognition monitoring also includes a posture control unit, which is connected to the output end of the image processing unit. The posture control unit is used to control the chassis posture and the boom posture. The chassis posture control includes chassis position control, and the boom posture control includes boom rotation angle control and height lifting control.

[0038] In order to achieve the above object, in a second aspect, the present invention provides a steel structure hoisting method based on posture image recognition monitoring, comprising:

[0039] S1. Video acquisition: The image acquisition unit 1 acquires real-time video of the chassis posture and the boom posture of the crawler crane;

[0040] S2. Key frame selection: The key frame identifier 21 of the image processing unit 2 takes the real-time video of the chassis posture and the real-time video of the boom posture input by the image acquisition unit 1 as input, selects the key frames appearing in the video using a pre-trained neural network, and outputs the key frame images in the video;

[0041] S3. Key frame storage: The video key frames identified by the image processing unit 21 are stored in the memory 22 as input to the hoisting posture detector 23. The memory also contains the internal parameters of the camera in the image processing unit;

[0042] S4. Posture recognition: If the image of the chassis posture or the real-time image of the boom posture is judged to be in an abnormal state, the alarm is triggered; if the real-time image of the chassis posture and the real-time image of the boom posture are considered to be in a normal state, return to step S1 and continue to acquire real-time images.

[0043] Optionally, the steel structure hoisting method based on posture image recognition monitoring also includes:

[0044] S3a memory image storage and transfer: the image processing unit 2 memory 22 of the key frame extractor 21 outputs the image in accordance with the image acquisition unit 1 in the order of the camera to sequentially write to the memory, after writing and sequentially passed to the lifting attitude detector 23;

[0045] S4a. Image processing of the hoisting posture detector: The hoisting posture detector receives the image transmitted by the memory, and after processing and dedistortion by the hoisting instance segmentation module, the normal image of the boom and chassis posture is obtained and can be stored in the memory as a log.

[0046] Optionally, the steel structure hoisting method based on posture image recognition monitoring also includes:

[0047] S7. Posture control: the posture control unit controls the chassis posture and the boom posture until the real-time image of the chassis posture and the real-time image of the boom posture are both within the set range.

[0048] The present invention provides a steel structure hoisting equipment based on posture image recognition monitoring. During the movement of the chassis and the boom, the camera of the image acquisition unit acquires the real-time image of the chassis posture and the real-time image of the boom posture. The image processing unit receives the real-time image of the chassis posture and compares it with the set range image to determine whether the chassis posture exceeds the set range, receives the real-time image of the boom posture and compares it with the set range image to determine whether the boom posture exceeds the set range; when the chassis posture or the boom posture exceeds the set range, the alarm of the alarm unit sends an alarm signal. Thus, the posture of the hoisting equipment is monitored in real time and controlled within a reasonable range, avoiding the problem of overturning due to the posture of the hoisting equipment exceeding the reasonable range, and improving the hoisting construction level and safety level. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0050] Figure 1 It is a system configuration diagram of the first preferred embodiment of the steel structure hoisting equipment based on posture image recognition and monitoring provided by the present invention.

[0051] Figure 2 It is a structural schematic diagram of the image acquisition unit.

[0052] Figure 3 It is a system configuration diagram of the second preferred embodiment of the steel structure hoisting equipment based on posture image recognition and monitoring provided by the present invention.

[0053] Figure 4 It is a flowchart of a preferred embodiment of the steel structure hoisting method based on posture image recognition and monitoring provided by the present invention.

[0054] Figure 5It is a flowchart of the second preferred embodiment of the steel structure hoisting method based on posture image recognition and monitoring provided by the present invention.

[0055] Figure 6 It is a flowchart of the third preferred embodiment of the steel structure hoisting method based on posture image recognition and monitoring provided by the present invention.

[0056] Figure 7 The process flow chart of the embodiment of the steel structure hoisting method based on posture image recognition monitoring is as follows: Figure 1 .

[0057] Figure 8 The process flow chart of the embodiment of the steel structure hoisting method based on posture image recognition monitoring is as follows: Figure 2 .

[0058] In the figure, 1 is an image acquisition unit, 11 is a first camera, 12 is a second camera, 13 is a third camera; 2 is an image processing unit, 21 is a key frame identifier, 22 is a memory, 23 is a hoisting posture detector; 3 is an alarm unit, 31 is an alarm; 4 is a posture control unit. DETAILED DESCRIPTION

[0059] The steel structure of large buildings is divided into frame structure and large-span truss structure. The main components include steel columns, steel beams, and roof trusses. The span of large buildings is often more than 100 meters. After the overall hoisting of the frame structure is completed, the truss sliding construction needs to be carried out. The current hoisting and sliding construction mostly use crawler hoists. During the construction process, crawler hoists can be installed while walking, which is highly efficient. However, in the existing steel structure installation process, the driver of the hoisting equipment judges the status of the hoisting equipment based on experience. If you are not careful, once the posture of the boom and hoisting equipment exceeds the reasonable range, it is easy to overturn.

[0060] To this end, the present invention provides a steel structure hoisting device and method based on posture image recognition monitoring. The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use.

[0061] Figure 1 1 is a system configuration diagram of the first preferred embodiment of the steel structure hoisting equipment based on posture image recognition monitoring provided by the present invention. Figure 1As shown, in some preferred embodiments, the present invention provides a steel structure hoisting equipment based on posture image recognition monitoring, the hoisting equipment includes a chassis and a boom, includes an image acquisition unit 1, an image processing unit 2 and an alarm unit 3, the image acquisition unit 1 is connected to the input end of the image processing unit 2, and the alarm unit 3 is connected to the output end of the image processing unit 2; the image acquisition unit 1 includes a camera, the camera is configured to acquire a real-time image of the chassis posture and a real-time image of the boom posture; the image processing unit 2 includes a key frame identifier 21, a memory 22 and a hoisting posture detector 23, the key frame identifier 21 is configured to receive the chassis posture The real-time video of the state is output and the key frames in the video are stored in the memory. The memory 22 is configured to store the key frames and internal parameters of the video collected by the cameras 1, 2, and 3 in the image acquisition unit 1. The hoisting posture detector is configured to be an image instance segmentation module, a dedistortion module, and a hoisting posture recognition module. After the key frames in the memory 22 are instance segmented and dedistorted, the hoisting posture recognition module determines whether the chassis posture and the boom posture are normal, and the hoisting detector outputs the detection result; the alarm unit 3 includes an alarm 31, and the alarm unit 3 is configured to send an alarm signal when the chassis posture or the boom posture exceeds the set range. In the specific implementation process, the image acquisition unit 1, the image processing unit 2, and the alarm unit 3 are an integrated control system, such as a computer, an industrial computer, and an artificial intelligence system, which can realize an automated control system for image comparison, analysis, recognition, judgment, learning, and understanding.

[0062] Compared with the prior art, the steel structure hoisting equipment based on posture image recognition monitoring provided by the present invention comprises a chassis and a boom; the camera of the image acquisition unit is configured to acquire real-time video of the chassis posture and real-time video of the boom posture; the image processing unit comprises a key frame identifier, a memory and a hoisting posture detector, the key frame identifier is configured to be a backbone network ResNet50 for selecting key frames of the video acquired by the image acquisition unit, the memory is configured to store key frame images of the chassis posture and key frame images of the boom posture, the hoisting posture detector is configured to receive images in the memory, separate the boom and the chassis using a hoisting segmentation module, and after de-distortion, the backbone network ResNet50 of the hoisting posture detection module determines whether the chassis and the boom are beyond the set range; the alarm unit is configured to send an alarm signal when the chassis posture or the boom posture is beyond the set range. The posture of the hoisting equipment can be monitored to avoid the problem of overturning caused by the posture of the hoisting equipment exceeding the reasonable range, thereby improving the level of hoisting construction and safety.

[0063] Figure 2 It is a schematic diagram of the structure of the image acquisition unit. Figure 2As shown, in some preferred embodiments, the camera includes a first camera 11 located at the top of the boom, a second camera 12 located at the middle and outer side of the boom, and a third camera 13 arranged on the chassis. The first camera 11 and the second camera 12 are used to obtain real-time images of the boom posture, and the boom posture includes the rotation angle and height lifting of the boom. The third camera 13 is used to obtain real-time images of the chassis posture, and the chassis posture includes the position of the chassis. By installing the first camera, the second camera and the third camera on the top, the side and the chassis of the boom respectively, the posture, hoisting progress, sliding travel, etc. of the hoisting equipment can be effectively monitored, thereby improving the monitoring range and monitoring accuracy. Preferably, the camera can be a video camera, a visual detector, or an ultrasonic detector.

[0064] Figure 3 1 is a system configuration diagram of a second preferred embodiment of a steel structure hoisting device based on posture image recognition monitoring provided by the present invention. Figure 3 As shown, in some preferred embodiments, the image processing unit 2 also includes a key frame identifier, a memory and a lifting posture detector, the key frame identifier is set to a neural network model ResNet38 for selecting key frames of the video acquired by the image acquisition unit, the memory is set to store the key frame image of the chassis posture and the key frame image of the boom posture, the lifting posture detector is set to receive the image in the memory, use the lifting segmentation module GRBN to separate the boom and the chassis, and after de-distortion, the lifting posture detection module VGG determines whether the chassis and the boom are beyond the set range. In this embodiment, the image processor 23 performs instance segmentation and de-distortion on the real-time image of the chassis posture and the real-time image of the boom posture, so that the information of the chassis posture and the boom posture can be obtained more accurately, and the invalid information in the image recognition can be eliminated, and the amount of calculation in the image recognition process can be reduced, thereby reducing the amount of calculation of the image processing unit, improving the processing speed, and being able to more quickly discover the problem that the chassis posture and the boom posture are beyond the set range, more timely avoid the occurrence of the overturning problem, and improve the safety of the equipment operation.

[0065] like Figure 3As shown, in some preferred embodiments, the steel structure hoisting equipment based on posture image recognition monitoring also includes a posture control unit 4, which is connected to the output end of the image processing unit 2. The posture control unit 4 is used to control the chassis posture and the boom posture. The chassis posture control includes the chassis position control, and the boom posture control includes the boom rotation angle control and the height lifting control. In this embodiment, the equipment is connected to the posture control unit 4 by the image processing unit 2, and whether the chassis posture and the boom posture exceed the set range is obtained in time. When the chassis posture and the boom posture exceed the set range, the posture control unit 4 is used to adjust the chassis posture and the boom posture to within the set range in time, so as to actively avoid the occurrence of overturning problems, without the need for driver operation, so that the operation level and safety of the equipment are effectively improved. In the specific implementation, the posture control unit is an automated control device such as a computer or PLC controller or an industrial computer. The posture control unit and the control subject that restores the abnormal posture of the boom or chassis after the alarm device sounds should have sufficient plans for various situations of abnormal boom or chassis status. When the boom posture is abnormal, the extension of the boom should be stopped immediately, the movement of the hoisted object should be stopped, and the normal direction of the boom posture should be adjusted until the alarm stops.

[0066] like Figure 4 As shown above Figure 3 The key frame identifier 21 is the backbone network 1 ResNet50, which inputs the video frame and outputs a clear picture of the boom and chassis. The lifting posture detector 23 is composed of the backbone network 2 ResNet50, the graph reasoning attention module, the dedistortion, and the backbone network 3 ResNet50. Among them, the number of video frames is determined by the frequency of the camera, and the key frames only need to be extracted according to the frequency during the operation of the boom and chassis, and one image is extracted from each video as a key frame. The key frame is input into the backbone network 2 of the lifting detector, and then the features extracted by the backbone network are passed to the graph reasoning attention module to locate the position contour of the boom and chassis, output the segmentation results of the boom and chassis, and obtain the posture of the boom and chassis under the standard perspective through the dedistortion algorithm. This image is input into the backbone network 3 for feature sorting and analysis, and the output is normal or abnormal.

[0067] like Figure 5 As shown in the figure, the design of the graph reasoning attention module in the lifting posture detector is as follows. Given a feature graph G = (V, E), A is the adjacency matrix of the graph, V is the point set, and E is the edge set, so the corresponding Lapalace matrix can be obtained D is the degree matrix of the graph nodes, I n is the identity matrix, According to the definition, L is a real symmetric matrix, and L can be orthogonalized, so L = UΛU T, U is a set of orthogonal bases, Λ is the eigenvalue of the matrix, where after Fourier transform, Chebynet transform and low-order approximation definition of GCN, the definition of graph convolution can be converted into the following form:

[0068] Gconv(x)=U(I-Λ)U T x (1)

[0069] Given input feature X∈R C×H×W ,Y∈R C×H×W , after channel dimensionality reduction, the features are aggregated into a single channel, as shown in the formula, where is torch.mul(), and sigmoid is torch.Sigmoid().

[0070]

[0071] After flatten, conv1d, and reshape, the state transfer matrix is ​​formed, which is the similarity matrix required for the graph convolution operation. This matrix is ​​multiplied with the input encoder feature map, and the final output result is obtained after multiple stacked graph convolution modules.

[0072] In summary, the feature graph can be viewed as follows after being converted by the graph reasoning module:

[0073] X'=X*Conv1d(Flatten(map)) (3)

[0074] The feature map is constantly updated, and the stacking of the graph reasoning module is here. X' is used as the input of formula 2 to obtain the updated map and X. In this way, after several cycles, the weights of the output graph nodes tend to be stable, which is conducive to the model's grasp of the overall image and improves the stability of the lifting posture detector.

[0075] Figure 6 1 is a flowchart of a preferred embodiment of the steel structure hoisting method based on posture image recognition and monitoring provided by the present invention. Figure 4 As shown, in some preferred embodiments, the present invention provides a steel structure hoisting method based on posture image recognition monitoring, comprising:

[0076] S1. Video acquisition: The image acquisition unit 1 acquires real-time video of the chassis posture and the boom posture of the crawler crane;

[0077] S2. Key frame selection: The key frame identifier 21 of the image processing unit 2 takes the real-time video of the chassis posture and the real-time video of the boom posture input by the image acquisition unit 1 as input, selects the key frames appearing in the video using a pre-trained neural network, and outputs the key frame images in the video;

[0078] S3. Key frame storage: The video key frames identified by the key frame identifier 21 of the image processing unit are stored in the memory 22 as input to the hoisting posture detector 23. The memory also contains the internal parameters of the camera in the image processing unit;

[0079] S4. Posture recognition: The key frame is sent to the hoisting instance segmentation module in the hoisting posture detector for processing, the positions of the boom and the hoisting equipment in the picture are calibrated, and the images of the key positions are extracted; the boom and the hoisting equipment are dedistorted using the camera internal parameters in the memory 22 to obtain the image under normal viewing angle; the dedistorted image is input into the hoisting posture recognition module, and the posture of the boom and the chassis is judged to be out of range through neural network calculation.

[0080] S5 alarm: If the contour of the real-time image of the chassis posture exceeds the contour of the set range or the contour of the real-time image of the boom posture exceeds the contour of the set range, the alarm 31 is triggered;

[0081] S6: If the contour of the real-time image of the chassis posture exceeds the contour of the set range and the contour of the real-time image of the boom posture does not exceed the set range, return to step S1 to continue acquiring real-time images.

[0082] Compared with the prior art, the steel structure hoisting method based on posture image recognition monitoring provided by the present invention, during the movement of the chassis and the boom, the camera of the image acquisition unit acquires the real-time video of the chassis posture and the real-time video of the boom posture, and the image processing unit receives the real-time image of the chassis posture and the real-time image of the boom posture to determine whether the hoisting posture exceeds the set range; when the chassis posture or the boom posture is abnormal, the alarm of the alarm unit sends an alarm signal. Thus, the posture of the hoisting equipment is monitored in real time and controlled within a reasonable range, avoiding the problem of overturning due to the posture of the hoisting equipment exceeding the reasonable range, and improving the hoisting construction level and safety level.

[0083] Figure 7 1 is a flowchart of a second preferred embodiment of the steel structure hoisting method based on posture image recognition monitoring provided by the present invention. Figure 7 As shown, in some preferred embodiments, the steel structure hoisting method based on posture image recognition monitoring provided by the present invention also includes:

[0084] S3a memory image storage and transfer: the image processing unit 2 memory 22 of the key frame extractor 21 outputs the image in accordance with the image acquisition unit 1 in the order of the camera to sequentially write to the memory, after writing and sequentially passed to the lifting attitude detector 23;

[0085] S4a. Hoisting posture detection: The image in the memory 22 is input into the hoisting posture detector 23 to determine whether the arm posture and chassis posture are out of range; the normal image of the arm and chassis posture and the recognition result can be stored in the memory as a log.

[0086] In this embodiment, by writing the key frames into the memory in sequence, the information of the chassis posture and the boom posture at the same time can be obtained more accurately, thereby improving the recognition accuracy. In addition, the normal images and recognition results of the boom and chassis posture are stored, which is beneficial to the evaluation and maintenance of the image processing unit.

[0087] Figure 8 : is a flowchart of the third preferred embodiment of the steel structure hoisting method based on posture image recognition monitoring provided by the present invention. As shown in the figure, in some preferred embodiments, the steel structure hoisting method based on posture image recognition monitoring provided by the present invention also includes:

[0088] S7. Posture adjustment: the posture control unit 4 controls the chassis posture and the boom posture until the real-time image of the chassis posture and the real-time image of the boom posture are both within the set range.

[0089] In this embodiment, the attitude control unit 4 timely adjusts the chassis attitude and the boom attitude to within the set range without the need for driver operation, which can actively avoid overturning problems and improve the safety of the equipment.

[0090] The above specific implementation methods are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that the technical solutions of the invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A steel structure hoisting method based on posture image recognition monitoring, characterized in that: The hoisting method uses a trained neural network model as a posture detector and uses image processing technology to determine the normal posture of the boom and hoisting equipment within the monitoring range, including: Step 1: Use the hoisting camera to collect the video of the hoisting construction site and mark the posture of the crane arm and hoisting equipment in the key frames of the video; Step 1.1, use three sets of cameras to monitor the hoisted chassis and boom to obtain real-time video; Step 1.2, divide the real-time video into fixed frequencies; Step 1.3, select 1-2 frames of clear boom and chassis images in each group of divided real-time videos and mark them; Step 1.4, manually segment the postures of the boom and chassis in the marked image, and mark whether the postures are normal or not; Step 2: Use the labeled dataset to train the video key frame separation model and output the key frames in the video; Step 2.1, use backbone network 1 ResNet50 to extract video features; Step 2.2, use the classification head to classify the key frames in the video; Step 2.3, output the image classified as a key frame; Step 3: Input the key frame image into the hoisting posture detection model. The hoisting segmentation module of the hoisting posture detection model performs instance segmentation on the boom and the hoisting equipment, and outputs the regional segmentation results of the boom and the hoisting equipment. Step 3.1, use backbone network 2 ResNet50 to extract features from the key frame image; Step 3.2: Use the graph reasoning attention module to locate the position of the boom and chassis; Step 3.3, use the decoder to perform pixel-level segmentation on the detailed positions of the boom and chassis; Step 3.4, the error generated by segmentation is back-propagated to adjust the parameters of the graph inference attention and decoder; Step 4: Use the camera internal parameters to dedistort the segmented area, input the dedistorted image into the hoisting posture detection module, and output the detection result; Step 4.1, multiply the segmentation result with the original image to obtain an image containing the boom and chassis. Step 4.2, combining the camera's internal parameters, using the dedistortion algorithm to calculate the segmented image and output a normal viewing angle image; Step 5: Compare the annotation results with the detection results and adjust the parameters of the lifting posture detection module; Step 5.1, use the input backbone network 3 ResNet50 to extract features from the dedistorted image; Step 5.2, use the classification head to classify the normal posture of the image; Step 5.3, the loss formed by misclassification is back-propagated to adjust the parameters of the classification head; Step 6: The parameters of the lifting posture detection model are frozen immediately after training; Step 6.1, after training on the manually annotated dataset, the parameters of the model are fixed; Step 6.2, deploy the fixed parameters to the applied platform; Step 7: After loading the model parameters, use the hoisting posture detection model to analyze the real-time video of the camera and repeat steps 2, 3, and 4.

2. A steel structure hoisting device based on posture image recognition monitoring that implements the method as claimed in claim 1, characterized in that: The hoisting equipment includes a chassis and a boom, and also includes an image acquisition unit, an image processing unit and an alarm unit, wherein the image acquisition unit is connected to an input end of the image processing unit, and the alarm unit is connected to an output end of the image processing unit; the image acquisition unit includes a camera, and the camera is configured to acquire a real-time video of the chassis posture and a real-time video of the boom posture; the image processing unit includes a key frame identifier, a memory and a hoisting posture detector, wherein the key frame identifier is configured to be a neural network model for performing key frame selection on the video acquired by the image acquisition unit, the memory is configured to store key frame images of the chassis posture and key frame images of the boom posture, and the hoisting posture detector is configured to receive images in the memory, separate the boom and the chassis using a hoisting segmentation module GRBN, and after dedistortion, the hoisting posture detection module VGG determines whether the chassis and the boom are beyond a set range; the alarm unit includes an alarm, and the alarm unit is configured to send an alarm signal when the chassis posture or the boom posture is beyond a set range.

3. The steel structure hoisting equipment based on posture image recognition and monitoring according to claim 2 is characterized in that: The camera includes a first camera located at the top of the boom and a second camera located at the middle and outer side of the boom. The first camera and the second camera are used to obtain real-time video of the boom posture, and the boom posture includes the rotation angle and height increase and decrease of the boom.

4. The steel structure hoisting equipment based on posture image recognition and monitoring according to claim 2 is characterized in that: The camera also includes a third camera disposed on the chassis, the third camera being used to obtain a real-time video of the chassis posture, the chassis posture including the position of the chassis; In the image processing unit, the key frame identifier selects the videos of the three cameras indiscriminately, and selects one key frame from each camera per unit time. The time interval needs to be set manually, and the default value is 0.5s.

5. The steel structure hoisting equipment based on posture image recognition and monitoring according to claim 2 is characterized in that: The recognition result of the image processing unit is stored in a memory.

6. The steel structure hoisting equipment based on posture image recognition monitoring according to claim 2 is characterized in that: The key frame images of the chassis and the boom stored in the memory are automatically stored after the lifting posture detector has finished reading.

7. The steel structure hoisting equipment based on posture image recognition and monitoring according to claim 2 is characterized in that: The steel structure lifting equipment based on posture image recognition monitoring also includes a posture control unit, which is connected to the output end of the image processing unit. The posture control unit is used to control the chassis posture and the boom posture. The chassis posture control includes chassis position control, and the boom posture control includes boom rotation angle control and height lifting control.

8. The steel structure hoisting equipment based on posture image recognition and monitoring according to claim 2, characterized in that: The implementation method of the device includes: S1 video acquisition: the image acquisition unit acquires real-time video of the chassis posture of the crawler crane and the real-time video of the boom posture; S2. Key frame selection: The key frame recognizer of the image processing unit takes the real-time video of the chassis posture and the real-time video of the boom posture input by the image acquisition unit as input, uses a pre-trained neural network to select key frames appearing in the video, and outputs the key frame images in the video; S3 key frame storage: the image processing unit to identify the video key frame stored in the memory as the input of the hoisting posture detector; the memory also contains the image processing unit in the camera internal parameters; S4. Posture recognition: If the image of the chassis posture or the real-time image of the boom posture is determined to be in an abnormal state, an alarm is triggered; if the real-time image of the chassis posture and the real-time image of the boom posture are considered to be in a normal state, return to step S1 to continue acquiring real-time images.

9. The steel structure hoisting equipment based on posture image recognition and monitoring according to claim 8, characterized in that: The steel structure hoisting method based on posture image recognition monitoring also includes: Image storage and transmission in memory: The memory of the image processing unit writes the images output by the key frame extractor into the memory in sequence according to the order of the cameras in the image acquisition unit, and transmits them to the hoisting posture detector in sequence after writing; Image processing of the crane attitude detector: The crane attitude detector receives the image transmitted by the memory. After processing and dedistortion by the crane instance segmentation module, the normal image of the crane arm and chassis attitude is obtained and stored in the memory as a log.

10. The steel structure hoisting equipment based on posture image recognition and monitoring according to claim 8, characterized in that: The steel structure hoisting method based on posture image recognition monitoring also includes: Posture control: The posture control unit controls the chassis posture and the boom posture until the real-time image of the chassis posture and the real-time image of the boom posture enter the set range.

Images