Steel structure hoisting equipment and method based on posture image recognition monitoring

Abstract

The application provides a steel structure hoisting equipment and method based on posture image recognition monitoring, the hoisting equipment comprises a chassis and a lifting arm; a camera of an image acquisition unit is arranged to acquire real-time videos of the posture of the chassis and the posture of the lifting arm; an image processing unit comprises a key frame extractor, a memory and a hoisting posture detector, the key frame extractor is arranged as an image recognition model, the memory is arranged as key frame storage, the hoisting posture detector is arranged as a hoisting instance segmentation module, a distortion removal module and a hoisting posture recognition module, and whether the hoisting equipment exceeds a set range is determined; an alarm unit is arranged to send an alarm signal by an alarm when the posture of the chassis or the posture of the lifting arm exceeds the set range. The posture of the hoisting equipment can be monitored, the problem of overturning caused by the posture of the hoisting equipment exceeding a reasonable range is avoided, and hoisting construction level and safety level are improved.

Description

Technical Field

[0001] This invention relates to the field of building construction technology, and in particular to a steel structure hoisting equipment and method based on posture image recognition monitoring. Background Technology

[0002] The steel structure of large buildings is divided into frame structure and large-span truss structure. The main components include steel columns, steel beams and roof trusses. The span of large buildings is often more than 100 meters. After the frame structure is hoisted as a whole, truss sliding construction is required.

[0003] Current hoisting and sliding construction methods primarily utilize crawler-mounted hoisting machines. These machines employ a chassis-based walking mechanism and a separately driven boom for swinging and extension. The boom itself lacks a dedicated power source, drawing power from the main hydraulic pump via hydraulic lines. The chassis and boom can operate in tandem or independently, freely switching between different operating modes. During construction, crawler-mounted hoisting machines can move and install simultaneously, resulting in high efficiency. However, in current steel structure installation processes, the operator relies on experience to judge the hoisting equipment's condition. Slight errors can easily lead to accidents if the boom or hoisting equipment's posture deviates from the reasonable range. In the face of the increasing automation and intelligence of equipment, designing a hoisting posture recognition and control device and method for large-scale construction projects would effectively increase the level of automation and intelligence in hoisting, improve the convenience of construction organization and management, and enhance safety. Summary of the Invention

[0004] In the existing steel structure installation process, drivers rely on experience to judge the state of lifting equipment, which can easily lead to overturning accidents due to the boom and lifting equipment exceeding reasonable limits. This invention provides a steel structure lifting equipment and method based on posture image recognition monitoring to achieve real-time monitoring of the posture of the lifting equipment, thereby avoiding the failure to detect collisions and overturning problems caused by the posture of the lifting equipment exceeding reasonable limits in time.

[0005] The technical solution adopted in this invention is a steel structure hoisting method based on posture image recognition monitoring. It proposes to use a trained neural network model as a posture detector and to determine the normal posture of the boom and hoisting equipment within the monitoring range through image processing technology, including:

[0006] Step 1: Use a hoisting camera to capture video of the hoisting site construction and annotate the posture of the boom and hoisting equipment in the key frames of the video.

[0007] Step 1.1: Use three sets of cameras to monitor the hoisted chassis and boom and obtain real-time video.

[0008] Step 1.2: Divide the real-time video into segments according to a fixed frequency;

[0009] Step 1.3: Select 1-2 clear frames of the boom and chassis from each group of divided real-time videos and mark them.

[0010] Step 1.4: Manually segment the posture of the boom and chassis in the marked image and mark whether the posture is normal or not.

[0011] Step 2: Train a video keyframe separation model using the labeled dataset and output the keyframes in the video.

[0012] Step 2.1: Use the backbone network 1ResNet50 to extract video features;

[0013] Step 2.2: Use the classification head to classify the keyframes in the video;

[0014] Step 2.3: Output the images classified as keyframes.

[0015] Step 3: Input the keyframe image into the hoisting attitude detection model. The hoisting attitude detection model's hoisting segmentation module performs instance segmentation on the boom and hoisting equipment, and outputs the region segmentation results of the boom and hoisting equipment.

[0016] Step 3.1: Use the backbone network 2ResNet50 to extract features from the keyframe images;

[0017] Step 3.2: Use the graph reasoning attention module to locate the positions of the boom and chassis;

[0018] Step 3.3: Use a decoder to perform pixel-level segmentation of the detailed positions of the boom and chassis;

[0019] Step 3.4: Backpropagate the error generated by the segmentation to adjust the parameters of the graph inference attention and decoder.

[0020] Step 4: Use the camera's intrinsic parameters to perform distortion correction on the segmented region, input the distortion-corrected image into the hoisting posture detection module, and output the detection results.

[0021] Step 4.1: Multiply the segmentation result with the original image to obtain an image containing the boom and chassis.

[0022] Step 4.2: Using the camera's intrinsic parameters, a distortion correction algorithm is used to process the segmented image and output a normal viewpoint image.

[0023] Step 5: Compare the annotation results and the detection results to adjust the parameters of the hoisting attitude detection module.

[0024] Step 5.1: Use the input backbone network 3ResNet50 to extract features from the distortion-free image;

[0025] Step 5.2: Use the classification head to classify the pose of the image as normal.

[0026] Step 5.3: Backpropagate the loss caused by misclassification to adjust the parameters of the classification head.

[0027] Step 6: The parameters of the hoisting attitude detection model are frozen immediately after training.

[0028] Step 6.1: After training on the manually labeled dataset, fix the model's parameters;

[0029] Step 6.2: Deploy the fixed parameters to the platform to which they are applied.

[0030] Step 7: After loading the model parameters, use the hoisting attitude detection model to analyze the real-time video from the camera, and repeat steps two, three, and four.

[0031] To achieve the above objectives, in a first aspect, the present invention provides a steel structure hoisting device based on posture image recognition monitoring. The hoisting device includes a chassis and a boom, and further includes an image acquisition unit, an image processing unit, and an alarm unit. The image acquisition unit is connected to the input terminal of the image processing unit, and the alarm unit is connected to the output terminal of the image processing unit. The image acquisition unit includes a camera configured to acquire real-time video of the chassis posture and real-time video of the boom posture. The image processing unit includes a keyframe recognizer, a memory, and a hoisting posture detector. The keyframe recognizer is configured to use a ResNet38 neural network model to select keyframes from the video acquired by the image acquisition unit. The memory is configured to store keyframe images of the chassis posture and the boom posture. The hoisting posture detector is configured to receive the images in the memory, use a hoisting segmentation module GRBN to separate the boom and chassis, perform distortion correction, and use a hoisting posture detection module VGG to determine whether the chassis and boom exceed a set range. The alarm unit includes an alarm device configured to issue an alarm signal when the chassis posture or boom posture exceeds the set range.

[0032] Optionally, the camera includes a first camera located at the top of the boom and a second camera located on the outer side of the middle of the boom. The first camera and the second camera are used to acquire real-time video of the boom's posture, which includes the boom's rotation angle and height increase / decrease.

[0033] Optionally, the camera further includes a third camera disposed on the chassis, the third camera being used to acquire real-time video of the chassis attitude, the chassis attitude including the position of the chassis.

[0034] Optionally, in the image processing unit, the keyframe recognizer selects the video from the three cameras indiscriminately, selecting one keyframe from each camera per unit time. The time interval needs to be set manually, with a default value of 0.5s.

[0035] Optionally, the image processing unit stores the recognition results in a memory.

[0036] Optionally, the keyframe images of the chassis and boom stored in the memory are automatically stored after the lifting attitude detector has finished reading them.

[0037] Optionally, the steel structure hoisting equipment based on posture image recognition monitoring further includes a posture control unit, which is connected to the output of the image processing unit. The posture control unit is used to control the chassis posture and the boom posture. The chassis posture control includes chassis position control, and the boom posture control includes boom rotation angle control and height lifting control.

[0038] To achieve the above objectives, in a second aspect, the present invention provides a steel structure hoisting method based on posture image recognition and monitoring, comprising:

[0039] S1. Video Acquisition: Image acquisition unit 1 acquires real-time video of the chassis posture and the boom posture of the tracked crane;

[0040] S2. Keyframe selection: The keyframe recognizer 21 of the image processing unit 2 takes the real-time video of the chassis posture and the real-time video of the boom posture input by the image acquisition unit 1 as input, uses a pre-trained neural network to select keyframes appearing in the video, and outputs keyframe images in the video.

[0041] S3. Keyframe Storage: The video keyframes identified by the image processing unit 21 are stored in the memory 22 as input to the hoisting attitude detector 23. The memory also contains the intrinsic parameters of the camera in the image processing unit;

[0042] S4. Attitude Recognition: If the image of the chassis attitude or the real-time image of the boom attitude is determined to be in an abnormal state, an alarm is triggered; if the real-time images of the chassis attitude and the boom attitude are considered to be in a normal state, return to step S1 and continue to acquire real-time images.

[0043] Optionally, the steel structure hoisting method based on attitude image recognition monitoring also includes:

[0044] S3a. Image storage and transmission in the memory: The memory 22 of the image processing unit 2 writes the images output by the keyframe extractor 21 into the memory in the order of the cameras in the image acquisition unit 1, and then transmits them to the hoisting attitude detector 23 in the order after writing.

[0045] S4a. Image processing of the hoisting attitude detector: The hoisting attitude detector receives the image transmitted from the memory, and after processing and distortion removal by the hoisting instance segmentation module, it obtains a normal image of the boom and chassis attitude, which can be stored in the memory as a log.

[0046] Optionally, the steel structure hoisting method based on attitude image recognition monitoring also includes:

[0047] S7. Attitude control: The attitude control unit controls the attitude of the chassis and the attitude of the boom until the real-time images of the chassis attitude and the real-time images of the boom attitude both enter the set range.

[0048] This invention provides a steel structure hoisting equipment based on posture image recognition monitoring. During the movement of the chassis and boom, the camera of the image acquisition unit acquires real-time images of the chassis posture and the boom posture. The image processing unit receives the real-time chassis posture image and compares it with images within a set range to determine if the chassis posture exceeds the set range; it also receives the real-time boom posture image and compares it with images within a set range to determine if the boom posture exceeds the set range. When either the chassis posture or the boom posture exceeds the set range, the alarm unit's alarm sounds. Thus, the posture of the hoisting equipment is monitored in real time and controlled within a reasonable range, preventing overturning due to the hoisting equipment's posture exceeding the reasonable range, thereby improving the hoisting construction level and safety level. Attached Figure Description

[0049] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The accompanying drawings, which are included to provide a further understanding of the invention and constitute a part of this application, illustrate exemplary embodiments of the invention and, together with their descriptions, serve to explain the invention and do not constitute an undue limitation thereof.

[0050] Figure 1 This is a system configuration diagram of the first preferred embodiment of the steel structure hoisting equipment based on posture image recognition monitoring provided by the present invention.

[0051] Figure 2 This is a schematic diagram of the image acquisition unit.

[0052] Figure 3 This is a system configuration diagram of the second preferred embodiment of the steel structure hoisting equipment based on posture image recognition monitoring provided by the present invention.

[0053] Figure 4 This is a flowchart of a preferred embodiment of the steel structure hoisting method based on posture image recognition and monitoring provided by the present invention.

[0054] Figure 5This is a flowchart of a second preferred embodiment of the steel structure hoisting method based on posture image recognition and monitoring provided by the present invention.

[0055] Figure 6 This is a flowchart of the third preferred embodiment of the steel structure hoisting method based on posture image recognition and monitoring provided by the present invention.

[0056] Figure 7 This is a flowchart of an embodiment of a steel structure hoisting method based on posture image recognition monitoring. Figure 1 .

[0057] Figure 8 This is a flowchart of an embodiment of a steel structure hoisting method based on posture image recognition monitoring. Figure 2 .

[0058] In the figure, 1—image acquisition unit, 11—first camera, 12—second camera, 13—third camera; 2—image processing unit, 21—keyframe recognizer, 22—memory, 23—hoisting attitude detector; 3—alarm unit, 31—alarm; 4—attitude control unit. Detailed Implementation

[0059] The steel structures of large buildings are divided into frame structures and large-span truss structures. The main components include steel columns, steel beams, and roof trusses. The span of large buildings often exceeds 100 meters. After the overall hoisting of the frame structure is completed, truss sliding construction is required. Currently, crawler cranes are mostly used for hoisting and sliding construction. During construction, crawler cranes can move and install simultaneously, which is highly efficient. However, in the current steel structure installation process, the crane operator relies on experience to judge the condition of the crane. If the operator is not careful, the boom and crane equipment may overturn if their posture exceeds the reasonable range.

[0060] Therefore, the present invention provides a steel structure hoisting device and method based on posture image recognition monitoring. The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use.

[0061] Figure 1 This is a system configuration diagram of a first preferred embodiment of the steel structure hoisting equipment based on posture image recognition monitoring provided by the present invention. Figure 1As shown in some preferred embodiments, the steel structure hoisting equipment based on posture image recognition monitoring provided by the present invention includes a chassis and a boom, and includes an image acquisition unit 1, an image processing unit 2, and an alarm unit 3. The image acquisition unit 1 is connected to the input terminal of the image processing unit 2, and the alarm unit 3 is connected to the output terminal of the image processing unit 2. The image acquisition unit 1 includes a camera, which is configured to acquire real-time images of the chassis posture and the boom posture. The image processing unit 2 includes a keyframe recognizer 21, a memory 22, and a hoisting posture detector 23. The keyframe recognizer 21 is configured to receive the chassis posture... The system captures real-time video of the crane's posture and outputs key frames from the video to a memory for storage. The memory 22 is configured to store key frames and intrinsic parameters of the video captured by cameras 1, 2, and 3 in the image acquisition unit 1. The crane posture detector is configured as an image instance segmentation module, a distortion correction module, and a crane posture recognition module. After performing instance segmentation and distortion correction on the key frames in the memory 22, the crane posture recognition module determines whether the chassis posture and boom posture are normal, and the crane posture detector outputs the detection result. The alarm unit 3 includes an alarm 31, which is configured to issue an alarm signal when the chassis posture or boom posture exceeds a set range. In specific implementation, the image acquisition unit 1, image processing unit 2, and alarm unit 3 form an integrated control system, such as a computer, industrial control computer, or artificial intelligence system, which can realize automated control system for image comparison, analysis, recognition, judgment, learning, and understanding.

[0062] Compared with existing technologies, the steel structure hoisting equipment based on posture image recognition monitoring provided by this invention includes a chassis and a boom. The image acquisition unit's camera is configured to acquire real-time video of the chassis posture and the boom posture. The image processing unit includes a keyframe recognizer, a memory, and a hoisting posture detector. The keyframe recognizer is configured as a ResNet50 backbone network for selecting keyframes from the video acquired by the image acquisition unit. The memory is configured to store keyframe images of the chassis posture and the boom posture. The hoisting posture detector is configured to receive the images in the memory, separate the boom and chassis using a hoisting segmentation module, and after distortion correction, determine whether the chassis and boom exceed a set range using the ResNet50 backbone network of the hoisting posture detection module. An alarm unit is configured to issue an alarm signal when the chassis posture or boom posture exceeds the set range. This allows for real-time monitoring of the hoisting equipment's posture, preventing overturning problems caused by the hoisting equipment's posture exceeding a reasonable range, and improving the level of hoisting construction and safety.

[0063] Figure 2 This is a schematic diagram of the image acquisition unit. (Example) Figure 2As shown, in some preferred embodiments, the camera includes a first camera 11 located at the top of the boom, a second camera 12 located on the outer side of the middle of the boom, and a third camera 13 mounted on the chassis. The first camera 11 and the second camera 12 are used to acquire real-time images of the boom's posture, including its rotation angle and height adjustment. The third camera 13 is used to acquire real-time images of the chassis's posture, including its position. By installing the first camera, the second camera, and the third camera at the top, side, and chassis of the boom respectively, the posture, lifting progress, and sliding movement of the lifting equipment can be effectively monitored, thereby improving the monitoring range and accuracy. Preferably, the cameras can be video cameras, visual detectors, or ultrasonic detectors.

[0064] Figure 3 This is a system configuration diagram of a second preferred embodiment of the steel structure hoisting equipment based on attitude image recognition monitoring provided by the present invention. Figure 3 As shown, in some preferred embodiments, the image processing unit 2 further includes a keyframe recognizer, a memory, and a hoisting posture detector. The keyframe recognizer is configured as a ResNet38 neural network model for selecting keyframes from the video acquired by the image acquisition unit. The memory is configured to store keyframe images of the chassis posture and the boom posture. The hoisting posture detector is configured to receive the images in the memory, use the hoisting segmentation module GRBN to separate the boom and chassis, and after distortion correction, the hoisting posture detection module VGG determines whether the chassis and boom exceed the set range. In this embodiment, by segmenting and distorting the real-time images of the chassis posture and the boom posture by the image processor 23, the information of the chassis posture and boom posture can be obtained more accurately, and invalid information in image recognition can be eliminated, reducing the amount of computation in the image recognition process. This reduces the amount of computation of the image processing unit, improves the processing speed, and enables faster detection of problems where the chassis posture and boom posture exceed the set range, thus avoiding overturning problems more promptly and improving the safety of equipment operation.

[0065] like Figure 3As shown, in some preferred embodiments, the steel structure hoisting equipment based on posture image recognition monitoring further includes a posture control unit 4. The posture control unit 4 is connected to the output of the image processing unit 2. The posture control unit 4 is used to control the chassis posture and boom posture. The chassis posture control includes chassis position control, and the boom posture control includes boom rotation angle control and height lifting control. In this embodiment, by connecting the image processing unit 2 and the posture control unit 4, the equipment can promptly obtain whether the chassis posture and boom posture exceed the set range. When the chassis posture and boom posture exceed the set range, the posture control unit 4 promptly adjusts the chassis posture and boom posture back to the set range, thereby actively avoiding overturning problems. This eliminates the need for driver operation, effectively improving the equipment's operational level and safety. In practical implementation, the attitude control unit is an automated control device such as a computer, PLC controller, or industrial control computer. The attitude control unit and the control body that restores the posture of the boom or chassis in an abnormal state after the alarm device sounds should have sufficient contingency plans for various situations of abnormal boom or chassis status. When the boom posture is abnormal, the extension of the boom should be stopped immediately, the movement of the hoisted object should be stopped, and the boom posture should be adjusted according to the normal direction until the alarm stops.

[0066] like Figure 4 As shown above, Figure 3 The keyframe recognizer 21 is the backbone network 1 (ResNet50), which takes video frames as input and outputs clear images of the boom and chassis. The lifting posture detector 23 consists of a backbone network 2 (ResNet50), a graph inference attention module, distortion correction, and a backbone network 3 (ResNet50). The number of video frames is determined by the camera frequency. Keyframes are extracted according to the frequency during the operation of the boom and chassis, with one image extracted from each video. The keyframes are input to the backbone network 2 of the lifting detector, and the features extracted by the backbone network are then passed to the graph inference attention module to locate the position contours of the boom and chassis, outputting the segmentation results of the boom and chassis. After distortion correction, the posture of the boom and chassis from the standard viewpoint is obtained. This image is then input to the backbone network 3 for feature processing and analysis, outputting whether it is normal or abnormal.

[0067] like Figure 5 As shown, the design of the graph inference attention module in the hoisting attitude detector is as follows. Given a feature graph G = (V, E), where A is the adjacency matrix of the graph, V is the set of vertices, and E is the set of edges, the corresponding Laplace matrix can be calculated. D is the degree matrix of the graph nodes, I n It is the identity matrix. By definition, L is a real symmetric matrix, and L can be orthogonalized, therefore L = UΛU TU is a set of orthogonal bases, and Λ is the eigenvalue of the matrix. After Fourier transform, Chebynet transform, and low-order approximation of GCN, the definition of graph convolution can be transformed into the following form:

[0068] Gconv(x)=U(I-Λ)U T x (1)

[0069] Given input features X∈R C×H×W ,Y∈R C×H×W After channel dimensionality reduction, the features are aggregated into a single channel, as shown in the formula, where For `torch.mul()`, `sigmoid` is `torch.Sigmoid()`.

[0070]

[0071] After flattening, conv1d, and reshaping, a state transition matrix is ​​formed, which is the similarity matrix required for graph convolution operations. This matrix is ​​multiplied by the input encoder feature map, and the final output is obtained after passing through multiple stacked graph convolution modules.

[0072] In summary, the feature map, after being transformed by the graph reasoning module, can be viewed as follows:

[0073] X'=X*Conv1d(Flatten(map)) (3)

[0074] The feature map is continuously updated, and the graph inference modules are stacked here. X' is used as input to Equation 2 to obtain the updated map and X. After several iterations, the weights of the output graph nodes tend to stabilize, which helps the model grasp the overall image and improves the stability of the hoisting posture detector.

[0075] Figure 6 This is a flowchart of a preferred embodiment of the steel structure hoisting method based on attitude image recognition and monitoring provided by the present invention. Figure 4 As shown in some preferred embodiments, the present invention provides a steel structure hoisting method based on posture image recognition monitoring, comprising:

[0076] S1. Video Acquisition: Image acquisition unit 1 acquires real-time video of the chassis posture and the boom posture of the tracked crane;

[0077] S2. Keyframe selection: The keyframe recognizer 21 of the image processing unit 2 takes the real-time video of the chassis posture and the real-time video of the boom posture input by the image acquisition unit 1 as input, uses a pre-trained neural network to select keyframes appearing in the video, and outputs keyframe images in the video.

[0078] S3. Keyframe Storage: The video keyframes identified by the keyframe recognizer 21 of the image processing unit are stored in the memory 22 as input to the hoisting attitude detector 23. The memory also contains the intrinsic parameters of the camera in the image processing unit;

[0079] S4. Pose Recognition: Keyframes are processed by the hoisting instance segmentation module in the hoisting pose detector to calibrate the positions of the boom and hoisting equipment in the image, and the images of the key positions are extracted; the camera intrinsic parameters in memory 22 are used to distort the boom and hoisting equipment to obtain images from a normal viewpoint; the distorted images are input into the hoisting pose recognition module, and the neural network calculates whether the boom pose and chassis pose are out of range.

[0080] S5. Alarm: If the outline of the real-time image of the chassis posture exceeds the set range or the outline of the real-time image of the boom posture exceeds the set range, the alarm 31 will be triggered.

[0081] S6: If the outline of the real-time image of the chassis posture exceeds the set range and the outline of the real-time image of the boom posture does not exceed the set range, return to step S1 and continue to acquire real-time images.

[0082] Compared with existing technologies, the steel structure hoisting method based on posture image recognition monitoring provided by this invention, during the movement of the chassis and boom, uses a camera in the image acquisition unit to acquire real-time video of the chassis posture and a camera in the boom posture. The image processing unit receives these real-time images and determines whether the hoisting posture exceeds a set range. When either the chassis posture or the boom posture is abnormal, the alarm unit's alarm sounds. This allows for real-time monitoring and control of the hoisting equipment's posture within a reasonable range, preventing overturning due to the hoisting equipment's posture exceeding this range, and improving the overall hoisting construction level and safety.

[0083] Figure 7 This is a flowchart of a second preferred embodiment of the steel structure hoisting method based on attitude image recognition monitoring provided by the present invention. Figure 7 As shown, in some preferred embodiments, the steel structure hoisting method based on attitude image recognition monitoring provided by the present invention further includes:

[0084] S3a. Image storage and transmission in the memory: The memory 22 of the image processing unit 2 writes the images output by the keyframe extractor 21 into the memory in the order of the cameras in the image acquisition unit 1, and then transmits them to the hoisting attitude detector 23 in the order after writing.

[0085] S4a. Lifting posture detection: Input the image in memory 22 into lifting posture detector 23 to determine whether the boom posture and chassis posture are out of range; normal boom and chassis posture images and recognition results can be stored in memory as logs.

[0086] In this embodiment, by writing keyframes sequentially into the memory, information on the chassis attitude and boom attitude at the same moment can be obtained more accurately, improving the recognition accuracy. Furthermore, storing normal images of the boom and chassis attitudes along with the recognition results facilitates the evaluation and maintenance of the image processing unit.

[0087] Figure 8 This is a flowchart of a third preferred embodiment of the steel structure hoisting method based on attitude image recognition monitoring provided by the present invention. As shown in the figure, in some preferred embodiments, the steel structure hoisting method based on attitude image recognition monitoring provided by the present invention further includes:

[0088] S7. Attitude Adjustment: The attitude control unit 4 controls the attitude of the chassis and the attitude of the boom until the real-time images of the chassis attitude and the boom attitude both enter the set range.

[0089] In this embodiment, the attitude control unit 4 adjusts the chassis attitude and boom attitude to the set range in a timely manner without driver intervention, which can actively avoid overturning and improve the safety of the equipment.

[0090] The above specific embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A steel structure hoisting method based on posture image recognition and monitoring, characterized in that: This hoisting method uses a trained neural network model as an attitude detector and employs image processing technology to determine the normal attitude of the boom and hoisting equipment within the monitoring range, including: Step 1: Use a hoisting camera to capture video of the hoisting site construction and annotate the posture of the boom and hoisting equipment in the key frames of the video; Step 1.1: Use three sets of cameras to monitor the hoisted chassis and boom and obtain real-time video. Step 1.2: Divide the real-time video into segments according to a fixed frequency; Step 1.3: Select 1-2 clear frames of the boom and chassis from each group of divided real-time videos and mark them; Step 1.4: Manually segment the posture of the boom and chassis in the marked image and mark whether the posture is normal or not; Step 2: Train a video keyframe separation model using the labeled dataset and output the keyframes in the video; Step 2.1: Extract video features using the backbone network 1 ResNet50; Step 2.2: Use the classification head to classify the keyframes in the video; Step 2.3: Output the images classified as keyframes; Step 3: Input the keyframe image into the hoisting attitude detection model. The hoisting attitude detection model's hoisting segmentation module performs instance segmentation on the boom and hoisting equipment, and outputs the region segmentation results of the boom and hoisting equipment. Step 3.1: Use the backbone network 2 ResNet50 to extract features from the keyframe images; Step 3.2: Use the graph reasoning attention module to locate the positions of the boom and chassis; Step 3.3: Use a decoder to perform pixel-level segmentation of the detailed positions of the boom and chassis; Step 3.4: Backpropagate the error generated by the segmentation to adjust the parameters of the graph inference attention and decoder; Step 4: Use the camera's intrinsic parameters to perform distortion correction on the segmented region, input the distortion-corrected image into the hoisting posture detection module, and output the detection results; Step 4.1: Multiply the segmentation result with the original image to obtain an image containing the boom and chassis. Step 4.2: Combining the camera's intrinsic parameters, use a distortion correction algorithm to process the segmented image and output a normal viewpoint image; Step 5: Compare the annotation results and the detection results to adjust the parameters of the hoisting attitude detection module; Step 5.1: Use the input backbone network 3 ResNet50 to extract features from the distortion-free image; Step 5.2: Use the classification head to classify the pose of the image as normal. Step 5.3: Backpropagate the loss caused by misclassification to adjust the parameters of the classification head; Step 6: The parameters of the hoisting attitude detection model are frozen immediately after training; Step 6.1: After training on the manually labeled dataset, fix the model's parameters; Step 6.2: Deploy the fixed parameters to the platform to which they are applied; Step 7: After loading the model parameters, use the hoisting attitude detection model to analyze the real-time video from the camera, and repeat steps 2, 3, and 4. In the graph reasoning attention module of the hoisting attitude detector, given a feature map... Let A be the adjacency matrix of the graph, V be the set of vertices, and E be the set of edges. Find the corresponding Laplace matrix. D is the degree matrix of the graph nodes. It is the identity matrix. According to the definition, L is a real symmetric matrix, and L can be orthogonalized. U is an orthogonal basis. These are the eigenvalues ​​of the matrix. After Fourier transform, Chebynet transform, and a low-order approximation of GCN, the definition of graph convolution is transformed into the following form: （1）； Given input features After channel dimensionality reduction, features are aggregated into a single channel, as shown in the formula, where For `torch.mul()`, `sigmoid` is `torch.Sigmoid()`; （2）； After flattening, conv1d, and reshaping, a state transition matrix is ​​formed, which is the similarity matrix required for graph convolution operations. This matrix is ​​multiplied by the input encoder feature map, and the final output is obtained after passing through multiple stacked graph convolution modules. When the feature map is transformed by the graph inference module: （3）； The feature map is continuously updated. As input to equation (2), the updated map and The weights of the graph nodes output after the loop tend to stabilize, improving the stability of the hoisting attitude detector.

2. A steel structure hoisting equipment based on posture image recognition monitoring to implement the method described in claim 1, characterized in that: The hoisting equipment includes a chassis and a boom, and further includes an image acquisition unit, an image processing unit, and an alarm unit. The image acquisition unit is connected to the input of the image processing unit, and the alarm unit is connected to the output of the image processing unit. The image acquisition unit includes a camera configured to acquire real-time video of the chassis posture and the boom posture. The image processing unit includes a keyframe recognizer, a memory, and a hoisting posture detector. The keyframe recognizer is configured as a neural network model for selecting keyframes from the video acquired by the image acquisition unit. The memory is configured to store keyframe images of the chassis posture and the boom posture. The hoisting posture detector is configured to receive the images in the memory, use the hoisting segmentation module GRBN to separate the boom and chassis, perform distortion correction, and use the hoisting posture detection module VGG to determine whether the chassis and boom exceed a set range. The alarm unit includes an alarm device configured to issue an alarm signal when the chassis posture or boom posture exceeds the set range.

3. The steel structure hoisting equipment based on posture image recognition monitoring according to claim 2, characterized in that: The camera includes a first camera located at the top of the boom and a second camera located on the outer side of the middle of the boom. The first camera and the second camera are used to acquire real-time video of the boom's posture, which includes the boom's rotation angle and height increase / decrease.

4. The steel structure hoisting equipment based on posture image recognition monitoring according to claim 2, characterized in that: The camera also includes a third camera mounted on the chassis, which is used to acquire real-time video of the chassis attitude, including the position of the chassis. In the image processing unit, the keyframe recognizer selects the video from the three cameras without discrimination, selecting one keyframe from each camera per unit time. The time interval needs to be set manually, with a default value of 0.5s.

5. A steel structure hoisting equipment based on posture image recognition monitoring according to claim 2, characterized in that: The image processing unit's recognition results are stored in the memory.

6. The steel structure hoisting equipment based on posture image recognition monitoring according to claim 2, characterized in that: The keyframe images of the chassis and boom stored in the memory are automatically stored after the lifting attitude detector finishes reading them.

7. A steel structure hoisting equipment based on posture image recognition monitoring according to claim 2, characterized in that: The steel structure hoisting equipment based on posture image recognition monitoring also includes a posture control unit, which is connected to the output of the image processing unit. The posture control unit is used to control the posture of the chassis and the posture of the boom. The chassis posture control includes chassis position control, and the boom posture control includes boom rotation angle control and height lifting control.

8. A steel structure hoisting equipment based on posture image recognition monitoring according to claim 7, characterized in that: The implementation method of this device includes: S1. Video Acquisition: The image acquisition unit acquires real-time video of the chassis posture and the boom posture of the tracked crane; S2. Keyframe selection: The keyframe recognizer of the image processing unit takes the real-time video of the chassis posture and the real-time video of the boom posture input by the image acquisition unit as input, uses a pre-trained neural network to select keyframes appearing in the video, and outputs keyframe images in the video. S3. Keyframe storage: The video keyframes identified by the image processing unit are stored in the memory as input to the hoisting attitude detector; the memory also contains the intrinsic parameters of the camera in the image processing unit; S4. Attitude Recognition: If the image of the chassis attitude or the real-time image of the boom attitude is determined to be in an abnormal state, an alarm is triggered; if the real-time images of the chassis attitude and the boom attitude are considered to be in a normal state, return to step S1 and continue to acquire real-time images.

9. A steel structure hoisting equipment based on posture image recognition monitoring according to claim 8, characterized in that: Steel structure hoisting methods based on posture image recognition monitoring also include: Image storage and transmission in the memory: The memory of the image processing unit writes the images output by the keyframe extractor into the memory in the order of the cameras in the image acquisition unit, and then transmits them to the hoisting attitude detector in the order after writing. Image processing of the hoisting attitude detector: The hoisting attitude detector receives the image transmitted from the memory, and after processing and distortion removal by the hoisting instance segmentation module, it obtains a normal image of the boom and chassis attitude, which can be stored in the memory as a log.

10. A steel structure hoisting equipment based on posture image recognition monitoring according to claim 8, characterized in that: Steel structure hoisting methods based on posture image recognition monitoring also include: Attitude control: The attitude control unit controls the attitude of the chassis and the attitude of the boom until the real-time images of the chassis attitude and the boom attitude both enter the set range.

Images