Metallurgy crane hook state and steel ladle posture dynamic whole-process safety monitoring method and device

Abstract

The invention provides a dynamic whole-process safety monitoring method and device for a metallurgical crane hook state and a steel ladle posture. The method comprises the following steps: building a reduced scale crane model, and simulating a process of hoisting a steel ladle by a crane; a ladle crane image data set in the simulation process is obtained, and ladle key point labeling and lifting hook labeling are conducted on the image data set; using the marked image data set to train a YOLOv8 model to obtain a steel ladle key point detection model; the method comprises the following steps: acquiring a small amount of labeled ladle crane image data sets acquired from a real industrial site, and training a ladle key point detection model by using the image data sets to obtain a safety monitoring model; inputting ladle crane image data acquired in real time into the safety monitoring model, and outputting ladle key point coordinates, lifting hook barycentric coordinates and lifting lug barycentric coordinates; and according to the output coordinate information, the steel ladle posture and the lifting hook state are calculated.

Description

Methods and devices for dynamic full-process safety monitoring of hook status and ladle posture of metallurgical cranes Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and device for dynamic full-process safety monitoring of the hook status and ladle posture of a metallurgical crane. Background Technology

[0002] In the heavy industry of steel smelting, the ladle, as a key piece of equipment for carrying and transferring molten steel at high temperatures, directly affects the safety and stability of the entire production process. Currently, ladle hoisting operations still rely heavily on the experience and judgment of operators. In the complex industrial environment of high temperature, high noise, and high dust, manual monitoring suffers from problems such as strong subjectivity, delayed response, and susceptibility to fatigue. This can easily lead to serious accidents such as ladle instability, tilting, or even molten steel spillage during hoisting, causing not only equipment damage and production interruption but also posing a significant threat to personnel safety.

[0003] With the improvement of industrial automation and intelligence, computer vision-based safety monitoring technology is gradually being applied to industrial scenarios. Existing steel ladle safety monitoring methods mainly focus on single-target detection or position recognition. For example, CN113096186A proposes a machine vision-based steel ladle position measurement method, which focuses primarily on the recognition and localization of the ladle's outline. Relying on traditional image processing techniques (such as edge detection and morphological manipulation), it has the following limitations: 1. Sensitive to illumination and occlusion: Traditional methods show a significant decrease in detection performance under varying illumination or partial occlusion; 2. Lack of pose estimation capability: It can only identify the ladle's position but cannot determine its tilt state, making it difficult to warn of tipping risks; 3. Does not cover hook status recognition: It ignores the connection status between the hook and the lifting lugs, making it impossible to determine whether it is in a correct hanging state.

[0004] In summary, traditional steel ladle safety monitoring methods have limitations such as low detection accuracy and poor resistance to obstruction in complex environments. They also lack the ability to comprehensively perceive and estimate the attitude of the entire hoisting process, making it difficult to meet the safety monitoring needs of the entire steel ladle hoisting process. Summary of the Invention

[0005] To meet the safety monitoring requirements of the entire process of ladle hoisting, this invention proposes a method and device for dynamic full-process safety monitoring of the hook status and ladle posture of a metallurgical crane, realizing joint monitoring and early warning of the hook status and ladle posture, effectively improving the safety level of ladle hoisting operations.

[0006] In a first aspect, the present invention provides a method for dynamic full-process safety monitoring of the hook state and ladle posture of a metallurgical crane, comprising: constructing a scaled-down crane model, simulating the process of a crane lifting a ladle based on the scaled-down crane model; acquiring a ladle crane image dataset during the simulated ladle lifting process, denoted as a first image dataset, and annotating key points of the ladle and the hook in the first image dataset; training an improved YOLOv8 model using the annotated first image dataset to obtain a ladle key point detection model; acquiring a small number of labeled ladle crane image datasets collected from real industrial sites, denoted as a second image dataset, and training the ladle key point detection model using the second image dataset to obtain a safety monitoring model; inputting real-time collected ladle crane image data into the safety monitoring model, and outputting the coordinates of the ladle key points, the coordinates of the hook center of gravity, and the coordinates of the lug center of gravity; calculating the ladle posture based on the output ladle key point coordinates; and calculating the hook state based on the coordinates of the hook center of gravity, the coordinates of the lug center of gravity, and the positional relationship between the hook and the lug.

[0007] Furthermore, key point annotation is performed on the first image dataset, specifically including: marking two key points P1 and P3 on the central axis of the ladle, marking one key point P2 at the connection between the hook and the ladle, marking one key point P4 at the ladle outlet, and marking one key point P5 at the suspension rotating joint.

[0008] Furthermore, the construction process of the improved YOLOv8 model includes: replacing the convolutional layers connected to the C2f modules in the YOLOv8n backbone network with Ladle Conv modules; and inputting four feature maps—the feature maps output by the first three C2f modules and the SPPF module—into the neck network; for the YOLOv8n neck network, replacing all C2f modules with C2f_SDW modules, all convolutional layers with Ladle Conv modules, and all Concat layers with BiFPN modules; adding four parallel convolutional layers at the input of the neck network; and adding a BiFPN module and a C2f_SDW module sequentially at the front end of the bottom-up fusion path. The newly added C2f_SDW module and the replaced Ladle Conv module are connected together. The Ladle Conv module is connected; correspondingly, the feature fusion process of the improved back neck network includes: the four feature maps from the improved backbone network are first processed by the newly added convolutional layer to obtain four new feature maps; the first feature map in the four new feature maps is directly input into the newly added BiFPN module; the latter three feature maps are simultaneously input into the BiFPN module in the top-down fusion path and the BiFPN module in the bottom-up fusion path, respectively; the feature maps output by the three C2f_SDW modules in the bottom-up fusion path are input into the detection head; wherein, the Ladle Conv module includes a main branch, a side branch, a feature fusion layer and an SE attention mechanism layer; the main branch includes a 3×3 depthwise separable convolution and a 3×3 partial convolution connected in sequence; the side branch includes a 5×5 depthwise separable convolution and a 1×1 convolution connected in sequence; the outputs of the main branch and the side branch are fused by the feature fusion layer and then input into the SE attention mechanism layer to achieve feature enhancement; the construction process of the C2f_SDW module is: the convolutional layer in the C2f module is replaced with a depthwise separable convolution, and the Bottleneck layer is replaced with a StarBlock.

[0009] Furthermore, based on the output coordinates of the ladle's key points, the ladle's attitude is calculated, specifically including: calculating directed line segment A based on the coordinates of key points P1 and P3; calculating directed line segment B based on the coordinates of key points P2 and P5; and calculating the tilt angle θ of the two directed line segments in the image coordinate system. A and θ B The absolute value of the difference between the two tilt angles is taken as the deflection angle of the ladle.

[0010] Furthermore, it also includes: setting a deflection angle threshold; if the detected deflection angle of the ladle is greater than the deflection angle threshold, an alarm is triggered.

[0011] Furthermore, based on the coordinates of the hook's center of gravity, the coordinates of the lifting lug's center of gravity, and the positional relationship between the hook and the lifting lug, the hook state is calculated. Specifically, this includes: calculating the Euclidean distance between the two centers of gravity; if the distance is less than a preset threshold, the hook is in the suspended and positioned state; if the distance is greater than the preset threshold and the hook's center of gravity is above the lifting lug's center of gravity, the hook is in the state of approaching the ladle; when any of the above two situations occur, the hook is in the state of being detached from the ladle.

[0012] Secondly, this invention provides a dynamic full-process safety monitoring device for the hook state and ladle posture of a metallurgical crane, comprising: a pre-training module, used to acquire a ladle crane image dataset during the simulated ladle lifting process based on a scaled-down crane model, denoted as a first image dataset, and to annotate key points of the ladle and hook in the first image dataset; and to train an improved YOLOv8 model using the annotated first image dataset to obtain a ladle key point detection model; a fine-tuning module, used to acquire a small number of labeled ladle crane image datasets collected from real industrial sites, denoted as a second image dataset, and to train the ladle key point detection model using the second image dataset to obtain a safety monitoring model; a target detection module, used to input real-time collected ladle crane image data into the safety monitoring model, and output the coordinates of key points of the ladle, the coordinates of the hook center of gravity, and the coordinates of the lug center of gravity; and a posture estimation module, used to calculate the ladle posture based on the output ladle key point coordinates; and to calculate the hook state based on the coordinates of the hook center of gravity and the lug center of gravity.

[0013] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in the first aspect.

[0014] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0015] The beneficial effects of the present invention are: (1) Attitude estimation and tilt warning: by defining five key points of the steel ladle (P1~P5) and calculating the deflection angle, it is determined in real time whether the steel ladle is in a safe attitude, and an alarm is immediately triggered when the deflection angle of the steel ladle exceeds the safety threshold.

[0016] (2) Full-process monitoring: This invention covers the entire process of status identification from the hook approaching the ladle (Pre-hook), the hoisting into place (hook) to the detachment (Unhook), realizing continuous monitoring of hoisting operations.

[0017] (3) Lightweight model and edge deployment: By improving the YOLOv8 structure and introducing modules such as LadleConv, C2f-SDW and BiFPN, the number of model parameters and computational complexity are significantly reduced, and real-time inference on Jetson TX1 is realized. Attached Figure Description

[0018] Figure 1 is a flowchart illustrating a method for dynamic full-process safety monitoring of the hook state and ladle attitude of a metallurgical crane according to an embodiment of the present invention; Figure 2 is a scaled-down crane model according to an embodiment of the present invention: (a) is a physical image, (b) is an exploded view; wherein, 1 and 2 are screw-slider mechanisms moving in the x-axis direction; 3 is a screw-slider mechanism moving in the y-axis direction; 4 is a motor-like hoist device moving in the z-axis direction, using a motor to drive the steel chain for vertical movement; Figure 3 is a schematic diagram of ladle key point selection and annotation according to an embodiment of the present invention: (a) is a schematic diagram of ladle key point selection, (b) is a schematic diagram of ladle annotation dataset; Figure 4 is a structural schematic diagram of the improved YOLOv8 model according to an embodiment of the present invention; Figure 5 is a structural schematic diagram of the Ladle Conv module according to an embodiment of the present invention; Figure 6 is a construction process of the C2f_SDW module according to an embodiment of the present invention; Figure 7 is a structural schematic diagram of the BiFPN module according to an embodiment of the present invention; Figure 8 is a ladle attitude estimation process according to an embodiment of the present invention; Figure 9 is a hook state judgment process according to an embodiment of the present invention; Figure 10 is a Jetson model according to an embodiment of the present invention. Figure 11 is a schematic diagram of the detection results provided in the embodiment of the present invention; Figure 12 is a schematic diagram of the structure of a dynamic full-process safety monitoring device for the hook status and ladle posture of a metallurgical crane provided in the embodiment of the present invention; Figure 13 is a structural block diagram of an electronic device provided in the embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] As shown in Figure 1, this embodiment of the invention provides a method for dynamic full-process safety monitoring of the hook state and ladle posture of a metallurgical crane, including the following steps: S101: Source domain scaled-down model experiment. A scaled-down crane model is built, and the process of the crane lifting the ladle is simulated based on the scaled-down crane model; specifically, the scaled-down crane model built in this embodiment is shown in Figure 2. The main body of the crane model is constructed of aluminum alloy, and the three axes are all driven by motor-driven lead screw slides, equipped with a PAC control system.

[0021] S102: Construct the source domain dataset. Obtain an image dataset of a steel ladle crane simulating the process of a crane lifting a steel ladle, denoted as the first image dataset. Annotate the key points of the steel ladle and the hook in the first image dataset.

[0022] Specifically, during the process of lifting steel ladles using a scaled-down crane model, video image data was acquired using an Intel RealSense D435i depth camera. Steps S101 and S102 aim to address the difficulty in obtaining steel ladle crane datasets.

[0023] Due to the complex working postures of the ladle, which can be vertical, tilted, or parallel, selecting appropriate key points to characterize its morphological features is crucial for the visual positioning system. Since different types of ladles have different appearances, this embodiment proposes a method for selecting key points on the ladle, based on a common ladle model (1:40) for 3D printing. As shown in Figure 3, based on the ladle's approximate cylindrical shape, five key points are set on its outline. P1 and P3 are located on the central axis of the ladle, P2 is located at the connection between the hook and the ladle, P4 is located at the ladle's discharge port, and P5 is located at the suspension rotating joint. The skeleton formed by these key points helps identify the angle and position information of the ladle relative to the suspension, providing a foundation for subsequent attitude estimation of the ladle using the position information of its key points.

[0024] Finally, 6577 images were annotated using annotation software, as shown in Figure 3. All annotated images were then divided into training, validation, and test sets in an 8:1:1 ratio.

[0025] S103: Pre-training on source domain data. The improved YOLOv8 model is trained using the labeled first image dataset to obtain the ladle keypoint detection model. Specifically, the original YOLOv8 model has already been trained on a large general dataset (such as ImageNet), thus possessing general image feature extraction capabilities (such as edge, texture, and shape recognition). This invention makes structural improvements to the original YOLOv8 model. Next, the improved YOLOv8 model is fully trained using 6577 labeled images (keypoints and hook states) produced in step S102. The aim is to make the model forget some general features unrelated to ladle detection (such as cats, dogs, cars, etc.) and instead learn specific features related to the ladle and hook, such as the cylindrical outline of the ladle, the geometry of the hook, and the spatial relationship between keypoints (P1-P5). After training, the ladle keypoint detection model is obtained. It should be noted that this ladle keypoint detection model is currently mainly good at working in a scaled-down crane model environment in the laboratory.

[0026] S104: Domain Fine-tuning. A small dataset of labeled images of ladle cranes collected from real industrial sites is obtained, designated as the second image dataset. This dataset is used to train the ladle key point detection model, resulting in a safety monitoring model. Specifically, because the ladle key point detection model obtained in step S103 already possesses a strong foundation in ladle features, a small amount of data from real industrial scenarios is sufficient for model calibration and adaptation. This includes: adapting to the size, color, and texture of real ladles; overcoming challenges in real industrial environments, such as steam, dust, and complex backgrounds; and learning performance under different lighting conditions (daytime, nighttime, and furnace fire illumination). Therefore, this fine-tuning process does not require massive amounts of data from real industrial scenarios.

[0027] At this point, step S104 completes the knowledge transfer from the model laboratory to the real industrial site, solving the problem of scarce data in real industrial scenarios.

[0028] S105: Input the real-time acquired image data of the ladle crane into the safety monitoring model, and output the coordinates of key points of the ladle, the coordinates of the hook center of gravity, and the coordinates of the lug center of gravity; S106: Calculate the ladle attitude based on the output coordinates of key points of the ladle; calculate the hook state based on the coordinates of the hook center of gravity, the coordinates of the lug center of gravity, and the positional relationship between the hook and the lug. The detection results are shown in Figure 11.

[0029] In one embodiment, as shown in Figure 4, this embodiment of the invention provides an improved YOLOv8 model based on the YOLOv8n architecture. The main improvements to this architecture are as follows: the convolutional layers connected to the C2f modules in the YOLOv8n backbone network are replaced with Ladle Conv modules; four feature maps—the first three C2f modules and the feature maps output by the SPPF module—are input to the neck network; for the YOLOv8n neck network, all C2f modules are replaced with C2f_SDW modules, all convolutional layers are replaced with Ladle Conv modules, and all Concat layers are replaced with BiFPN modules; four parallel convolutional layers are added at the input of the neck network; a BiFPN module and a C2f_SDW module are added sequentially at the front end of the bottom-up fusion path, and the newly added C2f_SDW module and the replaced Ladle... Conv layer connection; correspondingly, the feature fusion process of the improved neck network includes: the four feature maps from the improved backbone network are first processed by the newly added convolutional layer to obtain four new feature maps; the first feature map of the four new feature maps is directly input into the newly added BiFPN module; the latter three feature maps are simultaneously input into the BiFPN module in the top-down fusion path and the BiFPN module in the bottom-up fusion path, respectively; the feature maps output by the three C2f_SDW modules in the bottom-up fusion path are input into the detection head; wherein, as shown in Figure 5, the Ladle The Conv module employs a multi-scale convolutional structure, including a main branch, side branches, a feature fusion layer, and an SE attention mechanism layer. The main branch consists of sequentially connected 3×3 depthwise separable convolutions and 3×3 partial convolutions. The side branches consist of sequentially connected 5×5 depthwise separable convolutions and 1×1 convolutions. The outputs of the main branch and side branches are fused by the feature fusion layer and then input to the SE attention mechanism layer to achieve feature enhancement. As shown in Figure 6, the construction process of the C2f_SDW module is as follows: the convolutional layers in the C2f module are replaced with depthwise separable convolutions, and the Bottleneck layer is replaced with StarBlock, thereby significantly reducing the number of parameters and computational complexity.

[0030] The structure of the Bidirectional Feature Pyramid Network (BiFPN) module is shown in Figure 7. By introducing the BiFPN module into the neck network, features of different scales are fused through learnable weights to improve the detection performance of small targets.

[0031] In one embodiment, as shown in Figure 8, the 2D coordinates of the key points output by the security monitoring model (YOLOv8-pose) in the image coordinate system are as follows: : Midpoint of the bottom of the ladle Midpoint of the top of the ladle The connection point between the hook and the ladle. : Suspension pivot point; Based on the above key points, the attitude estimation process of the ladle is as follows: (1) Construct two directed line segments with physical meaning: Directed line segment A (central axis of the ladle): It is formed by connecting P1 and P3. This directed line segment directly reflects the tilt direction of the ladle body, and its vector representation is: .

[0032] Directed line segment B (suspension system reference line): formed by connecting P2 and P5. This directed line segment represents the reference direction of the suspension device such as the hook and lifting lug. During normal vertical hoisting, it should be approximately perpendicular to the ground. Its vector representation is... .

[0033] (2) Calculate the direction angle of the two line segments. Calculate the tilt angle (angle with the X-axis) of each line segment in the image coordinate system. This embodiment uses the arctangent function (arctan) for calculation. Direction angle of directed line segment A. : in, It is the four-quadrant arctangent function, returning an angle within the range of [-180°, 180°], accurately reflecting direction. * (180 / π) converts radians to degrees.

[0034] Direction angle of directed line segment B : (3) Calculate the deflection angle Yaw. Calculate the difference between the direction angles of the two line segments, which is the deflection angle Yaw of the ladle: (4) Compare the calculated Yaw angle with the preset safety threshold (25° in this embodiment) and trigger the corresponding control command.

[0035] Specifically, the judgment logic is as follows: If Yaw ≤ 25°: The ladle is within the normal tilt range, the system outputs a "normal" signal, and the hoisting operation continues. If Yaw > 25°: The ladle is in an abnormal tilt state, posing a risk of molten steel tipping over. The system immediately triggers an alarm and sends an emergency stop command to the motor.

[0036] In one embodiment, the hook state is calculated based on the coordinates of the hook's center of gravity and the center of gravity of the lifting lug, as shown in Figure 9. Specifically, this includes the following steps: calculating the Euclidean distance between the two centers of gravity; if the distance is less than a preset threshold, the hook is in the hooked state; if the distance is greater than the preset threshold and the hook's center of gravity is located above the center of gravity of the lifting lug, the hook is in the pre-hooked state; when any of the above two situations occur, the hook is in the unhooked state.

[0037] To verify whether the improved YOLO v8 model can meet the requirements of edge deployment and small-scale computing device deployment, this invention also conducted related experiments, as follows: The improved YOLO v8 model was deployed on a Jetson TX1 development board for testing. This experiment equipped the development board with a Logitech industrial camera and used an STM32 microcontroller as the control platform to control a three-axis motion platform, enabling real-time detection of a ladle crane model. The system connection diagram of the Jetson TX1 deployment is shown in Figure 10. The acquired ladle video is processed in real time by the camera, and then processed and its angle is identified by the already deployed improved YOLO v8 model. After identification, the angle value is determined and communicated via serial port. The sent digital signal is recognized by the C language script embedded in the STM32 microcontroller, transmitting PWM (Pulse Width Modulation), and the digital signal output is used to control the analog circuit, realizing the emergency stop or operation of the motor.

[0038] Based on the same inventive concept, as shown in Figure 12, this embodiment of the invention provides a dynamic full-process safety monitoring device for the hook state and ladle posture of a metallurgical crane, including a pre-training module, a fine-tuning module, a target detection module, and a posture estimation module. The pre-training module is used to acquire a ladle crane image dataset during the simulated ladle lifting process based on a scaled-down crane model, denoted as the first image dataset, and to annotate key points of the ladle and the hook in the first image dataset. The pre-training module also uses the annotated first image dataset to train an improved YOLOv8 model. The system employs a training module to obtain a ladle key point detection model. A fine-tuning module acquires a small dataset of labeled ladle crane images collected from real industrial sites, designated as the second image dataset. This second image dataset is used to train the ladle key point detection model, resulting in a safety monitoring model. A target detection module inputs real-time collected ladle crane image data into the safety monitoring model, outputting the coordinates of the ladle key points, the hook center of gravity, and the lifting lug center of gravity. An attitude estimation module calculates the ladle attitude based on the output ladle key point coordinates and calculates the hook state based on the hook center of gravity and lifting lug center of gravity coordinates.

[0039] It should be noted that the full-process safety monitoring device provided in this embodiment of the invention is for implementing the above method, and its specific functions can be referred to the above method embodiments, which will not be repeated here.

[0040] Figure 13 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 13, the electronic device may include: a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304. The processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304. The processor 1301 can call logic instructions in the memory 1303 to execute a dynamic full-process safety monitoring method for the hook state and ladle posture of a metallurgical crane. This method includes: constructing a scaled-down crane model; simulating the ladle lifting process based on the scaled-down crane model; acquiring a ladle crane image dataset during the simulated ladle lifting process, denoted as the first image dataset; annotating key points of the ladle and the hook in the first image dataset; training an improved YOLOv8 model using the annotated first image dataset to obtain a ladle key point detection model; acquiring a small dataset of labeled ladle crane images collected from real industrial sites, denoted as the second image dataset; training the ladle key point detection model using the second image dataset to obtain a safety monitoring model; inputting real-time collected ladle crane image data into the safety monitoring model; outputting the ladle key point coordinates, hook center of gravity coordinates, and lug center of gravity coordinates; calculating the ladle posture based on the output ladle key point coordinates; and calculating the hook state based on the hook center of gravity coordinates and lug center of gravity coordinates.

[0041] Furthermore, when the logical instructions in the aforementioned memory 1303 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0042] This invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the dynamic full-process safety monitoring method for the hook state and ladle posture of the metallurgical crane provided in the above-described method embodiments.

[0043] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the dynamic full-process safety monitoring method for the hook state and ladle posture of the metallurgical crane provided in the above-described method embodiments.

[0044] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamic full-process safety monitoring of the hook status and ladle posture of a metallurgical crane, characterized in that, include: A scaled-down crane model was constructed, and the process of a crane lifting a steel ladle was simulated based on the scaled-down crane model. Obtain an image dataset of a steel ladle crane during the simulated process of a crane lifting a steel ladle, denoted as the first image dataset. Then, annotate the key points of the steel ladle and the hook in the first image dataset. The improved YOLOv8 model is trained using the first labeled image dataset to obtain a ladle key point detection model; a small dataset of labeled ladle crane images collected from real industrial sites is obtained and designated as the second image dataset. The ladle key point detection model is trained using the second image dataset to obtain a safety monitoring model; real-time collected ladle crane image data is input into the safety monitoring model, which outputs the coordinates of the ladle key points, the coordinates of the hook center of gravity, and the coordinates of the lug center of gravity; the ladle attitude is calculated based on the output ladle key point coordinates. Calculate the hook state based on the coordinates of the hook's center of gravity, the coordinates of the lug's center of gravity, and the positional relationship between the hook and the lug.

2. The method for dynamic full-process safety monitoring of the hook state and ladle posture of a metallurgical crane according to claim 1, characterized in that, The first image dataset is labeled with key points of the ladle, specifically including: labeling two key points P1 and P3 on the central axis of the ladle, labeling one key point P2 at the connection between the hook and the ladle, labeling one key point P4 at the ladle outlet, and labeling one key point P5 at the suspension rotating joint.

3. The method for dynamic full-process safety monitoring of the hook status and ladle posture of a metallurgical crane according to claim 1, characterized in that, The construction process of the improved YOLOv8 model includes: replacing the convolutional layers connected to the C2f modules in the YOLOv8n backbone network with Ladle Conv modules; inputting four feature maps (the first three C2f modules and the feature maps output by the SPPF module in the improved backbone network) into the neck network; for the YOLOv8n neck network, replacing all C2f modules with C2f_SDW modules, all convolutional layers with Ladle Conv modules, and all Concat layers with BiFPN modules; adding four parallel convolutional layers at the input of the neck network; adding a BiFPN module and a C2f_SDW module sequentially at the front end of the bottom-up fusion path, with the newly added C2f_SDW module and the replaced Ladle... The Ladle Conv module is connected; correspondingly, the feature fusion process of the improved back neck network includes: the four feature maps from the improved backbone network are first processed by the newly added convolutional layer to obtain four new feature maps; the first feature map in the four new feature maps is directly input into the newly added BiFPN module; the latter three feature maps are simultaneously input into the BiFPN module in the top-down fusion path and the BiFPN module in the bottom-up fusion path, respectively; the feature maps output by the three C2f_SDW modules in the bottom-up fusion path are input into the detection head; wherein, the Ladle Conv module includes a main branch, a side branch, a feature fusion layer and an SE attention mechanism layer; the main branch includes a 3×3 depthwise separable convolution and a 3×3 partial convolution connected in sequence; the side branch includes a 5×5 depthwise separable convolution and a 1×1 convolution connected in sequence; the outputs of the main branch and the side branch are fused by the feature fusion layer and then input into the SE attention mechanism layer to achieve feature enhancement; the construction process of the C2f_SDW module is: the convolutional layer in the C2f module is replaced with a depthwise separable convolution, and the Bottleneck layer is replaced with a StarBlock.

4. The method for dynamic full-process safety monitoring of the hook status and ladle posture of a metallurgical crane according to claim 2, characterized in that, Based on the output coordinates of the key points of the ladle, the ladle's attitude is calculated, specifically including: calculating directed line segment A based on the coordinates of key points P1 and P3; calculating directed line segment B based on the coordinates of key points P2 and P5; and calculating the tilt angle θ of each of the two directed line segments in the image coordinate system. A and θ B The absolute value of the difference between the two tilt angles is taken as the deflection angle of the ladle.

5. The method for dynamic full-process safety monitoring of the hook state and ladle posture of a metallurgical crane according to claim 4, characterized in that, Also includes: A deflection angle threshold is set. If the detected deflection angle of the ladle is greater than the deflection angle threshold, an alarm is triggered.

6. The method for dynamic full-process safety monitoring of the hook status and ladle posture of a metallurgical crane according to claim 1, characterized in that, Based on the coordinates of the hook's center of gravity, the coordinates of the lifting lug's center of gravity, and the positional relationship between the hook and the lifting lug, the hook state is calculated. Specifically, this includes: calculating the Euclidean distance between the two centers of gravity; if the distance is less than a preset threshold, the hook is in the suspended and positioned state; if the distance is greater than the preset threshold and the hook's center of gravity is above the lifting lug's center of gravity, the hook is in the state of approaching the ladle; when any of the above two situations occur, the hook is in the state of detaching from the ladle.

7. A dynamic full-process safety monitoring device for the hook status and ladle posture of a metallurgical crane, characterized in that, include: The pre-training module is used to acquire a steel ladle crane image dataset during the simulated steel ladle lifting process based on the built scaled crane model. This dataset is denoted as the first image dataset. The first image dataset is then labeled with key points of the steel ladle and the hook. And the improved YOLOv8 model was trained using the first labeled image dataset to obtain the ladle key point detection model; The fine-tuning module is used to acquire a small dataset of labeled images of ladle cranes collected from real industrial sites, referred to as the second image dataset. The second image dataset is used to train the ladle key point detection model to obtain a safety monitoring model. The target detection module is used to input real-time collected image data of the ladle crane into the safety monitoring model and output the coordinates of key points of the ladle, the coordinates of the hook center of gravity, and the coordinates of the lug center of gravity. The attitude estimation module is used to calculate the attitude of the ladle based on the output coordinates of the key points of the ladle; and to calculate the state of the hook based on the coordinates of the center of gravity of the hook and the center of gravity of the lifting lug.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images