Vision-based crane steel wire rope winding abnormity detection system and method

Abstract

The invention provides a crane steel wire rope winding abnormity detection system and method based on vision, and the system comprises an image collection module which is installed at a position where a crane roller is located and is used for collecting an image of a steel wire rope winding condition in a hoisting process of the crane roller in real time; the light source module is arranged at the position of the image acquisition module and is used for providing illumination for the image acquisition module so as to ensure the image acquisition quality; the image processing module is connected with the image acquisition module, and is used for receiving the image acquired by the image acquisition module, processing and identifying image information by adopting a neural network model, and carrying out winding abnormity detection so as to detect whether the state of the steel wire rope is normal or not; and the man-machine interaction module is connected with the image processing module and is used for receiving the retrieval result of the image processing module and obtaining final anomaly detection judgment, and if the steel wire rope is abnormal, the man-machine interaction module is used for giving an alarm and controlling a crane roller to take corresponding shutdown actions.

Description

Technical Field

[0001] The present invention relates to the technical field of hoisting monitoring, and in particular to a vision-based crane wire rope winding anomaly detection system and method. Background Art

[0002] With the continuous development of my country's economy and the continuous improvement of the industrial field, the demand for the use of special equipment is getting higher and higher. Among them, cranes are an important special equipment, which play an extremely important role in industries involving the handling of large equipment and objects such as ports, hydropower stations, bridges and construction. Cranes refer to lifting machinery that can carry materials vertically and horizontally within a certain range. They have the working characteristics of intermittent movement, that is, their work is a cycle of actions such as material collection, transfer, and unloading. Commonly used large lifting machinery include tower cranes, gantry cranes, bridge cranes, crawler cranes and tire cranes.

[0003] Crane is a heavy machine with a complex working environment. Generally speaking, the more complex the equipment is, the higher the possibility of failure. For heavy machinery such as crane, any failure may lead to very serious consequences, causing certain economic losses and even endangering personal safety.

[0004] In order to reduce the number of accidents and ensure the personal safety of employees, many companies and institutions at home and abroad have begun to conduct research on the diagnosis and prevention of crane failures. After decades of continuous development, crane fault detection has gradually become a complete discipline system. In the field of fault detection, foreign countries started earlier. For example, the United States, Germany, Britain, France and Japan have relatively mature fault diagnosis products. American companies such as Scientific and Lip have established fault diagnosis technology research centers and have developed mature state detection and fault diagnosis systems. Caterpillar has developed a computer monitoring system with fault detection capabilities, and has also realized the working state monitoring, diagnosis and remote control of mechanical equipment; Germany's LIEBHERR company has a relatively strong market in the field of cranes, and has developed a number of state monitoring systems including video monitoring systems, operating parameter monitoring systems, torque limiting systems, anti-collision systems, etc., and has applied them to a series of its crane products. The state parameters of the machinery can be viewed through electronic displays; and in China, in the past 20 years, some leading companies have jointly conducted some research on the fault detection of heavy machinery with universities. Professor Liu Chengliang's team from Shanghai Jiao Tong University conducted research on real-time status monitoring and fault diagnosis technology for large crawler cranes, mainly using GPRS / GSM wireless communication technology and GPS positioning technology to study key issues such as data transmission and remote monitoring in fault diagnosis.

[0005] In the current fault diagnosis research, there is relatively little research on abnormal winding of crane wire ropes, including tangled ropes, loose ropes, and skipped ropes. There is even less research on how to achieve intelligent detection of abnormal winding of wire ropes in complex industrial environments such as hydropower stations. If the crane wire rope is abnormally wound during the lifting process, it will also bring serious consequences. In order to protect the lives of hydropower station employees and promote safe production, it is necessary to conduct further research on abnormal winding monitoring of crane wire ropes. Summary of the invention

[0006] The present invention provides a vision-based crane wire rope winding anomaly detection system and method. This method uses an industrial camera to collect images of a wire rope reel, which are then transmitted to an industrial computer. A neural network model is used to perform winding anomaly detection to obtain a final anomaly detection judgment. If an abnormality occurs in the wire rope, an alarm is sounded through a human-computer interaction device, thereby achieving real-time detection of winding anomalies, avoiding safety hazards, and ensuring the safety of the lifting operation process.

[0007] In order to achieve the above technical features, the purpose of the present invention is to achieve: a vision-based crane wire rope winding abnormality detection system, comprising: An image acquisition module is installed at the location of the crane drum and is used to collect real-time images of the winding of the wire rope of the crane drum during the hoisting process; The light source module is arranged at the position where the image acquisition module is located and is used to provide lighting for the image acquisition module to ensure the image acquisition quality; An image processing module is connected to the image acquisition module, and is used to receive the image acquired by the image acquisition module, and use a neural network model to process and identify the image information, and perform winding anomaly detection to detect whether the wire rope is in a normal state; The human-computer interaction module is connected to the image processing module and is used to receive the retrieval results of the image processing module to obtain the final abnormality detection judgment. If the wire rope is abnormal, the human-computer interaction module will sound an alarm and control the crane drum to take corresponding shutdown actions.

[0008] The image acquisition module adopts an industrial camera or an industrial camera head, and is installed on the wall of the trolley room and the left and right rotary crane room through a fixed base arrangement, and the industrial camera or industrial camera head is aimed at the crane drum.

[0009] The industrial camera or industrial camera head adopts a top-down angle to shoot and obtain the image of the wire rope on the crane drum.

[0010] The light source module adopts a bar-shaped light source, which is used to generate soft and uniform light and is adjusted to a suitable position and angle to avoid interference from natural light.

[0011] A reflector is installed at the location of the light source module to disperse the light and reduce the highlight phenomenon on the target object.

[0012] A vision-based crane wire rope winding abnormality detection method, the detection method is implemented by a detection system, and includes the following steps: Step 1: Collect image dataset: A sufficient number of normal and abnormal wire rope winding samples are collected through the image acquisition module to form an image data set; Step 2, image resizing: Resize the images in the image dataset to ensure that all input images meet the uniform size required by the YOLO model to ensure the consistency and efficiency of subsequent YOLO model training; Step 3: Manually calibrate the abnormal status: LabelImg software was used to manually calibrate the abnormal state of the acquired images, and the tangled rope, loose rope, and skipping rope areas in the images were manually marked to identify the abnormal states represented by these areas, and a txt file dataset in YOLO format with precise annotation information was generated; Step 4: Data enhancement processing: A series of random and programmed modifications are performed on the original images of the manually calibrated dataset, including flipping, rotating, scaling, cropping, color adjustment, and adding noise operations, thereby generating a large number of new samples that are similar to the original images but different from them, thereby expanding the scale and diversity of the training set; Step 5, YOLOv8 model training and verification: The dataset created after data augmentation is input into the YOLOv8 neural network model for improved training and tested on the validation set; the model continuously iterates the loss function through gradient descent to complete the fitting of the dataset and complete the training of the model; Step 6: Compression: Perform parameter pruning and sharing, and knowledge distillation and compression on the trained model; Step 7: Deploy the application locally: Import the trained and adjusted model into the local industrial computer to perform real-time detection of abnormal conditions of the wire rope.

[0013] The step 1 specifically includes: Step 1.1: Install the industrial camera or industrial camera on the wall of the trolley room and the left and right rotary crane room. The installation method is fixed base installation, and the connection method with the image processing module is wired connection; Step 1.2, obtain the wire rope image from a bird's-eye view through an industrial camera or industrial camera. In addition, in order to improve the image quality when collecting the image data set, it is necessary to install a light source to reduce the interference of natural light to assist in shooting; Step 1.3, use a soft and uniform light source and adjust it to a suitable position and angle. Use a reflector to help disperse the light, avoid interference from natural light, and reduce the highlight phenomenon on the target; Step 1.4, collect sufficient normal and abnormal wire rope winding samples under the correct light source environment.

[0014] The step 2 specifically includes: Step 2.1, first determine a target size, which is usually the input size recommended or required by the neural network model. The main operations include cropping and scaling. Scaling directly adjusts the image size to the target size and uses bilinear interpolation method to maintain image quality; When resizing an image, the bounding box coordinates of the object in the image also need to be adjusted according to the same scaling ratio. Assuming that the original image size is (width, height) and the resized size is (new_width, new_height), for the coordinates of each bounding box (x, y, width, height), the adjustment formula is as follows: x_new = x * (new_width / width); y_new = y * (new_height / height); width_new = width * (new_width / width); height_new = height * (new_height / height); Where (x_new, y_new, width_new, width_new, height_new) are the coordinates of the adjusted bounding box; This operation ensures that the coordinates of the bounding box always correspond to the actual position of the adjusted image; Step 2.2, normalize the adjusted coordinates to the interval [0, 1] to facilitate model processing. This step is completed before the final input to the model. The formula is: normalized_x = x_new / new_width; normalized_y= y_new / new_height; normalized_width= width_new / new_width; normalized_height= height_new / new_height; Where (normalized_x, normalized_y, normalized_width, normalized_height) are the coordinates of the bounding box after normalization.

[0015] In the process of image resizing in step 2, if the resizing is performed in Python, the resizing is implemented using a function of the PIL or OpenCV library; If the policy allows, you can also choose to crop the image to the target size to ensure that key target information is not lost; or you need to scale the image first to maintain its aspect ratio, and then crop or pad the edges to reach the target size to avoid distorting the image.

[0016] The step 3 specifically includes: Step 3.1, first install LabelImg in the Python environment; Step 3.2, create a directory structure for the annotation project and configure the output format to be a txt file in YOLO format; Step 3.3, run LabelImg, click and drag on the image to draw a bounding box containing the abnormal state to mark the tangled rope, loose rope, and skipping rope areas. Make sure that the bounding box accurately surrounds the abnormal area during labeling, which directly affects the effect of subsequent model learning. Finally, save the labeling.

[0017] In the specific implementation of step 5, first download the YOLOv8 model to the local computer and configure the environment, and then import the previously collected and produced data set into the model for training. The specific steps are as follows: Step 5.1, download the YOLOv8 project from github; Step 5.2, configure the relevant environment including the dependent Python environment and pytorch, and install the required dependencies; Step 5.3, create your own yaml file in the datasets folder according to the established dataset, and then change the categories in the network model configuration file to your actual number of categories; Step 5.4, modify the training parameter configuration file; Step 5.5, start model training. During or after training, use the validation set to evaluate the model performance by calling the .val() method of the model. The following four operations are performed in step 6: Parameter pruning and sharing, the core of which is to delete parameters that have little impact on prediction accuracy, is applied to the convolutional and fully connected layers of the model. The pruning process first requires choosing the type of pruning: pruning the entire neuron, filter, channel, or weight. Then, the importance of each parameter in the model is evaluated using criteria, specifically calculating the absolute value of the weight, the size of the gradient, and the first or second order derivative of the Taylor expansion. Then, pruning is performed at the beginning, and then the pruning ratio is gradually increased, while monitoring the model performance to ensure that the accuracy does not drop significantly. The model is then fine-tuned to restore the accuracy lost due to pruning. Finally, the pruning and fine-tuning process is repeated until the ideal balance between model size and performance is reached. Low-rank factor decomposition uses matrices to decompose and estimate parameters. It is mainly used in the convolutional layer and fully connected layer of the model. Low-rank decomposition sparses the convolution kernel matrix by merging dimensions and imposing low-rank constraints. Based on the fact that most weight vectors are distributed in low-rank subspaces, a small number of basis vectors are selected to reconstruct the convolution kernel matrix to reduce storage space. The specific operations are binary and multivariate decomposition. The binary operation uses two K*1 convolution kernels to replace a K*K convolution kernel. The multivariate operation uses SVD decomposition, Tucker decomposition, and Block Term decomposition. Transfer / compact convolution filters, which save model parameters by designing convolution kernels, including Depthwise Convolution combined with Pointwise Convolution in the MobileNet series, or channel reordering and grouped convolution strategies in ShuffleNet to extract relevant features. This method is only applied to the convolution layer of the model; Knowledge distillation, training a more compact neural network to distill knowledge from a large model, is applied to the convolutional and fully connected layers of the model.

[0018] The step 6 specifically includes: Step 6.1, first, use a large amount of data and computing resources to train a high-performance teacher model, design a student model with a simple structure and few parameters, and use the teacher model to predict the training data set, but instead of directly using hard labels, use soft targets for distillation; Step 6.2, then train the student model, and its loss function contains two parts: one is the gap between the student model prediction result and the true label, and the other is the gap between the student model prediction result and the teacher model soft target; adjust the parameters of the student model by minimizing the weighted sum of these two parts of the loss; then adjust and optimize the parameters, and finally, after the training is completed, evaluate the performance of the student model on the validation set; if necessary, fine-tune to further improve the performance.

[0019] The step 7 specifically includes: first, the crane starts working, an industrial camera or an industrial camera takes a real-time image of the wire rope of the gantry crane lifting the heavy object, and transmits it to the local industrial computer for image processing. The device processes and identifies the image information and detects whether the status of the wire rope is normal. If the wire rope is abnormal, an alarm will be issued through the human-computer interaction module to remind the staff or control the shutdown to prevent accidents. If it is normal, the detection will continue until the lifting work is completed.

[0020] Beneficial effects of the present invention: 1. The present invention uses an industrial camera to collect images of the wire rope reel, and then transmits them to an industrial computer. A neural network model is used to detect winding anomalies to obtain a final abnormality detection judgment. If the wire rope is abnormal, an alarm is issued through a human-computer interaction module, thereby achieving real-time detection of winding anomalies and avoiding safety hazards.

[0021] 2. Invented and established a complete visual detection and intelligent alarm system.

[0022] 3. The visual algorithm adopted by the present invention has higher recognition accuracy and detection speed.

[0023] 4. The visual algorithm of the present invention has higher applicability and versatility.

[0024] 5. The model of the present invention provides a wealth of hyperparameters and model structure options, which facilitates users to perform customized adjustments and optimizations.

[0025] 6. The model of the present invention provides a more friendly installation and operation mode, making deployment and application more convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0027] Figure 1 It is a system diagram of the present invention.

[0028] Figure 2 This is the general process of training a more compact neural network in the present invention to distill knowledge from a large model.

[0029] Figure 3 This is the detection process of the present invention.

[0030] Figure 4 It is a diagram of the experimental results of the present invention.

[0031] Figure 5 It is the interface of the anomaly detection system of the present invention.

[0032] Figure 6 (a) and (b) are actual pictures of the industrial control computer of the present invention.

[0033] Figure 7 It is the test result diagram of the present invention. DETAILED DESCRIPTION

[0034] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0035] Embodiment 1: See also Figure 1 This embodiment provides a vision-based crane wire rope winding anomaly detection system, including: An image acquisition module is installed at the location of the crane drum and is used to collect real-time images of the winding of the wire rope of the crane drum during the hoisting process; The light source module is arranged at the position where the image acquisition module is located and is used to provide lighting for the image acquisition module to ensure the image acquisition quality; An image processing module is connected to the image acquisition module, and is used to receive the image acquired by the image acquisition module, and use a neural network model to process and identify the image information, and perform winding anomaly detection to detect whether the wire rope is in a normal state; The human-computer interaction module is connected to the image processing module and is used to receive the retrieval results of the image processing module to obtain the final abnormality detection judgment. If the wire rope is abnormal, the human-computer interaction module will sound an alarm and control the crane drum to take corresponding shutdown actions.

[0036] By adopting the above-mentioned abnormality detection system, it can be used to detect the abnormal state of the winding of the crane wire rope. It realizes image detection of the crane drum through the image acquisition module, and transmits the image to the image processing module. The image is processed by the image processing module to determine whether there is an abnormal winding situation. When there is an abnormal situation, an alarm is issued through the human-computer interaction module or the crane drum is adjusted through feedback to control the automatic shutdown of the crane drum.

[0037] Furthermore, the image acquisition module uses an industrial camera or an industrial camera, and is installed on the wall of the trolley room and the left and right rotary crane room through a fixed base, and the industrial camera or industrial camera is aimed at the crane drum. The above-mentioned image acquisition module can collect real-time images of the crane drum, so as to facilitate subsequent recognition and analysis of the image through the YOLOv8 model.

[0038] Furthermore, the industrial camera or industrial camera head adopts a top-down angle to capture the image of the wire rope on the crane drum. By adopting the above-mentioned shooting method, the winding situation of the wire rope on the crane drum can be maximized.

[0039] Furthermore, the light source module adopts a bar light source, which is used to generate soft and uniform light and is adjusted to a suitable position and angle to avoid interference from natural light. The quality of image acquisition can be guaranteed by the above light source module.

[0040] Furthermore, a reflector is arranged and installed at the location of the light source module to disperse the light and reduce the highlight phenomenon on the target object.

[0041] Embodiment 2: A vision-based crane wire rope winding abnormality detection method, characterized in that the detection method is implemented using the detection system, and includes the following steps: Step 1: Collect image dataset: The image acquisition module collects sufficient normal and abnormal samples of wire rope winding and forms an image data set, which includes: Step 1.1, install the industrial camera or industrial camera on the wall of the trolley room and the left and right rotary crane room. The installation method is fixed base installation, and the connection method with the image processing module is wired connection; Step 1.2, obtain the wire rope image from a bird's-eye view through an industrial camera or industrial camera. In addition, in order to improve the image quality when collecting the image data set, it is necessary to install a light source to reduce the interference of natural light to assist in shooting; Step 1.3, use a soft and uniform light source and adjust it to a suitable position and angle. Use a reflector to help disperse the light, avoid interference from natural light, and reduce the highlight phenomenon on the target; Step 1.4, collect sufficient normal and abnormal wire rope winding samples under the correct light source environment.

[0042] Step 2, image resizing: When training the YOLOv8 model, it is a common preprocessing step to resize the images in the dataset to ensure that all input images meet the uniform size required by the model. This operation is crucial to maintain the consistency and efficiency of model training.

[0043] Therefore, it is necessary to resize the images in the image dataset to ensure that all input images meet the uniform size required by the YOLO model to ensure the consistency and efficiency of subsequent YOLO model training; specifically, it includes: Step 2.1, first determine a target size, which is usually the input size recommended or required by the neural network model. For YOLOv8, this size is 640x640 pixels. The main operations include cropping and scaling. Scaling directly adjusts the image size to the target size. Bilinear interpolation or other interpolation methods can be used to maintain image quality. In Python, this can be implemented using functions of the PIL (Pillow library) or OpenCV library, such as the resize() function. If the strategy allows, you can also choose to crop the image to the target size to ensure that key target information is not lost. In some cases, you may need to scale the image first to maintain its aspect ratio, and then crop or fill the edges to reach the target size to avoid distorting the image.

[0044] When resizing an image, the bounding box coordinates of the object in the image also need to be adjusted according to the same scaling ratio. Assuming that the original image size is (width, height) and the resized size is (new_width, new_height), for the coordinates of each bounding box (x, y, width, height), the adjustment formula is as follows: x_new = x * (new_width / width); y_new = y * (new_height / height); width_new = width * (new_width / width); height_new = height * (new_height / height); Where (x_new, y_new, width_new, width_new, height_new) are the coordinates of the adjusted bounding box; This operation ensures that the coordinates of the bounding box always correspond to the actual position of the adjusted image; Step 2.2, normalize the adjusted coordinates to the interval [0, 1] to facilitate model processing. This step is completed before the final input to the model. The formula is: normalized_x = x_new / new_width; normalized_y= y_new / new_height; normalized_width= width_new / new_width; normalized_height= height_new / new_height; Where (normalized_x, normalized_y, normalized_width, normalized_height) are the coordinates of the bounding box after normalization.

[0045] Step 3: Manually calibrate the abnormal status: LabelImg software was used to manually calibrate the abnormal state of the acquired images, and the tangled rope, loose rope, and skipping rope areas in the images were manually marked to identify the abnormal states represented by these areas, and a txt file dataset in YOLO format with precise annotation information was generated; Step 3.1, first install LabelImg in the Python environment; Step 3.2, create a directory structure for the annotation project and configure the output format to be a txt file in YOLO format; Step 3.3, run LabelImg, click and drag on the image to draw a bounding box containing the abnormal state to mark the tangled rope, loose rope, and skipping rope areas. Make sure that the bounding box accurately surrounds the abnormal area during labeling, which directly affects the effect of subsequent model learning. Finally, save the labeling.

[0046] Step 4: Data enhancement processing: A series of random and programmed modifications are performed on the original images of the manually calibrated dataset, including flipping, rotating, scaling, cropping, color adjustment, and adding noise operations, thereby generating a large number of new samples that are similar to the original images but different from them, thereby expanding the scale and diversity of the training set; Step 5, YOLOv8 model training and verification: The dataset created after data augmentation is fed into the YOLOv8 neural network model for improved training and tested on the validation set. YOLOv8 is the latest in the YOLO-based object detection model family from Ultra-ytics, providing state-of-the-art performance. Compared to previous YOLO versions, the YOLOv8 model is faster and more accurate, while providing a unified framework for training models to perform tasks such as object detection, instance segmentation, and image classification.

[0047] Yolov8 is a SOTA model that builds on the historical versions of the Yolo series and introduces new features and improvements to further improve performance and flexibility, making it one of the best choices for tasks such as target detection, image segmentation, and pose estimation. Its specific innovations include a new backbone network, a new Ancher-Free detection head, and a new loss function, which can run on a variety of hardware platforms from CPU to GPU.

[0048] It can be roughly divided into three parts: 1) Backbone is responsible for extracting features from the input image and converting the image into a feature representation with rich semantic information. The backbone network as a whole implements feature extraction through four parts. The first part extracts the feature information of the input image through two convolutional layers, and then adds a C2f module to achieve feature fusion. Then there are two repeated parts, each consisting of a convolutional layer and a C2f module. The last part consists of a convolutional layer + C2f module + SPPF module. After completing the data feature extraction, the SPPF module helps complete the featherMap-level fusion of local features and global features.

[0049] 2) Neck (neck, connection part) is an intermediate layer used to fuse features from the backbone to improve the performance of the model. There is no Neck part in Yolov8, but there is a PANet functional module that also completes feature fusion. It is located in the front half of the head network structure and can also be divided into the neck part.

[0050] 3) Head (task head) is the last layer of the model, that is, the output layer of the model, and its structure will vary according to different tasks. For example, in image classification tasks, we usually use a softmax classifier as the Head, while in the target detection task of the present invention, we use a bounding box regressor and a classifier as the Head. In addition, when training this model, it is necessary to set a loss function, which consists of multiple parts, including classification loss (VFL Loss) and regression loss in the form of CIOU Loss + DFL. The classification loss function adopts VFL Loss (improved cross entropy loss), and the cross entropy loss (Cross-Entropy Loss), also known as logarithmic loss (Logarithmic Loss), is a loss function commonly used in classification problems. It measures the difference between the probability distribution predicted by the model and the actual label, and is used to measure the accuracy of the model's prediction. The calculation formula is as follows:

[0051] In the formula, q is the IoU (intersection over union) of bbox (prediction box) and gt (real box), IoU is the intersection of the prediction box and the real box divided by the union of the two boxes, and then p is the probability; then the two boxes intersect, that is, q>0, which is a positive sample, and the two boxes have no intersection, then let q=0, which is a negative sample.

[0052] The IOU loss function calculation formula is as follows: =1-IoU+ + v; In this formula, IoU is the intersection over union ratio, b and bgt represent the center points of the two rectangular boxes, ρ represents the Euclidean distance between the two rectangular boxes, c represents the diagonal distance of the closed area of ​​the two rectangular boxes, v is used to measure the consistency of the relative proportions of the two rectangular boxes, and α is the weight coefficient.

[0053] The model continuously iterates the loss function through gradient descent to complete the fitting of the data set and complete the training of the model.

[0054] In the specific implementation, first download the YOLOv8 model to the local computer and configure the environment, and then import the previously collected data set into the model for training. The general steps are as follows: Step 5.1, download the YOLOv8 project from github; Step 5.2, configure the relevant environment including the dependent Python environment and pytorch, and install the required dependencies; Step 5.3, create your own yaml file in the datasets folder according to the established dataset, and then change the categories in the network model configuration file to your actual number of categories; Step 5.4, modify the training parameter configuration file; Step 5.5, start model training. During or after training, use the validation set to evaluate the model performance by calling the .val() method of the model. Step 6: Compression: Compression processing plays a vital role in the fields of deep learning and neural networks, which can reduce the size of the model and improve operating efficiency.

[0055] The present invention performs compression processing such as parameter pruning and sharing, knowledge distillation, etc. on the trained model. The following four operations are mainly performed: Parameter pruning and sharing, the core of which is to delete parameters that have little impact on prediction accuracy, is applied to the convolutional and fully connected layers of the model. The pruning process first requires choosing the type of pruning: pruning the entire neuron, filter, channel, or weight. Then, the importance of each parameter in the model is evaluated using criteria, specifically calculating the absolute value of the weight, the size of the gradient, and the first or second order derivative of the Taylor expansion. Then, pruning is performed at the beginning, and then the pruning ratio is gradually increased, while monitoring the model performance to ensure that the accuracy does not drop significantly. The model is then fine-tuned to restore the accuracy lost due to pruning. Finally, the pruning and fine-tuning process is repeated until the ideal balance between model size and performance is reached. Low-rank factor decomposition uses matrices to decompose and estimate parameters. It is mainly used in the convolutional layer and fully connected layer of the model. Low-rank decomposition sparses the convolution kernel matrix by merging dimensions and imposing low-rank constraints. Based on the fact that most weight vectors are distributed in low-rank subspaces, a small number of basis vectors are selected to reconstruct the convolution kernel matrix to reduce storage space. The specific operations are binary and multivariate decomposition. The binary operation uses two K*1 convolution kernels to replace a K*K convolution kernel. The multivariate operation uses SVD decomposition, Tucker decomposition, and Block Term decomposition. Transfer / compact convolution filters, which save model parameters by designing convolution kernels, including Depthwise Convolution combined with Pointwise Convolution in the MobileNet series, or channel reordering and grouped convolution strategies in ShuffleNet to extract relevant features. This method is only applied to the convolution layer of the model; Knowledge distillation, training a more compact neural network to distill knowledge from a large model, applied to the convolutional and fully connected layers of the model, see the process Figure 2 .

[0056] First, use a large amount of data and computing resources to train a high-performance large model (teacher model), design a model with a simpler structure and fewer parameters (student model), and use the teacher model to predict the training data set, but do not directly use hard labels (such as one-hot encoding). Instead, soft targets are used for distillation.

[0057] The student model is then trained, and its loss function usually consists of two parts: one is the gap between the student model's prediction and the true label (hard loss), and the other is the gap between the student model's prediction and the teacher model's soft target (soft loss). The parameters of the student model are adjusted by minimizing the weighted sum of these two losses. Parameter adjustment and optimization are then performed, and finally, after training is completed, the performance of the student model on the validation set is evaluated. If necessary, fine-tuning can be performed to further improve performance.

[0058] Step 7: Deploy the application locally: Import the trained and adjusted model into the local industrial computer to perform real-time detection of abnormal conditions of the wire rope.

[0059] First, the crane starts working. The industrial camera or industrial camera takes real-time images of the wire rope of the gantry crane lifting the heavy object and transmits them to the local industrial computer for image processing. The device processes and identifies the image information and detects whether the wire rope is normal. If the wire rope is abnormal, the human-computer interaction module will sound an alarm to remind the staff or control the shutdown to prevent accidents. If it is normal, continue to detect until the lifting work is completed. Figure 3 .

[0060] Embodiment 3: In this embodiment, the experiment adopts a deep learning framework based on the Pytorch2.0.1 environment, the software environment is Windows10, python3.8.19, CUDA11.7, the training graphics card is configured as NVIDIA GeForce GTX 1660 Ti, the system running memory is 16G, and the processor is AMD Ryzen 5 4600H with Radeon Graphics 3.00 GHz.

[0061] The training batch size was set to 16, and the training was performed 100 times. No weights were used. The specific network hyperparameter settings are shown in Table 1. The experimental results are shown in Figure 4 .

[0062] Table 1 Network hyperparameter settings

[0063] The software development of the wire rope anomaly detection system is completed based on the PyQt5 database and PyCharm software. The system mainly consists of four modules: wire rope drum image acquisition module, image processing module, system data management module and human-computer interaction interface. Figure 5 The interface for the anomaly detection system.

[0064] To verify the actual operation of the wire rope anomaly detection system, this embodiment deploys the system on an industrial computer for experimental verification. The parameters of the industrial computer are shown in Table 2. Figure 6 The test data set contains 100 wire rope images, and the experimental results are shown in Table 3.

[0065] Table 2 Industrial computer parameters

[0066] Table 3 System integration test results

[0067] As can be seen from Table 3, the model has high accuracy, with an average inference speed of 19.2ms per image, an average non-maximum suppression processing speed of 1.4ms, and a detection speed of 46.1FPS. The detection results are as follows Figure 7 .

[0068] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A vision-based crane wire rope winding anomaly detection system, characterized in that: include: An image acquisition module is installed at the location of the crane drum and is used to collect real-time images of the winding of the wire rope of the crane drum during the hoisting process; The light source module is arranged at the position where the image acquisition module is located and is used to provide lighting for the image acquisition module to ensure the image acquisition quality; An image processing module is connected to the image acquisition module, and is used to receive the image acquired by the image acquisition module, and use a neural network model to process and identify the image information, and perform winding anomaly detection to detect whether the wire rope is in a normal state; The human-computer interaction module is connected to the image processing module and is used to receive the retrieval results of the image processing module to obtain the final abnormality detection judgment. If the wire rope is abnormal, the human-computer interaction module will sound an alarm and control the crane drum to take corresponding shutdown actions.

2. According to claim 1, a vision-based crane wire rope winding anomaly detection system is characterized in that: The image acquisition module adopts an industrial camera or an industrial camera head, and is installed on the wall of the trolley room and the left and right rotary crane room through a fixed base arrangement, and the industrial camera or industrial camera head is aimed at the crane drum.

3. According to claim 2, a vision-based crane wire rope winding anomaly detection system is characterized in that: The industrial camera or industrial camera head adopts a top-down angle to shoot and obtain the image of the wire rope on the crane drum.

4. The vision-based crane wire rope winding anomaly detection system according to claim 1 is characterized in that: The light source module adopts a bar-shaped light source, which is used to generate soft and uniform light and is adjusted to a suitable position and angle to avoid interference from natural light.

5. The vision-based crane wire rope winding anomaly detection system according to claim 4 is characterized in that: A reflector is installed at the location of the light source module to disperse the light and reduce the highlight phenomenon on the target object.

6. A vision-based crane wire rope winding anomaly detection method, characterized in that: The detection method is implemented by using the detection system according to any one of claims 1 to 5, comprising the following steps: Step 1: Collect image dataset: A sufficient number of normal and abnormal wire rope winding samples are collected through the image acquisition module to form an image data set; Step 2, image resizing: Resize the images in the image dataset to ensure that all input images meet the uniform size required by the YOLO model to ensure the consistency and efficiency of subsequent YOLO model training; Step 3: Manually calibrate the abnormal status: LabelImg software was used to manually calibrate the abnormal state of the acquired images, and the tangled rope, loose rope, and skipping rope areas in the images were manually marked to identify the abnormal states represented by these areas, and a txt file dataset in YOLO format with precise annotation information was generated; Step 4: Data enhancement processing: A series of random and programmed modifications are performed on the original images of the manually calibrated dataset, including flipping, rotating, scaling, cropping, color adjustment, and adding noise operations, thereby generating a large number of new samples that are similar to the original images but different from them, thereby expanding the scale and diversity of the training set; Step 5, YOLOv8 model training and verification: The dataset created after data augmentation is input into the YOLOv8 neural network model for improved training and tested on the validation set; the model continuously iterates the loss function through gradient descent to complete the fitting of the dataset and complete the training of the model; Step 6: Compression: Perform parameter pruning and sharing, and knowledge distillation and compression on the trained model; Step 7: Deploy the application locally: Import the trained and adjusted model into the local industrial computer to perform real-time detection of abnormal conditions of the wire rope.

7. A method for detecting abnormal winding of crane wire rope based on vision according to claim 6, characterized in that: The step 1 specifically includes: Step 1.1: Install the industrial camera or industrial camera on the wall of the trolley room and the left and right rotary crane room. The installation method is fixed base installation, and the connection method with the image processing module is wired connection; Step 1.2, obtain the wire rope image from a bird's-eye view through an industrial camera or industrial camera. In addition, in order to improve the image quality when collecting the image data set, it is necessary to install a light source to reduce the interference of natural light to assist in shooting; Step 1.3, use a soft and uniform light source and adjust it to a suitable position and angle. Use a reflector to help disperse the light, avoid interference from natural light, and reduce the highlight phenomenon on the target; Step 1.4, collect sufficient normal and abnormal wire rope winding samples under the correct light source environment.

8. The vision-based crane wire rope winding anomaly detection method according to claim 6, characterized in that: The step 2 specifically includes: Step 2.1, first determine a target size, which is usually the input size recommended or required by the neural network model. The main operations include cropping and scaling. Scaling directly adjusts the image size to the target size and uses bilinear interpolation method to maintain image quality; When resizing an image, the bounding box coordinates of the object in the image also need to be adjusted according to the same scaling ratio. Assuming that the original image size is (width, height) and the resized size is (new_width, new_height), for the coordinates of each bounding box (x, y, width, height), the adjustment formula is as follows: x_new = x * (new_width / width); y_new = y * (new_height / height); width_new = width * (new_width / width); height_new = height * (new_height / height); Where (x_new, y_new, width_new, width_new, height_new) are the coordinates of the adjusted bounding box; This operation ensures that the coordinates of the bounding box always correspond to the actual position of the adjusted image; Step 2.2, normalize the adjusted coordinates to the interval [0, 1] to facilitate model processing. This step is completed before the final input to the model. The formula is: normalized_x = x_new / new_width; normalized_y= y_new / new_height; normalized_width= width_new / new_width; normalized_height= height_new / new_height; Where (normalized_x, normalized_y, normalized_width, normalized_height) are the coordinates of the bounding box after normalization.

9. The method for detecting abnormal winding of crane wire rope based on vision according to claim 8, characterized in that: In the process of image resizing in step 2, if the resizing is performed in Python, the resizing is implemented using a function of the PIL or OpenCV library; If the policy allows, you can also choose to crop the image to the target size to ensure that key target information is not lost; or you need to scale the image first to maintain its aspect ratio, and then crop or pad the edges to reach the target size to avoid distorting the image.

10. A method for detecting abnormal winding of crane wire ropes based on vision according to claim 9, characterized in that: The step 3 specifically includes: Step 3.1, first install LabelImg in the Python environment; Step 3.2, create a directory structure for the annotation project and configure the output format to be a txt file in YOLO format; Step 3.3, run LabelImg, click and drag on the image to draw a bounding box containing the abnormal state to mark the tangled rope, loose rope, and skipping rope areas. Make sure that the bounding box accurately surrounds the abnormal area during labeling, which directly affects the effect of subsequent model learning. Finally, save the labeling.

11. The vision-based crane wire rope winding anomaly detection method according to claim 9, characterized in that: In the specific implementation of step 5, first download the YOLOv8 model to the local computer and configure the environment, and then import the previously collected and produced data set into the model for training. The specific steps are as follows: Step 5.1, download the YOLOv8 project from github; Step 5.2, configure the relevant environment including the dependent Python environment and pytorch, and install the required dependencies; Step 5.3, create your own yaml file in the datasets folder according to the established dataset, and then change the categories in the network model configuration file to your actual number of categories; Step 5.4, modify the training parameter configuration file; Step 5.5, start model training. During or after training, use the validation set to evaluate the model performance by calling the .val() method of the model for verification.

12. The vision-based crane wire rope winding anomaly detection method according to claim 9, characterized in that: The following four operations are performed in step 6: Parameter pruning and sharing, the core of which is to delete parameters that have little impact on prediction accuracy, is applied to the convolutional and fully connected layers of the model. The pruning process first requires choosing the type of pruning: pruning the entire neuron, filter, channel, or weight. Then, the importance of each parameter in the model is evaluated using criteria, specifically calculating the absolute value of the weight, the size of the gradient, and the first or second order derivative of the Taylor expansion. Then, pruning is performed at the beginning, and then the pruning ratio is gradually increased, while monitoring the model performance to ensure that the accuracy does not drop significantly. The model is then fine-tuned to restore the accuracy lost due to pruning. Finally, the pruning and fine-tuning process is repeated until the ideal balance between model size and performance is reached. Low-rank factor decomposition uses matrices to decompose and estimate parameters. It is mainly used in the convolutional layer and fully connected layer of the model. Low-rank decomposition sparses the convolution kernel matrix by merging dimensions and imposing low-rank constraints. Based on the fact that most weight vectors are distributed in low-rank subspaces, a small number of basis vectors are selected to reconstruct the convolution kernel matrix to reduce storage space. The specific operations are binary and multivariate decomposition. The binary operation uses two K*1 convolution kernels to replace a K*K convolution kernel. The multivariate operation uses SVD decomposition, Tucker decomposition, and Block Term decomposition. Transfer / compact convolution filters, which save model parameters by designing convolution kernels, including Depthwise Convolution combined with Pointwise Convolution in the MobileNet series, or channel reordering and grouped convolution strategies in ShuffleNet to extract relevant features. This method is only applied to the convolution layer of the model; Knowledge distillation, training a more compact neural network to distill knowledge from a large model, is applied to the convolutional and fully connected layers of the model.

13. A method for detecting abnormal winding of crane wire ropes based on vision according to claim 12, characterized in that: The step 6 specifically includes: Step 6.1, first, use a large amount of data and computing resources to train a high-performance teacher model, design a student model with a simple structure and few parameters, and use the teacher model to predict the training data set, but instead of directly using hard labels, use soft targets for distillation; Step 6.2, then train the student model, and its loss function contains two parts: one is the gap between the student model prediction result and the true label, and the other is the gap between the student model prediction result and the teacher model soft target; adjust the parameters of the student model by minimizing the weighted sum of these two parts of the loss; then adjust and optimize the parameters, and finally, after the training is completed, evaluate the performance of the student model on the validation set; if necessary, fine-tune to further improve the performance.

14. The vision-based crane wire rope winding anomaly detection method according to claim 12, characterized in that: The step 7 specifically includes: first, the crane starts working, an industrial camera or an industrial camera takes a real-time image of the wire rope of the gantry crane lifting the heavy object, and transmits it to the local industrial computer for image processing. The device processes and identifies the image information and detects whether the status of the wire rope is normal. If the wire rope is abnormal, an alarm will be issued through the human-computer interaction module to remind the staff or control the shutdown to prevent accidents. If it is normal, the detection will continue until the lifting work is completed.

Images