A hoisting rope disorder visual detection method and system based on optical flow anomaly detection

Abstract

The application discloses a hoist rope disorder visual detection method and system based on optical flow anomaly detection, obtains video frames and CAN messages, segments a drum and a steel wire rope traction area, obtains a steel wire rope direction and a center through morphological processing, determines a fluctuation level according to center frame change; the instantaneous optical flow vector of the steel wire rope foreground pixels is calculated by using Farneback optical flow, and is decomposed into longitudinal winding and unwinding speed and transverse jumping speed; working conditions are divided according to speed and load, and online statistics are generated to obtain transverse speed and acceleration thresholds, which are used for detecting rope jumping or groove jumping precursors and checking CAN rope speed; the transverse jumping speed heat map and the drum color image are stacked to input an unsupervised anomaly detection model, and a surface anomaly score is obtained; the fluctuation level, the precursor event, the checking result and the anomaly score are fused, and three levels of states of normal, attention or alarm are output; the application can relieve post-detection lag and sample shortage, and realize active pre-warning.

Description

Technical Field

[0001] This invention belongs to the field of crane safety monitoring technology, and in particular relates to a visual detection method and system for hoisting rope tangling based on optical flow anomaly detection. Background Technology

[0002] The hoisting system is the core actuator of a crane, and the orderly winding state of its wire rope on the drum directly determines the safety and efficiency of equipment operation. Currently, the industry generally relies on manual visual observation through rearview mirrors or electronic screens to monitor the hoisting status. This method is inefficient and overly dependent on the operator's attention and experience, making it difficult to meet the needs of modern construction for continuous, real-time, and intelligent safety monitoring.

[0003] To replace manual monitoring, existing technologies have proposed various automated detection schemes. Patent CN105438983A discloses a method and device for monitoring tangled ropes in construction machinery and its winches. It acquires information on the hoisting weight and drum rotation, calculates the tension and speed of the wire rope, and uses camera images to analyze the parallelism of the wire rope. An alarm is triggered when the tension or angle exceeds a threshold. Patent CN112634592A discloses a method and device for identifying tangled ropes in winches using a camera. It employs a target detection network to locate the rope-winding area, then uses a classification network to identify the normal or tangled state of the area images, and combines multi-frame statistical results for graded early warning. Patent CN114057106A discloses a method, device, and computer-readable storage medium for identifying tangled ropes in dual winches. It determines whether tangled ropes have occurred and the type of tangled rope based on the degree of offset between the sampling differences of the main and auxiliary winch encoders.

[0004] However, all of the aforementioned existing technologies suffer from the fundamental flaw of detection lag. Essentially, they all rely on post-event identification or judgment of rope tangling; by the time the system triggers an alarm, the wire rope has often already been compressed and deformed, causing irreversible damage. More importantly, existing solutions cannot effectively capture the root cause of rope tangling—the millisecond-level transient process of the wire rope jumping or skipping between the rope grooves. Furthermore, rope tangling is a low-probability event, and the scarcity of positive samples results in poor generalization ability of deep learning-based classification models, making it difficult to achieve the technological leap from post-event alarms to proactive prevention. Summary of the Invention

[0005] Purpose of the Invention: The purpose of this invention is to provide a visual detection method and system for hoisting wire rope tangling based on optical flow anomaly detection. It aims to capture in real time the root cause transient precursor event of hoisting wire rope skipping or sluice jumping, which leads to rope tangling, by using optical flow anomaly detection technology. This alleviates the problems of existing technologies, which can only identify tangling after it has occurred, such as detection lag, inability to capture millisecond-level transient anomalies, and lack of low-probability fault samples, and achieves proactive early warning of hoisting wire rope tangling.

[0006] Technical solution: The visual detection method for tangled ropes in a winch based on optical flow anomaly detection described in this invention includes the following steps:

[0007] Acquire continuous winch video frames containing the drum and wire rope, as well as the on-vehicle CAN message corresponding to the winch video frames, and obtain the winch speed and load from the on-vehicle CAN message;

[0008] The hoist video frame is segmented into foreground and background to obtain a segmentation mask containing pixel position information of the drum and wire rope. The drum region and wire rope traction region are determined based on the segmentation mask. The wire rope traction region is morphologically processed to obtain the wire rope direction vector and center position. The fluctuation level is determined based on the inter-frame change of the center position.

[0009] The corresponding hoisting video frame is masked using the mask region corresponding to the wire rope in the segmentation mask. The wire rope foreground image region that retains the original pixel information of the wire rope is extracted. The Farneback dense optical flow algorithm is used to calculate the instantaneous optical flow vector of each pixel in the wire rope foreground image region. The instantaneous optical flow vector of each pixel constitutes a dense optical flow field. With the wire rope direction vector as the dynamic reference, the instantaneous optical flow vector of each pixel is decomposed into a longitudinal winding and unwinding velocity component along the rope direction and a lateral jumping velocity component perpendicular to the rope direction.

[0010] The working conditions are divided into intervals according to the hoisting speed and the load. The lateral velocity component and short-term lateral acceleration collected in each interval when the wire rope does not jump or slip out of the groove, the drum surface is not tangled, and there is no abnormal lateral jumping are used as threshold reference data. The threshold reference data is statistically analyzed online, and the lateral velocity threshold and lateral acceleration threshold are dynamically generated using exponential moving average and sliding standard deviation. The currently detected lateral velocity component and short-term lateral acceleration are compared with the lateral velocity threshold and the lateral acceleration threshold, respectively. When the lateral velocity component exceeds the lateral velocity threshold and continues for a preset number of frames, or when the short-term lateral acceleration exceeds the lateral acceleration threshold, a jump or slip out of the groove precursor event is determined to have occurred.

[0011] The global average value of the longitudinal winding and unwinding speed component is checked for consistency with the rope speed obtained by parsing the CAN message from the upper vehicle. The sensitivity of the lateral speed threshold and the lateral acceleration threshold is adjusted according to the fluctuation level. A lateral jumping speed heatmap aligned with the drum area space is generated based on the lateral jumping speed component. The color image of the drum area and the lateral jumping speed heatmap are stacked to form multi-channel input data. The multi-channel input data is input into an unsupervised anomaly detection model to obtain an anomaly score on the drum surface. The unsupervised anomaly detection model is trained only using the multi-channel input data collected when the wire rope does not jump or slip and there is no rope tangling on the drum surface.

[0012] By integrating the fluctuation level, the rope skipping or sluice gate warning event, the consistency verification result, and the drum surface anomaly score, one of the following states is determined: normal state, alert state, and alarm state, and the corresponding display information and alarm information are output.

[0013] Preferably, the morphological processing of the wire rope traction area to obtain the wire rope direction vector and center position includes:

[0014] The mask portion corresponding to the wire rope traction area in the segmentation mask is etched to remove noise, and then the skeleton is extracted to obtain the center line of the wire rope.

[0015] Calculate the minimum bounding rectangle for the region formed by the centerline of the wire rope, determine the direction vector of the wire rope according to the major axis direction of the minimum bounding rectangle, and determine the center position of the wire rope according to the center point of the minimum bounding rectangle.

[0016] The curvature or broken line angle change of the center line of the wire rope is calculated. When the curvature change exceeds a preset curvature threshold, or when the broken line angle change exceeds a preset angle threshold, a jump rope tendency warning signal is output.

[0017] Preferably, the instantaneous optical flow vector is denoted as v, and the instantaneous optical flow vector of each pixel is decomposed into a longitudinal release / retraction velocity component along the rope direction and a lateral jumping velocity component perpendicular to the rope direction according to the following formula:

[0018]

[0019]

[0020] in, Let be the unit vector along the direction vector of the wire rope. It is a unit vector perpendicular to the direction vector of the wire rope. The longitudinal extension and retraction speed component, The lateral sway velocity component is described above.

[0021] Preferably, after decomposing the instantaneous optical flow vector of each pixel into a longitudinal release / retraction velocity component along the rope direction and a lateral jumping velocity component perpendicular to the rope direction, the method further includes:

[0022] A lateral sway velocity heatmap, aligned with the space of the roll area, is generated based on the amplitude of the lateral sway velocity component and output to a display device.

[0023] The grayscale value or color value of each pixel in the lateral hopping velocity heatmap represents the magnitude of the lateral hopping velocity at that location.

[0024] Preferably, the online statistical analysis of the threshold benchmark data, using exponential moving average and moving standard deviation to dynamically generate the lateral velocity threshold and lateral acceleration threshold, includes:

[0025] The lateral velocity threshold and the lateral acceleration threshold are calculated using the following formulas:

[0026]

[0027]

[0028] in, The lateral velocity threshold, The lateral acceleration threshold, This is the exponential moving average of the lateral sway velocity component in the threshold baseline data. The sliding standard deviation of the lateral sway velocity component in the threshold baseline data. The threshold reference data is the exponential moving average of short-time lateral acceleration. The sliding standard deviation of short-time lateral acceleration in the threshold baseline data. and The security factor is adjusted according to the preset security policy.

[0029] Preferably, the consistency check between the global average value of the longitudinal extension / retraction speed components and the rope speed obtained from parsing the onboard CAN message includes:

[0030] Calculate the average value of the longitudinal extension and retraction velocity components of all pixels within the foreground image region of the wire rope to obtain the global average longitudinal velocity. ; Obtain the hoisting rope speed obtained by parsing the CAN message from the vehicle. When satisfied When the consistency check fails, a sensor or message anomaly warning is triggered; among them, This is a preset allowable deviation that is adaptively adjusted according to the magnitude of the winch rope speed.

[0031] Preferably, the sensitivity adjustment of the lateral velocity threshold and the lateral acceleration threshold according to the fluctuation level includes:

[0032] The distance between the center positions of the wire rope in adjacent hoisting video frames is determined as the inter-frame change in center position.

[0033] A first fluctuation threshold and a second fluctuation threshold greater than the first fluctuation threshold are set. When the inter-frame change at the center position is not greater than the first fluctuation threshold, the fluctuation level is determined to be a baseline fluctuation state. When the inter-frame change at the center position is greater than the first fluctuation threshold but not greater than the second fluctuation threshold, the fluctuation level is determined to be a first-level fluctuation state. When the inter-frame change at the center position is greater than the second fluctuation threshold, the fluctuation level is determined to be a second-level fluctuation state.

[0034] Safety factors k1 and k2 are set for the baseline fluctuation state, the first-level fluctuation state and the second-level fluctuation state respectively. The safety factor k1 for the first-level fluctuation state and the second-level fluctuation state is smaller than the safety factor k1 for the baseline fluctuation state, and the safety factor k2 for the first-level fluctuation state and the second-level fluctuation state is smaller than the safety factor k2 for the baseline fluctuation state.

[0035] Based on the determined fluctuation level, select the corresponding safety factors k1 and k2, and use the selected safety factors k1 and k2 to calculate the lateral velocity threshold and the lateral acceleration threshold, respectively.

[0036] Preferably, the unsupervised anomaly detection model includes an EfficientNet-B5 feature extraction network and a Gaussian mixture model; the step of inputting the multi-channel input data into the unsupervised anomaly detection model to obtain the roll surface anomaly score includes:

[0037] The multi-channel input data is input into the EfficientNet-B5 feature extraction network to extract multi-scale feature vectors. The multi-scale feature vectors are then input into the Gaussian mixture model to calculate the Mahalanobis distance or negative log-likelihood probability of the feature distribution represented by the Gaussian mixture model corresponding to the multi-scale feature vectors. The Mahalanobis distance or negative log-likelihood probability is used as the anomaly score of the roll surface. The anomaly score is compared with a dynamic threshold, which is determined online according to a preset quantile based on the extreme value distribution of the anomaly scores output by the unsupervised anomaly detection model on the training samples.

[0038] Preferably, the step of integrating the fluctuation level, the rope skipping or skipping warning event, the consistency check result, and the drum surface anomaly score to determine one of the following states: normal state, alert state, and alarm state:

[0039] The hoisting motion direction is obtained from the CAN message of the vehicle, and the drum speed direction is determined based on the hoisting motion direction. The lateral jitter intensity is characterized by the variance or peak value of the lateral jitter speed component. The angle between the main optical flow direction and the drum speed direction, the inter-frame change of the tilt angle of the minimum bounding rectangle, and the drum surface anomaly score are obtained. A first fusion threshold and a second fusion threshold higher than the first fusion threshold are set for the lateral jitter intensity, the angle, the inter-frame change of the tilt angle, and the drum surface anomaly score, respectively. When each judgment item is lower than the corresponding first fusion threshold, the normal state is output. When there is no judgment item that reaches the corresponding second fusion threshold and any judgment item reaches the corresponding first fusion threshold, the attention state is output. When any judgment item reaches the corresponding second fusion threshold, the alarm state is output.

[0040] The winch rope tangling visual inspection system based on optical flow anomaly detection described in this invention includes:

[0041] The image acquisition module is used to acquire continuous hoisting video frames containing the drum and wire rope, as well as the on-vehicle CAN message corresponding to the hoisting video frames, and to acquire the hoisting speed and load from the on-vehicle CAN message.

[0042] The segmentation and morphological analysis module is used to segment the foreground and background of the hoist video frame, obtain a segmentation mask containing pixel position information of the drum and wire rope, determine the drum region and wire rope traction region based on the segmentation mask, perform morphological processing on the wire rope traction region to obtain the wire rope direction vector and center position, and determine the fluctuation level based on the inter-frame changes of the center position.

[0043] The optical flow analysis module is used to perform masking processing on the corresponding hoisting video frames using the mask region corresponding to the wire rope in the segmentation mask, extract the wire rope foreground image region that retains the original pixel information of the wire rope, calculate the instantaneous optical flow vector of each pixel in the wire rope foreground image region using the Farneback dense optical flow algorithm, form a dense optical flow field from the instantaneous optical flow vectors of each pixel, and decompose the instantaneous optical flow vector of each pixel into a longitudinal winding and unwinding velocity component along the rope direction and a lateral jumping velocity component perpendicular to the rope direction, using the wire rope direction vector as a dynamic reference.

[0044] An adaptive threshold generation module is used to divide the working condition intervals according to the hoisting speed and the load. Within each working condition interval, when the wire rope has not skipped or jumped out of its groove, the drum surface is free of rope tangling, and there is no abnormal lateral jumping, the lateral velocity component and its short-term lateral acceleration are collected as threshold reference data. The threshold reference data is statistically analyzed online using exponential moving average and sliding standard deviation to dynamically generate lateral velocity thresholds and lateral acceleration thresholds. The currently detected lateral velocity component and short-term lateral acceleration are compared with the lateral velocity threshold and the lateral acceleration threshold, respectively. When the lateral velocity component exceeds the lateral velocity threshold and continues for a preset number of frames, or when the short-term lateral acceleration exceeds the lateral acceleration threshold, a skipping or jumping precursor event is determined to have occurred.

[0045] An unsupervised anomaly detection module is used to verify the consistency between the global average value of the longitudinal winding and unwinding speed component and the rope speed obtained by parsing the CAN message from the upper vehicle, and to adjust the sensitivity of the lateral speed threshold and the lateral acceleration threshold according to the fluctuation level; to generate a lateral jumping speed heatmap aligned with the space of the drum area based on the lateral jumping speed component, to stack the color image of the drum area and the lateral jumping speed heatmap into multi-channel input data, and to input the multi-channel input data into an unsupervised anomaly detection model to obtain an anomaly score on the drum surface, wherein the unsupervised anomaly detection model is trained only using the multi-channel input data collected when the wire rope does not jump or slip out of the groove and there is no rope tangling on the drum surface;

[0046] The integrated decision output module is used to integrate the fluctuation level, the rope skipping or sluice gate warning event, the consistency verification result, and the roll surface anomaly score to determine one of the normal state, attention state, and alarm state, and output the corresponding display information and alarm information.

[0047] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: 1. This invention focuses monitoring on the root causes of rope tangling or cable slippage, quantifying millisecond-level lateral abnormal jumping of the wire rope using optical flow technology. This effectively alleviates the lag in traditional post-event detection and the scarcity of low-probability fault samples, achieving proactive pre-event risk control in hoisting operations; 2. It finely divides working condition intervals based on speed and load information in CAN messages, and dynamically generates lateral velocity and acceleration thresholds using online statistical methods. This avoids the insufficient adaptability of fixed thresholds under varying working conditions. 3. The consistency between the rope speed calculated by vision and the CAN message is checked to eliminate sensor interference. The threshold sensitivity is adjusted in combination with the fluctuation level to build a three-level progressive early warning strategy of normal, attention and alarm, which provides operation and maintenance personnel with accurate and hierarchical risk assessment basis; 4. The unsupervised anomaly detection model is trained only with data under normal working conditions. It does not need to rely on a large number of rare rope tangling fault samples, which effectively alleviates the problem of lack of fault data. At the same time, it enables the system to have good generalization ability for different hoist models and complex working conditions. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0049] Figure 2 This is a schematic diagram of the YOLO-SEG segmentation results of the present invention;

[0050] Figure 3 This is a schematic diagram of the segmentation mask of the present invention after morphological operations;

[0051] Figure 4 This is a schematic diagram of the rope region obtained by morphological analysis according to the present invention.

[0052] Figure 5 This is a schematic diagram of the anomaly detection input image obtained by RGB matting in this invention;

[0053] Figure 6 This is a schematic diagram illustrating the optical flow method used in this invention to obtain the direction of roll motion. Detailed Implementation

[0054] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0055] The hoisting system drives the hook to rise and fall by winding the wire rope around the drum. The hook rises when the hoist rotates forward and retracts the rope, and falls when the hoist rotates backward and releases the rope. The term "tangled rope" in this article refers to a state where the wire rope is disordered, squeezed, or overlapped on the drum; "skipping rope" or "skipping groove" refers to the rapid lateral jumping of the wire rope between adjacent grooves. Skipping rope or skipping groove is short-lived, possibly appearing only within a few frames of a continuous video, but it can further cause wire rope squeezing, crossing grooves, and overlapping on the drum surface.

[0056] In this paper, the drum region refers to the image region containing the drum and the wire rope on its surface; the wire rope traction region refers to the image region containing the wire rope extending from the drum towards the boom pulley; the segmentation mask is pixel-level data that identifies the foreground pixel positions of the drum and wire rope; the wire rope foreground image region is the image region that retains the original pixel information of the wire rope, obtained by masking the hoisting video frame using the mask region corresponding to the wire rope in the segmentation mask. Therefore, obtaining the segmentation mask and extracting the wire rope foreground image region are two continuous but distinct processing steps.

[0057] Threshold baseline data refers to the lateral velocity component and short-term lateral acceleration collected when the wire rope has not skipped or jumped out of its groove, the drum surface is free of rope tangling, and there is no abnormal lateral jumping. Current detection data refers to the lateral velocity component and short-term lateral acceleration corresponding to the current hoisting video frame to be judged. Threshold baseline data is used to establish or update thresholds, while current detection data is used for comparison with the thresholds; the two data uses are distinct. The multi-channel data used for training the unsupervised anomaly detection model is also collected during the aforementioned operational phase where no skipping or jumping out of its groove, no rope tangling, and no abnormal lateral jumping have occurred. It consists of the drum surface texture without rope tangling and the optical flow pattern without abnormal lateral jumping.

[0058] At construction sites, operators typically observe the drum through rearview mirrors or electronic screens. The lateral movement of the wire rope across the groove can occur within milliseconds, making it difficult to capture visually in time. By the time the screen shows wire rope compression or overlap, rope tangling on the drum surface has already occurred, and the wire rope may have already undergone irreversible deformation. Contact sensors such as tension or limit sensors are also affected by installation conditions and other operating conditions. Furthermore, rope tangling is a low-probability event, making it difficult to collect complete samples of abnormalities under different lighting conditions, rope speeds, machine models, and operating scenarios. This implementation method directly detects the actual lateral movement of the wire rope and the arrangement of rope on the drum surface through the joint processing of continuous industrial images and on-board CAN messages, addressing the aforementioned on-site technical problems.

[0059] I. System Deployment and Overall Data Flow

[0060] The hoist rope tangling visual inspection system of this embodiment is deployed on a crane and includes an industrial camera, a CAN communication interface, an image acquisition module, a segmentation and morphological analysis module, an optical flow analysis module, an adaptive threshold generation module, an unsupervised anomaly detection module, a comprehensive decision output module, and a display and alarm unit. The industrial camera acquires hoist video frames, and the CAN communication interface acquires CAN messages from the crane. Each processing module processes the hoist video frames and the crane CAN messages synchronously, and the display and alarm unit outputs the detection screen, attention information, alarm information, or data source anomaly warnings.

[0061] The industrial camera is mounted at the base of the boom or diagonally above the winch, ensuring that the entire drum and the wire rope exit are within the camera's field of view. In one specific configuration, the industrial camera has a resolution of 1920×1080, a frame rate of at least 30 fps, and uses a global shutter to reduce motion blur caused by drum rotation and wire rope winding. Once the industrial camera's position is fixed, the spatial relationship between the camera coordinate system and the drum area remains stable, facilitating pixel-level alignment of the color image, segmentation mask, and lateral sag velocity heatmap.

[0062] Figure 1 This paper demonstrates the complete workflow of a visual detection method for hoisting rope tangles based on optical flow anomaly detection. Video frames of hoisting operations captured by an industrial camera are first input into the YOLO-SEG target detection and segmentation module to obtain a segmentation mask (binary image) and the original color image (original RGB image). Subsequently, these two types of data are processed in parallel through three independent branches.

[0063] The first branch performs morphological analysis on the segmentation mask, specifically including erosion denoising and skeleton extraction, and then calculates the minimum circumscribed rotation rectangle of the wire rope to obtain its center position, tilt angle, and direction vector. Based on the inter-frame changes in the center position, the branch extracts the dynamic fluctuation range and, according to preset first-level and second-level fluctuation thresholds, determines whether the current fluctuation exceeds the first-level or second-level range, and outputs a first-level fluctuation warning or a second-level fluctuation warning accordingly.

[0064] The second branch uses the original color image and segmentation mask to extract the foreground region of the wire rope. It then employs the Farneback dense optical flow algorithm to calculate the instantaneous optical flow vector for each pixel. Combined with the real-time direction vector provided by the first branch, the optical flow is decomposed into the longitudinal winding and unwinding speed along the rope and the lateral jumping speed perpendicular to the rope, simultaneously generating a lateral jumping speed heatmap. Simultaneously, this branch receives CAN messages from the upper vehicle to obtain operating parameters such as hoisting speed and load. Based on different operating conditions, it establishes a lateral vibration statistical benchmark and dynamically generates lateral speed and acceleration thresholds online. This branch also performs consistency verification between the drum speed calculated from the optical flow (i.e., the global average of the longitudinal winding and unwinding speed) and the rope speed parsed from the CAN message. This involves analyzing the drum speed and optical flow direction to determine if the visual rope speed matches the CAN signal. If they do not match, a data source anomaly warning is triggered; if they match, the current lateral jumping speed component and short-term lateral acceleration are further compared with dynamic thresholds. Based on the degree of exceedance (i.e., whether it exceeds the first or second level range), a corresponding first or second level warning is output.

[0065] The third branch spatially aligns the color image of the roll area with the lateral sway velocity heatmap generated by the second branch and stacks them into multi-channel input data, which is then fed into the unsupervised anomaly detection model. This model is pre-trained using only data under normal operating conditions and outputs anomaly scores on the roll surface to determine whether an abnormal state exists on the roll surface. Normal data is also marked in the figure, indicating that the model training phase relies solely on normal samples.

[0066] The outputs from the three branches ultimately converge to the integrated decision-making module. This module integrates the fluctuation level from the first branch, the rope skipping or chute skipping precursor events from the second branch (including level one or two warnings), the consistency verification results, and the drum surface anomaly score from the third branch. It then comprehensively determines the current state as one of three levels: normal, alert, or alarm, and outputs the corresponding display and alarm information. If the consistency verification fails, a data source anomaly warning is prioritized to ensure system reliability.

[0067] II. Video Preprocessing and Pixel-Level Segmentation

[0068] The industrial camera continuously outputs hoist video frames. Before model inference, pixel values ​​are mapped from 0-255 to 0-1, and the image size is adjusted to 640×640 to fit the lightweight YOLOv8n-seg instance segmentation model. YOLOv8n-seg outputs pixel-level segmentation masks and bounding boxes for locating regions of interest. Foreground pixels in the segmentation mask correspond to the areas where the drum and wire rope are located, while background pixels correspond to non-target areas such as the crane structure, sky, and construction site. Based on the spatial position and shape of the foreground pixels, the drum region and the wire rope traction region extending outward from the drum are further determined.

[0069] The images used to train the segmentation model came from different lighting conditions, rope speeds, machine models, and operating scenarios, with the foreground region of the hoist manually masked. This training data was used to adapt the segmentation model to various business scenarios. The role of the segmentation model is to limit the physical target range for subsequent morphological analysis, optical flow calculation, and drum surface detection, rather than directly outputting a conclusion of rope tangling based on a single frame image.

[0070] Figure 2 The YOLO-SEG segmentation results are shown. The image is a color image of a crane drum and the wire rope on its surface. A rectangle encloses the drum body and the wire rope on the drum surface, and the wire rope extending from the drum is visible above the drum. The rectangle indicates the location of the region of interest, and the pixel-level segmentation mask further indicates the location of the foreground pixels within that region.

[0071] The foreground portion of the segmentation mask corresponding to the drum and wire rope is eroded to remove isolated noise and small burrs, and then the skeleton is extracted. Figure 3 The black area represents the background, while the white area represents the foreground of the drum and wire rope, preserved after morphological processing. The slender white portion above the drum corresponds to the wire rope traction area, and the main white portion corresponds to the drum area. If a few isolated points remain in the image, they can be further removed using connected component area filtering.

[0072] III. Wire Rope Direction, Center Position, and Wave Rating

[0073] The skeleton of the corroded wire rope mask is extracted to obtain the centerline of the wire rope. The minimum bounding rectangle is calculated for the pixel set of the contour or centerline of the wire rope traction area. The minimum bounding rectangle includes the center point, major axis length, minor axis length, and tilt angle. The wire rope direction vector is determined by the major axis direction, the center point is determined by the center position of the wire rope, and the real-time tilt angle of the wire rope is determined by the angle of the major axis relative to the image coordinate axes.

[0074] Figure 4 The diagram shows the wire rope traction region determined from the binary mask. Red rectangles are arranged along the slender wire rope region, with their major axis aligned with the wire rope's extension direction. This rectangle provides the wire rope direction vector, center position, and tilt angle for the current frame. The wire rope direction vector is passed frame-by-frame to the optical flow dynamic analysis process, serving as the dynamic reference for decomposing the instantaneous optical flow vector.

[0075] The distance between the center positions of the wire rope in adjacent hoisting video frames is used as the inter-frame variation of the center position. A first fluctuation threshold and a second fluctuation threshold greater than the first fluctuation threshold are set. When the inter-frame variation of the center position is not greater than the first fluctuation threshold, it is determined to be a baseline fluctuation state; when the inter-frame variation of the center position is greater than the first fluctuation threshold but not greater than the second fluctuation threshold, it is determined to be a first-level fluctuation state; when the inter-frame variation of the center position is greater than the second fluctuation threshold, it is determined to be a second-level fluctuation state. The first fluctuation threshold and the second fluctuation threshold correspond to the first and second allowable fluctuation ranges within the installation field of view.

[0076] In addition to the center position, the inter-frame variation of the tilt angle of the minimum circumscribed rectangle is calculated, as well as the curvature or polygonal angle change of the wire rope centerline. When the curvature change exceeds a preset curvature threshold, or the polygonal angle change exceeds a preset angle threshold, a jump rope tilt warning signal is output. Thus, the first branch simultaneously provides information on the center position fluctuation level, tilt angle change, and abrupt change in centerline shape.

[0077] IV. Foreground Extraction of Wire Rope and Adaptive Optical Flow Analysis in the Main Direction

[0078] By using the mask region corresponding to the wire rope in the segmentation mask, pixel-by-pixel masking is performed on the corresponding hoisting video frame. The background pixels are set to background values ​​while the original color pixels of the wire rope and the drum surface are retained, thus obtaining the foreground image region of the wire rope. Figure 5 The black portion represents the removed background, while the colored portion represents the retained steel wire rope on the drum surface and extending from the drum. The foreground edges correspond to the boundaries of the segmentation mask. This foreground image is used for dense optical flow calculation; the colored image of the drum region within it is also used for multimodal unsupervised anomaly detection.

[0079] The Farneback dense optical flow algorithm is applied to the foreground image region of the wire rope corresponding to adjacent hoisting video frames to calculate the instantaneous optical flow vector v of each pixel in the region from the current frame to the next frame. The instantaneous optical flow vectors of each pixel together constitute the dense optical flow field. Figure 6 The red arrows in the image indicate the direction of motion and relative amplitude at each sampling position; the arrows are distributed on the surface of the drum where the wire rope is located, reflecting the pixel motion caused by the rotation of the drum and the winding and unwinding of the wire rope.

[0080] Let the unit vector of the wire rope direction vector in the current frame be... The unit vector perpendicular to the direction vector of the wire rope is .by and A dynamic reference coordinate system is constructed that varies with the direction of the wire rope, and the instantaneous optical flow vector v of each pixel is decomposed according to the following formula:

[0081]

[0082]

[0083] In the formula, The longitudinal winding and unwinding velocity component along the rope direction represents the main motion of the wire rope as it is wound or unwound from the drum. The lateral sway velocity component, perpendicular to the rope direction, characterizes the lateral movement of the wire rope deviating from the traction direction. Because... and With each frame update, even if the wire rope traction direction changes due to boom posture or camera viewpoint, the longitudinal and lateral components remain based on the current wire rope direction. The average optical flow vector within the foreground image region of the wire rope can also be calculated to characterize the overall motion of the wire rope, and... The variance or peak value represents the intensity of lateral sway.

[0084] V. Adaptive Threshold for Operating Conditions, Precursor Detection, and Data Consistency

[0085] The hoisting speed and load are acquired in real time from the on-board CAN message, and the operating conditions are divided into intervals according to the hoisting speed and load. Threshold reference data is maintained for each operating condition interval to ensure that different combinations of hoisting speeds and loads do not share the same statistical benchmark. Online statistics are performed on the threshold reference data within each operating condition interval. The threshold reference data is not used to represent currently occurring rope skips or trough jumps, but rather to describe the lateral motion benchmark when no rope skips or trough jumps, no rope tangling, and no abnormal lateral jumping occur within that operating condition interval.

[0086] One statistical method uses exponential moving average and moving standard deviation to generate the lateral velocity threshold according to the following formulas. and lateral acceleration threshold :

[0087]

[0088]

[0089] In the formula, The lateral velocity threshold, The lateral acceleration threshold; This is the exponential moving average of the lateral sway velocity component v⊥ in the threshold baseline data. This represents the sliding standard deviation of the lateral sway velocity component; for Short-term lateral acceleration, For threshold baseline data The exponential moving average, for The sliding standard deviation; For the lateral speed safety factor, The lateral acceleration safety factor is used. Another statistical method uses kernel density estimation instead of the combination of exponential moving average and moving standard deviation to determine the lateral velocity threshold and lateral acceleration threshold based on the probability density distribution of the threshold benchmark data within the corresponding operating condition interval.

[0090] get and Then, compare the current detection data with it. When any pixel position obtained by the current detection... Exceed And continue to the preset number of frames, or the currently detected instantaneous frame rate. Exceed At that time, it is determined that a precursor event such as rope skipping or job-hopping has occurred. The comparison object is the current detection data, while and It is generated from the threshold benchmark data of the corresponding working condition range, so there is no cyclic relationship that uses the event to be detected itself to define the threshold.

[0091] The volatility level is used to select a safety factor. Corresponding settings are provided for the baseline volatility state, Level 1 volatility state, and Level 2 volatility state. and The first-level fluctuation state and the second-level fluctuation state correspond to All are less than the baseline fluctuation state corresponding to The first-level fluctuation state and the second-level fluctuation state correspond to All are less than the baseline fluctuation state corresponding to Select the corresponding level based on the current volatility level. and and substitute and The calculation formula couples the center position fluctuation result of the first branch with the optical flow threshold generation process of the second branch.

[0092] To verify the consistency between the visual data and the on-vehicle CAN message, the number of pixels in the foreground image area of ​​the wire rope was calculated. The average value is used to obtain the global average longitudinal velocity. Simultaneously, the hoisting rope speed is analyzed from the CAN message received by the vehicle. And compare them according to the following formula:

[0093]

[0094] In the formula, This is the allowable deviation for adaptive adjustment based on the speed of the winch rope. When the above inequality holds, the consistency check fails, triggering a sensor or message anomaly warning to avoid using erroneous CAN data to determine the winch status. The winch movement direction can also be obtained from the onboard CAN message, and the drum speed direction can be determined accordingly; the angle between the main optical flow direction and the drum speed direction is used as the direction consistency judgment item.

[0095] Based on each pixel within the roll area | | Generate a lateral hop velocity heatmap. The grayscale or color value of each pixel in the lateral hop velocity heatmap represents the amplitude of the lateral hop velocity at the corresponding location. The lateral hop velocity heatmap and the color image of the roll area use the same spatial coordinates and are output to the display device in real time, while also being passed to the unsupervised anomaly detection process.

[0096] VI. Multimodal Unsupervised Anomaly Detection

[0097] The color image of the roll region is extracted based on the segmentation mask, and the edges of the roll region are shrunken inward to remove interference from segmentation burrs on surface anomaly detection. The shrunken color image of the roll region is adjusted to 224×224 and normalized. The lateral velocity heatmap is adjusted to the same size and aligned pixel-by-pixel spatially with the color image. Then, the RGB channels and the lateral velocity channel are stacked into multi-channel input data. Thus, the same spatial location simultaneously contains roll surface texture information and lateral motion information.

[0098] One implementation uses EfficientNet-B5 as the feature extraction network to extract multi-scale feature vectors from multi-channel input data, and then inputs these multi-scale feature vectors into a Gaussian Mixture Model (GMM). The GMM uses training data to establish a feature distribution, and during inference, it calculates the Mahalanobis distance or negative log-likelihood probability of the current multi-scale feature vector relative to this feature distribution, using the Mahalanobis distance or negative log-likelihood probability as the roll surface anomaly score. Another implementation uses the EfficientNet-AD unsupervised anomaly detection model, and uses the reconstruction error as the roll surface anomaly score. Neither implementation requires a large number of tangled rope samples as a necessary training condition.

[0099] The training data uses only multi-channel data collected when there are no rope skips or chute jumps, no rope tangling on the drum surface, and no abnormal lateral jumping. In other words, the data shows no rope tangling on the drum surface texture and no abnormal lateral jumping in the optical flow pattern. Based on the extreme value distribution of the abnormal scores output by the model on this training data, a dynamic threshold is determined online according to a preset quantile, such as the 99th percentile. When the abnormal score of the drum surface in the current multi-channel input data exceeds the dynamic threshold, a rope tangling abnormality is determined to have occurred on the drum surface.

[0100] VII. Comprehensive Decision-Making and Calibration Range

[0101] The integrated decision output module receives the center position fluctuation level, the inter-frame change in wire rope tilt angle, precursory events of rope skipping or slot skipping, CAN consistency verification results, lateral runout intensity, directional consistency results, and drum surface anomaly score. Lateral runout intensity can be determined by... The variance or peak value is characterized; in a set of specific calibrations shown in Table 1, using The standard deviation is used as the lateral jitter intensity. Directional consistency is characterized by the angle between the main optical flow direction and the drum velocity direction. The change in wire rope tilt angle is characterized by the inter-frame change of the minimum circumscribed rotation rectangle angle. The drum surface anomaly score is calculated using the negative log-likelihood of the GMM output. A first fusion threshold and a second fusion threshold higher than the first fusion threshold are set for the lateral jitter intensity, directional angle, inter-frame change in tilt angle, and drum surface anomaly score, respectively. The first fusion threshold corresponds to the boundary between the baseline interval and the attention interval, and the second fusion threshold corresponds to the boundary between the attention interval and the alarm interval.

[0102] Table 1. A set of calibration intervals for fusion judgment

[0103] Lateral jump intensity Standard deviation of lateral velocity component <0.5 pixels / frame 0.5~1.0 pixels / frame >1.0 pixel / frame Consistency of direction Angle between the principal direction of optical flow and the direction of drum velocity ＜15° 15°～30° ＞30° Changes in the angle of inclination of the wire rope Inter-frame variation of the angle of the minimum circumscribed rotation rectangle ＜5° / s 5～15° / s ＞15° / s Roll surface anomaly score The negative log-likelihood of GMM output ＜2.0 2.0～3.0 ＞3.0

[0104] The values ​​in Table 1 are derived from a set of specific calibration data and are used to illustrate the hierarchical relationship of each judgment item. Lateral runout intensity less than 0.5 pixels / frame is in the baseline range, 0.5–1.0 pixels / frame is in the attention range, and greater than 1.0 pixels / frame is in the alarm range; directional angle less than 15° is in the baseline range, 15°–30° is in the attention range, and greater than 30° is in the alarm range; tilt angle inter-frame variation less than 5° / s is in the baseline range, 5–15° / s is in the attention range, and greater than 15° / s is in the alarm range; GMM negative log-likelihood less than 2.0 is in the baseline range, 2.0–3.0 is in the attention range, and greater than 3.0 is in the alarm range. The above calibration ranges are used for comprehensive decision-making. The adaptive TV and Ta values ​​are still generated online based on hoisting speed, load, and threshold baseline data.

[0105] When all fused judgment items are within their corresponding baseline intervals, no rope skipping or job-hopping precursor events are determined, and the CAN consistency check passes, a normal state is output. When no judgment item is within the alarm interval and any judgment item is within the attention interval, or a level 1 fluctuation information occurs, an attention state is output. When any judgment item is within the alarm interval, a level 2 fluctuation information occurs, or a rope skipping or job-hopping precursor event is determined, an alarm state is output. If the CAN consistency check fails, a separate sensor or message anomaly warning is output. Normal state, attention state, and alarm state are names of the integrated decision results, and their judgment conditions are explicitly defined by the aforementioned intervals and event conditions.

[0106] VIII. Inspection Process During Crane Lifting Operations

[0107] This embodiment uses a crane lifting operation as an example to illustrate the correspondence between each technical step and the actual equipment status. An industrial camera is fixed at the base of the boom or diagonally above the winch as described above, capturing the entire drum and the wire rope outlet; the CAN communication interface synchronously acquires the winch speed, load, and winch movement direction. Before the lifting begins or during the equipment's baseline operation phase, data is collected showing no rope skipping or slotting, no rope tangling on the drum surface, and no abnormal lateral jumping. This data forms threshold baseline data for each winch speed and load range, which is then used to train a multimodal unsupervised anomaly detection model.

[0108] During the hook's ascent, the winch rotates forward and retracts the rope. After segmentation and masking of continuous video frames, the direction vector of the wire rope traction area is used to establish a dynamic reference coordinate system. The Farneback dense optical flow algorithm provides the motion vectors of each pixel on the drum surface and the exit wire rope. The movement of the wire rope along the rope direction mainly involves the entry... The instantaneous lateral movement of the wire rope crossing the rope groove mainly enters If the wire rope experiences lateral jumps within a few consecutive frames, the inter-frame variation in the center position, the variation in tilt angle, or the curvature of the centerline may change first, while the lateral jump velocity heatmap shows an area of ​​increased amplitude at the corresponding location. When the current... Continue to exceed or current Exceed When the system detects a warning event of rope skipping or cable slippage and outputs an alarm status, it enables operators to take action before the wire rope is further squeezed and overlapped.

[0109] If localized abnormalities have appeared on the roll surface, even if the lateral transient motion is not obvious at that moment, the changes in texture arrangement in the color image of the roll area and the corresponding lateral jump velocity heatmap will still create multi-channel feature differences, increasing the Mahalanobis distance or negative log-likelihood of the GMM, or increasing the reconstruction error of EfficientNet-AD. When the roll surface abnormality score reaches the corresponding judgment condition, the integrated decision outputs an attention state or an alarm state. Thus, the optical flow branch is used to capture short-term dynamic precursors of rope skipping or chute jumping, while the surface abnormality branch is used to identify abnormalities in the roll surface arrangement. The two branches deal with different physical phenomena but are interconnected through the lateral jump velocity heatmap and the integrated decision.

[0110] If the hoisting rope speed given in the CAN message from the vehicle matches the speed calculated visually... Deviation exceeds The system outputs a sensor or message anomaly warning; if the speed magnitudes of the two are the same but the angle between the main direction of optical flow and the direction of drum speed enters the attention zone or alarm zone, then the direction consistency judgment item is used to participate in the comprehensive decision. This process enables the system to distinguish between the actual lateral jump of the wire rope, abnormal arrangement of the drum surface, and abnormal CAN data source, avoiding the judgment of the hoisting status based solely on a single image classification result or a single sensor signal.

[0111] IX. Module Implementation and Data Interface

[0112] The image acquisition module outputs continuous hoisting video frames to the segmentation and morphological analysis module; the segmentation and morphological analysis module outputs the segmentation mask, drum region, wire rope traction region, wire rope direction vector, center position, tilt angle, centerline shape change, and fluctuation level; the optical flow analysis module receives the hoisting video frames, segmentation mask, wire rope direction vector, and CAN operating parameters, and outputs... , , The system receives a series of data points, including: a warning event of rope skipping or slugging; CAN consistency verification results; and a heatmap of lateral skipping speed. An unsupervised anomaly detection module receives a color image of the drum area and a heatmap of lateral skipping speed, and outputs an anomaly score for the drum surface. A comprehensive decision output module sends each output data point to the display and alarm unit. Each data point is associated with the hoist video frame that generated the data or the corresponding time period.

[0113] The aforementioned processing modules can be executed by the computing unit on the crane or by back-end processing equipment connected to industrial cameras and a CAN communication interface. Once deployed on the crane, the industrial camera monitors the crane drum and wire rope as the specific monitoring objects, while the CAN communication interface monitors the crane's hoisting speed, load, and direction of movement signals. The display and alarm unit provides operators with images of the drum area, a heatmap of lateral movement speed, graded status information, and data source anomaly information. By processing continuous industrial images and crane operating data, the system detects the actual lateral movement of the wire rope between the rope grooves and the actual arrangement of the wire rope on the drum surface.

Claims

1. A visual detection method for tangled ropes in hoisting machines based on optical flow anomaly detection, characterized in that, Includes the following steps: Acquire continuous winch video frames containing the drum and wire rope, as well as the on-vehicle CAN message corresponding to the winch video frames, and obtain the winch speed and load from the on-vehicle CAN message; The hoist video frame is segmented into foreground and background to obtain a segmentation mask containing pixel position information of the drum and wire rope. The drum region and wire rope traction region are determined based on the segmentation mask. The wire rope traction region is morphologically processed to obtain the wire rope direction vector and center position. The fluctuation level is determined based on the inter-frame change of the center position. The corresponding hoisting video frame is masked using the mask region corresponding to the wire rope in the segmentation mask. The wire rope foreground image region that retains the original pixel information of the wire rope is extracted. The Farneback dense optical flow algorithm is used to calculate the instantaneous optical flow vector of each pixel in the wire rope foreground image region. The instantaneous optical flow vector of each pixel constitutes a dense optical flow field. With the wire rope direction vector as the dynamic reference, the instantaneous optical flow vector of each pixel is decomposed into a longitudinal winding and unwinding velocity component along the rope direction and a lateral jumping velocity component perpendicular to the rope direction. The working conditions are divided into intervals according to the hoisting speed and the load. The lateral jumping speed component and its short-time lateral acceleration collected in each working condition interval when the wire rope does not jump or jump out of the groove, the drum surface does not have tangled rope, and there is no abnormal lateral jumping are used as threshold reference data. The threshold baseline data is statistically analyzed online, and exponential moving average and sliding standard deviation are used to dynamically generate lateral velocity threshold and lateral acceleration threshold. The currently detected lateral jumping velocity component and short-term lateral acceleration are compared with the lateral velocity threshold and the lateral acceleration threshold, respectively. When the lateral jumping velocity component exceeds the lateral velocity threshold and continues for a preset number of frames, or when the short-term lateral acceleration exceeds the lateral acceleration threshold, it is determined that a rope skipping or dumping precursor event has occurred. The global average value of the longitudinal winding and unwinding speed component is checked for consistency with the rope speed obtained by parsing the CAN message from the upper vehicle. The sensitivity of the lateral speed threshold and the lateral acceleration threshold is adjusted according to the fluctuation level. A lateral jumping speed heatmap aligned with the drum area space is generated based on the lateral jumping speed component. The color image of the drum area and the lateral jumping speed heatmap are stacked to form multi-channel input data. The multi-channel input data is input into an unsupervised anomaly detection model to obtain an anomaly score on the drum surface. The unsupervised anomaly detection model is trained only using the multi-channel input data collected when the wire rope does not jump or slip and there is no rope tangling on the drum surface. By integrating the fluctuation level, the rope skipping or sluice gate warning event, the consistency verification result, and the drum surface anomaly score, one of the following states is determined: normal state, alert state, and alarm state, and the corresponding display information and alarm information are output.

2. The method according to claim 1, characterized in that, The step of performing morphological processing on the wire rope traction area to obtain the wire rope direction vector and center position includes: The mask portion corresponding to the wire rope traction area in the segmentation mask is etched to remove noise, and then the skeleton is extracted to obtain the center line of the wire rope. Calculate the minimum bounding rectangle for the region formed by the centerline of the wire rope, determine the direction vector of the wire rope according to the major axis direction of the minimum bounding rectangle, and determine the center position of the wire rope according to the center point of the minimum bounding rectangle. The curvature or broken line angle change of the center line of the wire rope is calculated. When the curvature change exceeds a preset curvature threshold, or when the broken line angle change exceeds a preset angle threshold, a jump rope tendency warning signal is output.

3. The method according to claim 2, characterized in that, Let the instantaneous optical flow vector be denoted as v, and decompose the instantaneous optical flow vector of each pixel into a longitudinal contraction and release velocity component along the rope direction and a lateral jumping velocity component perpendicular to the rope direction according to the following formula: in, Let be the unit vector along the direction vector of the wire rope. It is a unit vector perpendicular to the direction vector of the wire rope. The longitudinal extension and retraction speed component, The lateral sway velocity component is described above.

4. The method according to claim 3, characterized in that, After decomposing the instantaneous optical flow vector of each pixel into a longitudinal release and take-off velocity component along the rope direction and a lateral jumping velocity component perpendicular to the rope direction, it also includes: A lateral sway velocity heatmap, aligned with the space of the roll area, is generated based on the amplitude of the lateral sway velocity component and output to a display device. The grayscale value or color value of each pixel in the lateral hopping velocity heatmap represents the magnitude of the lateral hopping velocity at that location.

5. The method according to claim 4, characterized in that, The online statistical analysis of the threshold benchmark data, using exponential moving average and moving standard deviation, to dynamically generate the lateral velocity threshold and lateral acceleration threshold includes: The lateral velocity threshold and the lateral acceleration threshold are calculated using the following formulas: in, The lateral velocity threshold, The lateral acceleration threshold, This is the exponential moving average of the lateral sway velocity component in the threshold baseline data. The sliding standard deviation of the lateral sway velocity component in the threshold baseline data. The threshold reference data is the exponential moving average of short-time lateral acceleration. The sliding standard deviation of short-time lateral acceleration in the threshold baseline data. and The security factor is adjusted according to the preset security policy.

6. The method according to claim 5, characterized in that, The consistency check between the global average value of the longitudinal extension / retraction speed components and the rope speed obtained by parsing the onboard CAN message includes: Calculate the average value of the longitudinal extension and retraction velocity components of all pixels within the foreground image region of the wire rope to obtain the global average longitudinal velocity. ; Obtain the hoisting rope speed obtained by parsing the CAN message from the vehicle. When satisfied When the consistency check fails, a sensor or message anomaly warning is triggered; among them, This is a preset allowable deviation that is adaptively adjusted according to the magnitude of the winch rope speed.

7. The method according to claim 6, characterized in that, The sensitivity adjustment of the lateral velocity threshold and the lateral acceleration threshold according to the fluctuation level includes: The distance between the center positions of the wire rope in adjacent hoisting video frames is determined as the inter-frame change in center position. A first fluctuation threshold and a second fluctuation threshold greater than the first fluctuation threshold are set. When the inter-frame change at the center position is not greater than the first fluctuation threshold, the fluctuation level is determined to be a baseline fluctuation state. When the inter-frame change at the center position is greater than the first fluctuation threshold but not greater than the second fluctuation threshold, the fluctuation level is determined to be a first-level fluctuation state. When the inter-frame change at the center position is greater than the second fluctuation threshold, the fluctuation level is determined to be a second-level fluctuation state. Safety factors k1 and k2 are set for the baseline fluctuation state, the first-level fluctuation state and the second-level fluctuation state respectively. The safety factor k1 for the first-level fluctuation state and the second-level fluctuation state is smaller than the safety factor k1 for the baseline fluctuation state, and the safety factor k2 for the first-level fluctuation state and the second-level fluctuation state is smaller than the safety factor k2 for the baseline fluctuation state. Based on the determined fluctuation level, select the corresponding safety factors k1 and k2, and use the selected safety factors k1 and k2 to calculate the lateral velocity threshold and the lateral acceleration threshold, respectively.

8. The method according to claim 7, characterized in that, The unsupervised anomaly detection model includes an EfficientNet-B5 feature extraction network and a Gaussian mixture model; the step of inputting the multi-channel input data into the unsupervised anomaly detection model to obtain the roll surface anomaly score includes: The multi-channel input data is input into the EfficientNet-B5 feature extraction network to extract multi-scale feature vectors. The multi-scale feature vectors are then input into the Gaussian mixture model to calculate the Mahalanobis distance or negative log-likelihood probability of the feature distribution represented by the Gaussian mixture model corresponding to the multi-scale feature vectors. The Mahalanobis distance or negative log-likelihood probability is used as the anomaly score of the roll surface. The anomaly score is compared with a dynamic threshold, which is determined online according to a preset quantile based on the extreme value distribution of the anomaly scores output by the unsupervised anomaly detection model on the training samples.

9. The method according to claim 8, characterized in that, The process of integrating the fluctuation level, the rope skipping or dumping precursor event, the consistency verification result, and the roll surface anomaly score to determine one of the following states: normal state, alert state, and alarm state, including: The hoisting motion direction is obtained from the CAN message of the vehicle, and the drum speed direction is determined based on the hoisting motion direction. The lateral jitter intensity is characterized by the variance or peak value of the lateral jitter speed component. The angle between the main optical flow direction and the drum speed direction, the inter-frame change of the tilt angle of the minimum bounding rectangle, and the drum surface anomaly score are obtained. A first fusion threshold and a second fusion threshold higher than the first fusion threshold are set for the lateral jitter intensity, the angle, the inter-frame change of the tilt angle, and the drum surface anomaly score, respectively. When each judgment item is lower than the corresponding first fusion threshold, the normal state is output. When there is no judgment item that reaches the corresponding second fusion threshold and any judgment item reaches the corresponding first fusion threshold, the attention state is output. When any judgment item reaches the corresponding second fusion threshold, the alarm state is output.

10. A visual detection method for tangled ropes in hoisting machines based on optical flow anomaly detection, characterized in that, include: The image acquisition module is used to acquire continuous hoisting video frames containing the drum and wire rope, as well as the on-vehicle CAN message corresponding to the hoisting video frames, and to acquire the hoisting speed and load from the on-vehicle CAN message. The segmentation and morphological analysis module is used to segment the foreground and background of the hoist video frame, obtain a segmentation mask containing pixel position information of the drum and wire rope, determine the drum region and wire rope traction region based on the segmentation mask, perform morphological processing on the wire rope traction region to obtain the wire rope direction vector and center position, and determine the fluctuation level based on the inter-frame changes of the center position. The optical flow analysis module is used to perform masking processing on the corresponding hoisting video frames using the mask region corresponding to the wire rope in the segmentation mask, extract the wire rope foreground image region that retains the original pixel information of the wire rope, calculate the instantaneous optical flow vector of each pixel in the wire rope foreground image region using the Farneback dense optical flow algorithm, form a dense optical flow field from the instantaneous optical flow vectors of each pixel, and decompose the instantaneous optical flow vector of each pixel into a longitudinal winding and unwinding velocity component along the rope direction and a lateral jumping velocity component perpendicular to the rope direction, using the wire rope direction vector as a dynamic reference. An adaptive threshold generation module is used to divide the working condition intervals according to the hoisting speed and the load, and to collect the lateral jumping velocity component and its short-time lateral acceleration when the wire rope does not jump or jump out of the groove, the drum surface does not have tangled ropes, and there is no abnormal lateral jumping in each working condition interval as threshold reference data. The threshold baseline data is statistically analyzed online, and exponential moving average and sliding standard deviation are used to dynamically generate lateral velocity threshold and lateral acceleration threshold. The currently detected lateral jumping velocity component and short-term lateral acceleration are compared with the lateral velocity threshold and the lateral acceleration threshold, respectively. When the lateral jumping velocity component exceeds the lateral velocity threshold and continues for a preset number of frames, or when the short-term lateral acceleration exceeds the lateral acceleration threshold, it is determined that a rope skipping or dumping precursor event has occurred. An unsupervised anomaly detection module is used to verify the consistency between the global average value of the longitudinal winding and unwinding speed component and the rope speed obtained by parsing the CAN message from the upper vehicle, and to adjust the sensitivity of the lateral speed threshold and the lateral acceleration threshold according to the fluctuation level; to generate a lateral jumping speed heatmap aligned with the space of the drum area based on the lateral jumping speed component, to stack the color image of the drum area and the lateral jumping speed heatmap into multi-channel input data, and to input the multi-channel input data into an unsupervised anomaly detection model to obtain an anomaly score on the drum surface, wherein the unsupervised anomaly detection model is trained only using the multi-channel input data collected when the wire rope does not jump or slip out of the groove and there is no rope tangling on the drum surface; The integrated decision output module is used to integrate the fluctuation level, the rope skipping or sluice gate warning event, the consistency verification result, and the roll surface anomaly score to determine one of the normal state, attention state, and alarm state, and output the corresponding display information and alarm information.

Images