A port machinery device visual perception steel wire rope swing early warning method

Abstract

The present application relates to the technical field of visual perception, in particular to a kind of port machinery equipment visual perception's steel wire rope swing early warning method, comprising the following steps: based on the input port machinery equipment video frame sequence, the time variation value of the pixel brightness between continuous frames is calculated and the spatial contrast value of the pixel neighborhood in single frame, the boundary box of connected region is defined.The present application calculates the time variation of the pixel brightness between continuous frames and the spatial contrast value of the pixel neighborhood in single frame, filters the boundary box of connected region and extracts significant dynamic region, reduces the interference of complex background and static interference on feature extraction, reduces the calculation resource consumption of non-key region.Based on significant dynamic region extraction boundary pixel and record image coordinate system position, construct single frame feature point coordinate set, track and match corresponding feature points between continuous video frames to form swing feature point sequence, improve the spatiotemporal consistency of feature point tracking, avoid the feature point loss problem caused by light change or occlusion.

Description

Technical Field

[0001] This invention relates to the field of visual perception technology, and in particular to a method for early warning of wire rope swaying in port machinery equipment using visual perception. Background Technology

[0002] The field of visual perception technology is an intersection of computer vision and artificial intelligence. Its core is to collect image or video data in the environment through devices such as cameras, lidar, and infrared sensors, and combine them with algorithms such as image processing, target detection, and dynamic tracking to achieve real-time monitoring, feature extraction, and behavior analysis of objects in the scene.

[0003] Current visual perception technologies mostly employ single-frame feature association based on appearance matching or optical flow, which struggles to handle scenarios with simple cable textures and high morphological similarity. During feature point tracking, trajectory breaks are easily caused by target deformation or brief occlusion, leading to accumulated errors in sway parameter calculation. Existing methods rely heavily on instantaneous displacement or velocity indices for quantifying sway features, lacking statistical stability analysis within a time window. This makes them prone to parameter jumps under sudden wind speed changes or equipment vibration interference, resulting in large fluctuations in warning results. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and propose a method for early warning of wire rope swaying based on visual perception in port machinery equipment.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for visually sensing steel wire rope sway warning in port machinery equipment, comprising the following steps:

[0006] Based on the input video frame sequence of the port machinery equipment, calculate the temporal change value of pixel brightness between consecutive frames and the spatial contrast value of the pixel neighborhood within a single frame, define the bounding box of the connected region, and obtain the significant dynamic region.

[0007] Based on the significant dynamic region, boundary pixels are extracted in the corresponding region of the current frame image, and the position coordinates of the pixels in the image coordinate system are recorded to obtain a single-frame feature point coordinate set. Based on the single-frame feature point coordinate set, the corresponding feature point coordinates are tracked and matched between consecutive video frames to establish a shaking feature point sequence.

[0008] Based on the shaking feature point sequence, the displacement vector magnitude of each feature point in the sequence between adjacent time frames is calculated, and the pairwise distance between feature points within the same frame is calculated to obtain frame-by-frame shaking feature parameters. Based on the frame-by-frame shaking feature parameters, the standard deviation of the displacement vector magnitude of all feature points is calculated within a time window to generate a shaking spatiotemporal structure metric.

[0009] Based on the aforementioned swaying spatiotemporal structure measurement, the preset reference values ​​for normal swaying displacement amplitude and swaying frequency of the port machinery equipment cable are compared to obtain the swaying index threshold comparison result. Based on the swaying index threshold comparison result, a cable swaying early warning state is established.

[0010] Preferably, the step of obtaining the significant dynamic region is as follows:

[0011] Based on the input video frame sequence of the port machinery equipment, calculate the absolute difference in brightness of each pixel (i,j) between frame t and frame t-1 to generate a time variation value matrix. At the same time, calculate the standard deviation of brightness in the 3×3 neighborhood of each pixel (i,j) in frame t to generate a spatial contrast value matrix. Thus, the time variation value matrix and the spatial contrast value matrix are obtained.

[0012] Based on the time-varying value matrix and the spatial contrast value matrix, the dynamic significance is calculated;

[0013] Based on the selected pixels, morphological closing operations are used to aggregate adjacent pixels to form connected regions. The minimum bounding rectangle of each connected region is extracted as the bounding box to generate significant dynamic regions.

[0014] Preferably, the steps for obtaining the single-frame feature point coordinate set are as follows:

[0015] Based on the bounding box coordinate range of the significant dynamic region, the corresponding bounding box interior region in the current frame image is traversed, the pixels are scanned row by row and the gradient magnitude difference with the neighboring pixels is detected, the pixels with gradient magnitude exceeding the preset gradient threshold are filtered, and a candidate boundary pixel set is generated.

[0016] Based on the candidate boundary pixel set, calculate the density distribution value of candidate points in a 5×5 neighborhood around each pixel, retain pixels with density distribution values ​​greater than a preset density threshold, remove invalid candidate points in isolated or low-density areas, and generate a valid boundary pixel set.

[0017] Based on the set of effective boundary pixels, according to the rules for defining the origin of the image coordinate system, the integer pairs of horizontal and vertical coordinates of each effective boundary pixel are recorded point by point to form a single-frame feature point coordinate set.

[0018] Preferably, the step of obtaining the shaking feature point sequence is as follows:

[0019] Based on the current frame coordinate point set of the single frame feature point coordinate set, the feature point coordinate sets of two adjacent frames are compared frame by frame. The coordinate offset of each feature point in the neighborhood range of the next frame is calculated. Matching point pairs with the same offset direction and smaller than the preset maximum displacement threshold are selected to generate a set of matching feature point pairs.

[0020] Based on the set of matching feature points, the distribution of motion vector direction angles of all matching pairs is statistically analyzed, the median of the direction angles is calculated as the reference value of the main motion trend, and matching pairs whose direction angles deviate from the median by more than 15 degrees are removed to generate a set of effective matching feature points.

[0021] Based on the set of effective matching feature points, the feature points are classified according to their unique identifiers. The coordinate values ​​of each feature point in different frames are recorded in time-stamp order and integrated into a continuous coordinate data chain that increases in time. A shaking feature point sequence is then established, where the unique identifier of the feature point is the initial frame appearance sequence number.

[0022] Preferably, the step of obtaining the frame-by-frame shaking feature parameters is as follows:

[0023] Based on the unique identifier of each feature point in the shaking feature point sequence, the coordinate data between two adjacent frames are traversed, the absolute value of the Euclidean distance between the feature point coordinates of the current frame and the corresponding feature point coordinates of the previous frame is calculated, and recorded as the displacement vector size, generating a set of displacement vector sizes.

[0024] Based on the coordinates of the feature points in the current frame, traverse all the unique identifiers of the feature points, calculate the Euclidean distance between every two feature points in the same frame, arrange them in the pairing order of the unique identifiers of the feature points, and generate a set of pairwise distances.

[0025] Based on the set of displacement vector magnitudes and the set of pairwise distances, the displacement vector magnitude data and pairwise distance data of all feature points in the same frame are merged according to the frame number and arranged into a displacement-distance parameter matrix indexed by the frame number to generate frame-by-frame shaking feature parameters.

[0026] Preferably, the step of obtaining the swaying spatiotemporal structure metric is as follows:

[0027] Based on the set of displacement vector magnitudes in the frame-by-frame shaking feature parameters, the data segments are divided according to the preset time window length, the standard deviation of all displacement vector magnitudes in each time window is calculated, and a displacement standard deviation sequence is generated.

[0028] Based on the pairwise distance set in the frame-by-frame shaking feature parameters, the ratio of the sum of the absolute values ​​of the distance differences between feature points between adjacent frames within each time window to the length of the time window is calculated to generate the spatiotemporal change rate.

[0029] Based on the displacement standard deviation sequence and the spatiotemporal change rate, the two are aligned according to the time window number and merged into a binary tuple to generate a swaying spatiotemporal structure metric.

[0030] Preferably, the step for obtaining the comparison result of the sway index threshold is as follows:

[0031] Based on the displacement standard deviation and spatiotemporal change rate in the swaying spatiotemporal structure measurement, the preset reference values ​​for normal swaying displacement amplitude and swaying frequency of the port machinery equipment cable are called to generate a set of displacement amplitude reference values ​​and swaying frequency reference values.

[0032] Based on the displacement standard deviation sequence and spatiotemporal change rate sequence of the swaying spatiotemporal structure measurement, each item is compared to see if the displacement standard deviation exceeds the reference value of normal swaying displacement amplitude and if the spatiotemporal change rate exceeds the reference value of swaying frequency. Items that meet any of the excess conditions are marked, and swaying index threshold comparison results are generated.

[0033] Preferably, the step of obtaining the cable sway warning status is as follows:

[0034] Based on the displacement standard deviation sequence and spatiotemporal rate of change sequence in the comparison results of the sway index threshold, the displacement standard deviation sequence and spatiotemporal rate of change sequence within the current time window are extracted, and preset displacement amplitude reference value and sway frequency reference value are obtained.

[0035] Calculate the normalized early warning index;

[0036] Based on the normalized early warning index, determine whether it is an early warning state, count the number of consecutive over-limit windows, and establish a cable sway early warning state.

[0037] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0038] This invention calculates the temporal variation of pixel brightness between consecutive frames and the spatial contrast value of pixel neighborhood in a single frame to filter connected region bounding boxes and extract significant dynamic regions. This reduces the interference of complex backgrounds and static disturbances on feature extraction and decreases the computational resource consumption of non-critical areas. Based on the extraction of boundary pixels from significant dynamic regions and recording their positions in the image coordinate system, a single-frame feature point coordinate set is constructed. Corresponding feature points are tracked and matched between consecutive video frames to form a shaking feature point sequence, improving the spatiotemporal consistency of feature point tracking and avoiding feature point loss due to lighting changes or occlusion. By calculating the displacement vector magnitude and the pairwise distance between feature points within the frame frame on a frame-by-frame basis, combined with the statistical analysis of displacement standard deviation within the time window, a shaking spatiotemporal structure metric is generated, enhancing the dynamic capture capability of shaking amplitude and frequency and suppressing the impact of instantaneous noise on parameter stability. By calling preset displacement amplitude and frequency reference values ​​for dual threshold comparison and combining them with the warning status judgment logic, the dual indicators of displacement anomaly and frequency anomaly are integrated to reduce the probability of misjudgment of a single parameter and improve the reliability of the warning results. This method achieves monitoring and early warning of port machinery cable swaying through multi-level feature screening, spatiotemporal consistency tracking, dynamic parameter fusion, and dual threshold judgment mechanism, adapting to the needs of complex lighting, multi-target interference, and long-term operation scenarios in ports. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the steps of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0041] Please see Figure 1 This invention provides a technical solution: a method for visually sensing steel wire rope sway warning in port machinery equipment, comprising the following steps:

[0042] Based on the input video frame sequence of the port machinery equipment, calculate the temporal change value of pixel brightness between consecutive frames and the spatial contrast value of the pixel neighborhood within a single frame, define the bounding box of the connected region, and obtain the significant dynamic region.

[0043] Based on the significant dynamic region, the boundary pixels are extracted in the corresponding region of the current frame image, and the position coordinates of the pixels in the image coordinate system are recorded to obtain a single frame feature point coordinate set. Based on the single frame feature point coordinate set, the corresponding feature point coordinates are tracked and matched between consecutive video frames to establish a shaking feature point sequence.

[0044] Based on the shaking feature point sequence, the displacement vector magnitude of each feature point in the sequence between adjacent time frames is calculated, and the pairwise distance between feature points within the same frame is calculated to obtain frame-by-frame shaking feature parameters. Based on the frame-by-frame shaking feature parameters, the standard deviation of the displacement vector magnitude of all feature points is calculated within a time window to generate a shaking spatiotemporal structure metric.

[0045] Based on the sway spatiotemporal structure measurement, the preset reference values ​​for normal sway displacement amplitude and sway frequency of the port machinery equipment cable are called and compared to obtain the sway index threshold comparison results. Based on the sway index threshold comparison results, a cable sway warning status is established.

[0046] The steps for obtaining the significant dynamic region are as follows:

[0047] Based on the input video frame sequence of the port machinery equipment, calculate the absolute difference in brightness of each pixel (i, j) between frame t and frame t-1 to generate a time variation value matrix. At the same time, calculate the standard deviation of brightness in the 3×3 neighborhood of each pixel (i, j) in frame t to generate a spatial contrast value matrix. Thus, the time variation value matrix and the spatial contrast value matrix are obtained.

[0048] The dynamic significance is calculated based on the time-varying value matrix and the spatial contrast value matrix, using the following formula:

[0049]

[0050] Filtering to satisfy D i,j >Pixels of T1; where D i,j The dynamic significance level is given by T1, where T1 is the preset dynamic significance threshold, and Δt is the dynamic significance level. i,j Let Δs be the time-varying value of pixel (i, j). i,j The spatial contrast value of pixel (i, j);

[0051] Based on the selected pixels, morphological closing operations are used to aggregate adjacent pixels to form connected regions. The minimum bounding rectangle of each connected region is extracted as the bounding box to generate significant dynamic regions.

[0052] Specifically, based on the input video frame sequence of the port machinery equipment, specifically consecutive image frames, for example, numbered F t Represents the current frame, F t-1 Representing the previous frame, each frame is first preprocessed by converting it to grayscale to simplify brightness calculations. Then, the current frame F is iterated through. t Compared to the previous frame F t-1 Each pixel in the image is represented by coordinates (i,j), where i is the row index and j is the column index. The absolute difference in brightness values ​​of the corresponding pixel between two frames is calculated and denoted as Δt. i,j =|B t (i,j)-B t-1 (i,j)|, where B t (i,j) and B t-1 (i,j) are the pixel brightness values ​​at position (i,j) in frames t and (t-1) respectively (typically ranging from 0 to 255). The calculation result Δt for all pixels is... i,j Combined, they form a matrix with the same size as a single frame image, namely the time-varying value matrix ΔT. Meanwhile, for the current frame F... t For each pixel (i,j), determine its 3×3 neighborhood, which includes (i,j) and its 8 directly adjacent pixels. Extract the brightness values ​​of these 9 pixels and calculate the standard deviation of these brightness values, denoted as Δs. i,j The calculation formula is: Where N = 9, B k It is the brightness value of the k-th pixel in the neighborhood. It is the average brightness of these 9 pixels, and the calculated result Δs for all pixels. i,j These are combined to form another matrix with the same size as a single frame image, namely the spatial contrast value matrix ΔS, ultimately yielding the time variation value matrix ΔT and the spatial contrast value matrix ΔS.

[0053] formula: The advantage of this formula is that it incorporates the time variation Δt of the pixel.i,j Spatial contrast Δs i,j To calculate the dynamic significance D i,j Temporal variation reflects the intensity of pixel motion or brightness changes, while spatial contrast reflects the texture complexity or edge strength of a pixel and its neighborhood. By multiplying the two and dividing by their maximum value, regions with both significant temporal variation and high spatial contrast (typically corresponding to the edges or textured areas of moving objects) can be effectively amplified, while suppressing regions with only a single significant characteristic (such as uniformly moving areas or static, strongly textured areas). This allows for more precise localization of objects in the video that are actually undergoing significant motion, such as a swaying cable.

[0054] Δt i,j The parameter acquisition steps are as follows: This parameter represents the absolute difference in brightness of pixel (i,j) between two consecutive frames (frame t and frame t-1). First, obtain two consecutive frames F from the input port machinery video frame sequence. t-1 and F t Convert both images to grayscale. Then, for each pixel position (i,j) in the image, read its value in F. t-1 Brightness value B t-1 (i,j) and in F t Brightness value B t (i,j). Calculate the absolute value of the difference between the two values. For example, if a pixel (100,200) has a brightness value of 150 in frame t-1 and a brightness value of 162 in frame t, then the absolute value of the difference between the two values ​​is Δt. 100,200 =|162-150|=12.

[0055] Δs i,j The steps for obtaining the parameter are as follows: This parameter represents the standard deviation of the pixel brightness within the 3×3 neighborhood of pixel (i,j) in frame t, reflecting the local texture complexity or contrast around that point. First, obtain the grayscale image F of frame t. t For each pixel location (i,j) in the image, determine its 3×3 neighborhood, including (i,j) itself and its eight surrounding pixels. Extract the brightness values ​​of these nine pixels. Calculate the average of these nine brightness values. Then, calculate the sum of squares of the differences between the brightness values ​​of each neighboring pixel and the average value, divide by (9-1) = 8, and finally take the square root. For example, the brightness values ​​of a pixel (100, 200) in the 3×3 neighborhood of frame t are [162, 160, 165, 158, 161, 163, 159, 164, 160], and the average value is... Calculate the standard deviation

[0056] The steps to obtain the T1 parameter are as follows: This parameter is a preset dynamic saliency threshold used to filter out sufficiently significant dynamic pixels. The setting of T1 needs to be determined based on the lighting conditions of the actual application scenario, the camera noise level, and the amplitude of shaking to be detected. One setting method is based on empirical statistics: collect a large number of video clips containing normal operation and minor disturbances (such as wind, equipment vibration), and calculate the D of all pixels in these clips. i,j Value, forming D i,j The distribution histogram. Simultaneously, video clips containing abnormal shaking requiring warning are collected, and their D values ​​are calculated. i,j Distribution. Choose a threshold that can effectively distinguish between the two distributions. For example, you can choose D under normal / small perturbation scenarios. i,j The 95th percentile of the value distribution was used as the initial threshold, which was then fine-tuned based on detection performance (e.g., recall and precision) in scenarios with abnormal shaking. D was calculated by analyzing 1000 frames of normal video and 100 frames of video with abnormal shaking. i,j Value. It was found that 95% of D values ​​were normal under normal circumstances. i,j Values ​​below 8.5, while most cable pixels exhibit D values ​​during abnormal shaking. i,j The value exceeds 10. Taking into account both missed detections and false alarms, T1 = 9.0 is selected.

[0057] Calculation process: This formula calculates the dynamic saliency D of pixel (i,j). i,j First, obtain the time change value Δt of this pixel. i,j Spatial contrast value Δs i,j These two values ​​have been calculated. Then, calculate their product: Δt i,j ·Δs i,j Next, find Δt. i,j and Δs i,j The maximum value in: max(Δt) i,j ,Δs i,j Finally, divide the product by the maximum value to get D. i,j .

[0058] Example: Taking the pixel (100, 200) as an example, Δt has been obtained in the previous steps. 100,200 =12Δs 100,200 ≈2.35;

[0059] Calculate D 100,200 :

[0060]

[0061] D 100,200 =2.35;

[0062] Next, the calculated D i,jThe result is compared to a preset dynamic significance threshold T1. In this example, T1 = 9.0. Comparison: D 100,200 Is 2.35 greater than T1 = 9.0? Result: 2.35 ≯ 9.0.

[0063] This result indicates that the dynamic significance D of pixel (100, 200) is... 100,200 The value is 2.35, which does not exceed the preset dynamic saliency threshold T1 = 9.0. Therefore, this pixel is not considered a saliency dynamic point and will be filtered out in the selection process. Only those pixels with D... i,j Only pixels with values ​​greater than T1 are retained for subsequent morphological processing and significant dynamic region generation. This value directly determines which pixels are considered potentially part of a moving object (such as a swaying cable), forming the basis for subsequent analysis. If D i,j If the value is greater than T1, it indicates that the pixel has undergone significant temporal changes and has certain spatial texture / edge features, making it more likely to be a point on a real moving object rather than a minor change in noise or background.

[0064] Based on the selected D i,j The set of pixels under condition T1 forms a binary image, where salient pixels are marked as 1 (foreground) and the rest as 0 (background). Morphological closing operations are then applied to this binary image. Closing typically consists of dilation followed by erosion. A predefined structuring element is used, such as a 3×3 or 5×5 square or circular structuring element. The size of the structuring element is chosen based on the spacing between the pixels to be connected and the size of the internal holes to be filled. For example, a 5×5 square structuring element is used. The dilation operation marks background pixels surrounding the foreground pixels within the structuring element's coverage area as foreground pixels, effectively connecting neighboring salient pixels and filling small holes within the foreground region. The subsequent erosion... The dilation operation removes foreground pixels that cannot be fully contained within the structuring element, helping to smooth region boundaries and remove small noise points or fine connecting bridges, while basically maintaining the size and shape of the main region connected by the dilation operation. After the closing operation, the previously scattered significant pixels are aggregated into more complete and better connected regions. Then, a connected component analysis algorithm, such as graph-based traversal (e.g., depth-first search or breadth-first search) or scanline algorithm, is applied to the processed binary image to identify all independent regions (i.e., connected regions) composed of interconnected foreground pixels. For each identified connected region, its bounding box is calculated, specifically by finding the minimum x-coordinate (column) of all pixels in that region. min and maximum value x max and the minimum value y of the ordinate (row). min and maximum value y maxThis determines the minimum bounding rectangle (MBR) of the region, whose four vertices are (x, y, y). min ,y min ),(x max ,y min ),(x max ,y max ),(x min ,y max These minimum bounding rectangles are defined as salient dynamic regions, containing potential moving targets detected in the image.

[0065] The steps for obtaining the coordinate set of feature points in a single frame are as follows:

[0066] Based on the bounding box coordinate range of the significant dynamic region, the corresponding bounding box interior region in the current frame image is traversed, the pixels are scanned row by row and the gradient magnitude difference with the neighboring pixels is detected, and the pixels with gradient magnitude exceeding the preset gradient threshold are selected to generate a candidate boundary pixel set.

[0067] Based on the candidate boundary pixel set, calculate the density distribution value of candidate points in a 5×5 neighborhood around each pixel, retain pixels with density distribution values ​​greater than a preset density threshold, remove invalid candidate points in isolated or low-density areas, and generate a valid boundary pixel set.

[0068] Based on the set of effective boundary pixels, and according to the rules for defining the origin of the image coordinate system, the integer pairs of horizontal and vertical coordinates of each effective boundary pixel are recorded point by point to form a single-frame feature point coordinate set.

[0069] Specifically, based on the bounding box coordinates of the significant dynamic regions obtained in the previous step, processing is performed within the rectangular region defined by each bounding box for the current frame image. This involves scanning all pixels within the region row by row, and for each pixel (i,j), calculating its gradient magnitude. This is typically achieved by applying a gradient operator (such as the Sobel operator), specifically calculating the horizontal gradient component G. x and the gradient component G in the vertical direction y Then according to The gradient magnitude G is calculated, reflecting the intensity or difference in brightness between pixel (i,j) and its immediate neighbors. Then, the calculated gradient magnitude G for each pixel is compared with a preset gradient threshold. This threshold distinguishes between true edge pixels and pixels with low gradient values ​​caused by noise or flat areas. The threshold is set based on the analysis of sample images containing typical port machinery cables, statistically analyzing the gradient magnitude range of cable edge pixels and the gradient magnitude range of non-edge areas (such as the sky, water surface, and flat parts of the hull). For example, under good lighting conditions, the gradient magnitude of cable edges is typically between 40 and 150 (e.g., brightness range 0-255), while noise or background... Since the gradient magnitude of most areas is below 20, a gradient threshold of 30 can be set to effectively extract edges while suppressing noise. The setting process can be as follows: Select 100 images of cables under different lighting and backgrounds, calculate the gradient magnitude of all pixels and draw a histogram, observe the distribution boundary between edge pixels and non-edge pixels, and select a value that can better separate the two. For example, select a threshold that excludes values ​​with a gradient magnitude below 90% of the background. After testing, it was set to 30. The filtering process is as follows: If the gradient magnitude G of pixel (i,j) is greater than this preset gradient threshold of 30, then the pixel is retained; otherwise, it is discarded. Collect the coordinates of all the filtered pixels (i.e., pixels with a gradient magnitude greater than 30) to generate a set of candidate boundary pixels.

[0070] Based on the generated candidate boundary pixel set, further filtering is needed to remove isolated noise points or unstructured point clouds. For each pixel (i,j) in the candidate boundary pixel set, a 5×5 neighborhood (i.e., a square region centered at (i,j) with a side length of 5 pixels, containing 25 pixel positions) is examined. The number of pixels belonging to the candidate boundary pixel set (including the center point (i,j) itself) within this 5×5 neighborhood is calculated. This number is the density distribution value of pixel (i,j). Then, the calculated density distribution value of each candidate boundary pixel is compared with a preset density threshold. This density threshold is used to determine whether a candidate point is located in a sufficiently dense cluster of boundary points, thus determining whether it might belong to a real object boundary. The threshold is set based on the fact that the boundary of a real object (such as a cable) in an image is usually represented by a series of continuous and high-density pixels, while noise points are often... Isolated or small clusters of noise points have low density. For example, through observation and analysis, it is found that candidate points on real cable boundaries usually contain at least 4 or more other candidate points in their 5×5 neighborhood, while isolated noise points often have only 1 or 2 candidate points in their neighborhood. Therefore, a density threshold of 3 can be set. The setting process can be as follows: Select a test image containing both noise and clear cable boundaries, calculate the density value of the 5×5 neighborhood of all candidate points, statistically analyze the density distribution of real boundary points and noise points, and select a threshold that can effectively distinguish between the two. For example, select a density value that can retain 95% of real boundary points while removing 90% of noise points. After testing, it was set to 3. The screening process is as follows: If the density distribution value of candidate boundary pixel (i,j) (i.e., the number of candidate points in its 5×5 neighborhood) is greater than the preset density threshold of 3, then retain the pixel. Otherwise, it is regarded as an invalid candidate point in an isolated or low-density region and is removed. After this step, the set of retained pixels is the set of valid boundary pixels.

[0071] Based on the set of valid boundary pixels obtained in the previous step, the position information of these pixels needs to be organized into structured data. Specifically, following the rules for defining the origin of the image coordinate system, the top left corner of the image is typically defined as the origin (0,0), the horizontal direction to the right is the positive x-axis (corresponding to the column index), and the vertical direction downwards is the positive y-axis (corresponding to the row index). Each pixel in the set of valid boundary pixels is traversed. For each valid boundary pixel, its integer coordinate pair (x,y) in the image is obtained, where x is the column index value of the pixel and y is the row index value of the pixel. This process is repeated point by point. Integer coordinate values ​​(x, y) are recorded to form a list or array structure. This structure contains the precise location information of all valid boundary points in the current frame image. For example, if the set of valid boundary pixels contains pixels (100, 150) (row 150, column 100), (101, 151) (row 151, column 101), etc., then the recorded coordinate pairs are (100, 150), (101, 151), etc. The coordinate pairs (x, y) of all valid boundary pixels are collected to form the data set, which is the single-frame feature point coordinate set of the current frame.

[0072] The steps for obtaining the shaking feature point sequence are as follows:

[0073] Based on the current frame coordinate point set of the single frame feature point coordinate set, the feature point coordinate sets of two adjacent frames are compared frame by frame. The coordinate offset of each feature point in the neighborhood range of the next frame is calculated. Matching point pairs with the same offset direction and amplitude less than the preset maximum displacement threshold are selected to generate a set of matching feature point pairs.

[0074] Based on the set of matching feature points, the distribution of motion vector direction angles of all matching pairs is statistically analyzed, the median of the direction angles is calculated as the reference value of the main motion trend, and matching pairs whose direction angles deviate from the median by more than 15 degrees are removed to generate a set of effective matching feature points.

[0075] Based on the effective matching feature point set, the feature points are classified according to their unique identifiers. The coordinate values ​​of each feature point in different frames are recorded in time-stamp order and integrated into a continuous coordinate data chain that increases in time. A shaking feature point sequence is then established, where the unique identifier of the feature point is the order number of its appearance in the initial frame.

[0076] Specifically, based on the single-frame feature point coordinate set of the current frame t and the single-frame feature point coordinate set of the next frame t+1 obtained in the previous step, a frame-by-frame feature point matching operation is performed. The specific process is as follows: for each feature point P in the current frame t... t (coordinates are (x) t ,y t In the next frame t+1, the image uses (x) t ,y tA search neighborhood window is defined centered on the location, for example, a square region with a side length of 2W+1 pixels, where W is the search radius determined based on the estimated maximum inter-frame displacement. Within this neighborhood, all candidate feature points P in the single-frame feature point coordinate set of the next frame t+1 are searched. t+1 (coordinates are (x) t+1 ,y t+1 For each found candidate matching point P t+1 Calculate from P t To P t+1 The coordinate offset vector (Δx, Δy) = (x t+1 -x t ,y t+1 -y t The amplitude of the vector, i.e., the Euclidean distance M between the two points, is calculated and compared with a preset maximum displacement threshold. This maximum displacement threshold is determined based on the physical motion limits of the port machinery, the video frame rate, and the camera calibration parameters. For example, if the maximum swaying speed of the cable is approximately 1.5 m / s, the video frame rate is 30 frames / s, and the camera resolution and field of view result in 50 pixels per meter, then the theoretical maximum inter-frame displacement is (1.5 / 30) × 50 = 2.5 pixels. Considering detection errors and occasional rapid movements, the maximum displacement threshold can be set to 5 pixels. The setting process is as follows: analyze the maximum pixel velocity of the cable swaying in historical operating data, calculate the theoretical maximum displacement in combination with the frame rate, and add a certain margin (e.g., 100%) to obtain the final threshold, i.e., 2.5 × (1 + 100%) = 5 pixels. During screening, only potential matching pairs with a displacement amplitude M less than 5 pixels are retained. For a P t If multiple P values ​​satisfying the amplitude threshold are found within the search neighborhood... t+1 A strategy can be used to select the best match, such as selecting the P with the smallest displacement amplitude. t+1 This can be used as a matching point, or all possible matching pairs can be temporarily retained for further filtering in subsequent steps. After the initial filtering, all pairs that meet the conditions (P) will be selected. t ,P t+1 Collect them to generate a set of matching feature point pairs.

[0077] Based on the set of matching feature point pairs generated in the previous step, it is necessary to further filter out incorrect matches by ensuring consistency in motion direction. First, iterate through all matching feature point pairs (P) in the set. t ,P t+1 For each matching pair, calculate its motion vector, i.e., from P t (x t ,y t ) points to P t+1 (x t+1 ,y t+1The vector (Δx, Δy) = (x t+1 -x t ,y t+1 -y t Then, the direction angle of the motion vector is calculated to obtain an angle value representing the direction of motion (e.g., between -180 degrees and +180 degrees). The direction angles of the motion vectors of all matching pairs in the entire set of matching feature points are counted to form an angle list. Next, the median of this angle list is calculated. The median is the middle value after sorting all the angle values. The median is chosen instead of the average value because it is less sensitive to a few mismatches (outliers) with large directional deviations and can more robustly reflect the overall main motion trend. After obtaining this median direction angle, it is used as the current inter-frame value. The main motion trend reference value is used to iterate through each matching pair in the matching feature point pair set again, and the absolute difference between its motion vector direction angle and the main motion trend reference value (i.e., the median angle) is calculated. When comparing, the periodicity of the angle needs to be considered. For example, the difference between 10 degrees and 350 degrees is 20 degrees. If the absolute difference between the direction angle of a matching pair and the median angle exceeds the preset 15 degrees, it is considered that the motion direction of the matching pair deviates greatly from the overall trend and may be an incorrect match. It is then removed from the set, and all matching pairs with direction angle deviations within 15 degrees are retained, finally obtaining the effective matching feature point set.

[0078] Based on the set of valid matching feature points obtained after motion direction consistency filtering, these discrete matching relationships need to be integrated into a continuous motion trajectory, i.e., a shaking feature point sequence. First, a unique identifier is assigned to each independent feature point tracked by the system. This identifier is assigned when the feature point is first stably detected and successfully matched (i.e., when it first appears in the set of valid matching feature points). A simple incrementing integer number can be used. For example, when processing the first pair of frames (e.g., frame 0 to frame 1), all P0 points appearing in the set of valid matching feature points of that frame pair are numbered 1, 2, 3, ... according to their order of appearance (or coordinate order) in frame 0. Then, a data structure is maintained to store the coordinate information of each feature point changing over time. This can be a dictionary or hash table with the unique feature point identifier as the key, and its value is a list of coordinates sorted by timestamp (frame number). When processing the set of valid matching feature points from frame t to frame t+1, for each matching pair (P0... t ,P t+1 ), search for P t The unique identifier of the feature point corresponding to the point (if P) t If P is a point that continues the previous trajectory, then it already has an identifier; tIf the point appears newly in frame t, then an identifier has already been assigned when processing frames t-1 to t; or if it is the first frame match after system startup, then a new identifier is assigned according to the rules. Based on this identifier, the corresponding coordinate list is found, and P is... t+1 coordinates (x) t+1 ,y t+1 ) along with the timestamp t+1, append it to the end of the list if P t If a feature point is newly identified and successfully matched for the first time in the current frame t (e.g., found again after being previously untracked or lost), a new unique identifier is assigned to it, and a new list of coordinates is created, and P... t The coordinates and timestamp t, and P t+1 The coordinates and timestamp t+1 are stored, and this process is repeated frame by frame. This integrates the information in the set of valid matching feature points into a continuous coordinate data chain that increases with time, forming multiple shaking feature point sequences. Each sequence corresponds to a unique identifier and records the motion trajectory of a feature point in consecutive frames.

[0079] The steps for obtaining frame-by-frame shaking feature parameters are as follows:

[0080] Based on the unique identifier of each feature point in the shaking feature point sequence, the coordinate data between two adjacent frames are traversed, the absolute value of the Euclidean distance between the feature point coordinates of the current frame and the corresponding feature point coordinates of the previous frame is calculated, and recorded as the displacement vector size, generating a set of displacement vector sizes.

[0081] Based on the coordinates of the feature points in the current frame, traverse all the unique identifiers of the feature points, calculate the Euclidean distance between every two feature points in the same frame, arrange them in the pairing order of the unique identifiers of the feature points, and generate a set of pairwise distances.

[0082] Based on the set of displacement vector magnitudes and the set of pairwise distances, the displacement vector magnitude data and pairwise distance data of all feature points in the same frame are merged according to the frame number and organized into a displacement-distance parameter matrix indexed by the frame number, generating frame-by-frame shaking feature parameters.

[0083] Specifically, based on the shake feature point sequence established in the previous step, which contains the coordinate information of feature points with unique identifiers in consecutive video frames, it is necessary to calculate the movement distance of each feature point between adjacent frames. The specific operation is as follows: for each pair of adjacent time frames in the video sequence, such as frame t-1 and frame t (where t starts from the second frame of the sequence), iterate through all unique identifiers of feature points that exist simultaneously in these two frames. For each such identifier k, extract its coordinates (x, y, k) in frame t-1 from the shake feature point sequence. t-1,k ,y t-1,k ) and the coordinates (x) in frame t t,k ,y t,kThe Euclidean distance between the two coordinate points is calculated by adding the square of the difference between the x-coordinates and the square of the difference between the y-coordinates, and then taking the square root of the result. This calculated distance value is the displacement vector magnitude of feature point k between frame t-1 and frame t. The displacement vector magnitude of each feature point k between frame t-1 and frame t is recorded and associated with the feature point identifier k and the current frame number t. This calculation process is repeated for all feature points that are successfully tracked between frame t-1 and frame t. All displacement vector magnitude values ​​generated in the same frame transition (i.e., from t-1 to t) are collected together to form the displacement vector magnitude set of the current frame transition.

[0084] Based on the coordinate data of all valid feature points in the current frame t, which can be extracted from the shaking feature point sequence by frame number t, it is necessary to calculate the spatial distribution relationship of these feature points within the same frame, i.e., the pairwise distances between them. Specifically, the operation is to first obtain a list of unique identifiers of all feature points appearing in frame t, for example, {k1,k2,...,k...} N} and their corresponding coordinates Then, systematically iterate through all possible combinations of feature point pairs, for each distinct feature point identifier (k) in the list. i ,k j (where i is not equal to j), extract their respective coordinates in frame t. and To calculate the straight-line distance between two coordinate points, i.e., the Euclidean distance, the calculation method is the same: add the square of the difference between the x-coordinates of the two points to the square of the difference between their y-coordinates, and then take the square root of the result. To avoid double-counting (e.g., k...),... i to k j The distance and k j to k i To ensure consistency of records (where the distances are the same), the points can be arranged and calculated according to the pairing order of their unique identifiers. For example, it is agreed that only identifier k is calculated. i Less than k j Pairing (k) i ,k j Each pairwise distance value calculated is then compared with the corresponding feature point pair identifier (k). i ,k j The distance calculation process is repeated for all possible N(N-1) / 2 feature point pairs in frame t, and all pairwise distance values ​​calculated for frame t are collected to generate the pairwise distance set of that frame.

[0085] Based on the first two steps, which generate the set of displacement vector magnitudes and the set of pairwise distances for each frame (or frame transition), this information needs to be integrated into a unified, frame-organized set of feature parameters. Specifically, using the video frame number t as an index, for each frame t (starting from the second frame, as displacement calculation requires the previous frame), the data related to that frame in the two sets is searched and aggregated: first, the displacement vector magnitudes of all feature points moving from frame t-1 to frame t (from the displacement vector magnitude set); second, the pairwise distances between all feature points within frame t (from the pairwise distance set). The two sets of data are combined to form a comprehensive data record describing the shaking state of frame t. This data record can be organized into a structured form, such as a matrix, a list, or a complex object, which contains all relevant dynamic information (point displacement) and static structural information (point spacing) of the frame. This data structure, which is organized by frame number index, is the displacement-distance parameter matrix. Arranging this matrix for each frame in chronological order yields a complete sequence of frame-by-frame shaking characteristic parameters, providing basic data for subsequent spatiotemporal structure analysis and early warning judgment.

[0086] The steps for obtaining the swaying spatiotemporal structure metric are as follows:

[0087] Based on the set of displacement vector magnitudes in the frame-by-frame shaking feature parameters, the data is divided into segments according to a preset time window length. The standard deviation of all displacement vector magnitudes within each time window is calculated to generate a displacement standard deviation sequence.

[0088] Based on the pairwise distance sets in the frame-by-frame motion feature parameters, the ratio of the sum of the absolute values ​​of the distance differences between feature points in adjacent frames within each time window to the length of the time window is calculated to generate the spatiotemporal change rate. The calculation formula is as follows:

[0089]

[0090] Among them, Q w Let d be the rate of change in time and space. f,p Let d be the distance between the p-th pair of feature points in the f-th frame. f-1,p is the distance between the p-th feature points in the (f-1)-th frame, where P is the total number of feature point pairs and W is the number of frames in the time window;

[0091] Based on the displacement standard deviation sequence and the spatiotemporal change rate, the two are aligned according to the time window number and merged into a binary tuple to generate a swaying spatiotemporal structure metric.

[0092] Specifically, based on the frame-by-frame shaking feature parameters obtained in the previous step, especially the set of displacement vector magnitudes of each feature point between adjacent frames, a time window length W needs to be set first. This length defines the time scale for analyzing the shaking features. Its selection is based on the main periodic characteristics of the cable shaking to be captured and the required early warning response speed. For example, if the cable shaking period is generally around 2 seconds and the video frame rate is 25 frames / second, then W = 50 frames (i.e., 2 seconds) can be selected as the time window length. The setting process can be as follows: analyze historical normal and abnormal shaking video data, and determine the main shaking frequency f through methods such as spectrum analysis. sway The window duration is set to approximately 1 to 2 swing cycles, meaning W is approximately 1 / f. sway × Frame rate to 2 / f sway Between the frame rate and W=50 frames, a sliding window approach is used to process the displacement vector magnitude data. The first time window contains the displacement vector magnitude values ​​of all feature points from frame 2 to frame W+1 (a total of W frames) (these values ​​are the inter-frame displacements from t=2 to t=W+1). The second time window contains the displacement vector magnitude values ​​from frame 3 to frame W+2, and so on. For all displacement vector magnitude values ​​collected in each time window, the standard deviation of these values ​​is calculated. The calculation of the standard deviation follows the conventional statistical method: first, calculate the mean of all displacement values ​​in the window; then, calculate the sum of squares of the differences between each displacement value and the mean; divide the sum of squares by (the total number of values ​​minus one); finally, take the square root of the result. Record the standard deviation value calculated for each time window and associate it with the identifier of that time window (e.g., the end frame number of the window), forming a sequence arranged by time (or window number), i.e., the displacement standard deviation sequence.

[0093] formula: The advantage of the formula is that it calculates the spatiotemporal rate of change Q. w This provides a metric for measuring the drastic changes in the internal structure of a cable (or tracked object) within a time window. It doesn't just focus on the velocity or displacement of individual points, but rather reflects the deformation rate or average intensity of the entire point cloud structure's internal relative motion by calculating the cumulative change in distances between all pairs of feature points. When the cable experiences violent shaking or twisting, the relative distances between feature points change significantly, leading to Q... w The value increases, while the distance between point pairs changes little during steady motion or at rest. w The value is also correspondingly lower, therefore, Q w It can effectively capture the structural dynamic characteristics during the shaking process, providing a basis for distinguishing different shaking modes (such as overall translation and violent deformation shaking). Compared with the standard deviation of displacement alone, it provides supplementary information about the shaking pattern.

[0094] The steps for obtaining the W parameter are as follows: This parameter represents the length of the time window used to calculate the spatiotemporal change rate, in video frames. It defines the time span for analyzing sway characteristics. Its value should be set according to the actual cable sway frequency to be detected and the response time requirements of the early warning system. A shorter window can respond to sway changes faster, but the result may be noisier. A longer window results in smoother and more stable results, but the response is lagging. One setting method is to refer to the main sway period. For example, by analyzing historical monitoring videos or estimating based on the physical characteristics of the port machinery, the typical cable sway period can be determined to be approximately T. period If the video frame rate is FPS (frames per second), then W can be selected at FPS×T. period Up to 2×FPS×T period Within this range, for example, if the shaking period is 2.5 seconds and the frame rate is 20 frames / second, then W can be selected between 50 and 100 frames. Taking into account both real-time performance and stability, W = 60 frames (corresponding to 3 seconds) is selected.

[0095] The steps for obtaining the P parameter are as follows: This parameter represents the total number of feature point pairs that can be continuously tracked and used to calculate distance changes within the calculation time window. This value is not fixed; it depends on the number of feature points that can be stably tracked within the time window (specifically, between each pair of adjacent frames f-1 and f). In actual calculation, for each frame f (from 2 to W) within the window, it is necessary to determine the set of feature point pairs that exist in both frame f-1 and frame f, and calculate the number P of these coexisting feature point pairs. f For example, within a time window of W = 60 frames, if an average of 12 feature points can be stably tracked per frame, then the average number of feature point pairs is approximately 12 × 11 / 2 = 66 pairs. In the example, it is set that during the calculation process of a certain time window, the average number of effective feature point pairs P per frame is 50 pairs.

[0096] d f,p The steps for obtaining the parameter are as follows: This parameter represents the Euclidean distance between the p-th pair of feature points at the f-th frame within the time window. This value is directly derived from the "pairwise distance set" in the "frame-by-frame shaking feature parameters" generated in the previous steps. Specifically, for a frame f in the time window w (f ranges from 1 to W; note that f starts from 2 in the formula, so distance data from frames 1 to W is required), the pairwise distance set of that frame is searched to find the feature point pair identified as p (e.g., by the unique identifier k). i and k j (The pairs are formed), and the distance between them is d. f,p This distance is expressed in pixels or other calibrated physical units (such as meters). For example, for the p=10th pair of feature points (e.g., points with IDs 3 and 8), at frame f=25 within the window, the distance d is found from the corresponding pairwise distance set.25,10 = 31.5 pixels.

[0097] d f-1,p The steps to obtain the parameter are as follows: This parameter represents the value of d at frame f-1 within the time window. f,p The Euclidean distance between the same pair of feature points (i.e., the p-th pair) also originates from the "pairwise distance set" in the "frame-by-frame shake feature parameters". Specifically, for frame f-1 in the time window w (f ranges from 2 to W), the pairwise distance set of that frame is searched to find the same pair of feature points identified as p (e.g., points with IDs 3 and 8), and the distance value between them is read as d. f-1,p For example, for the p=10th pair of feature points, at frame f-1=24 within the window, its distance is found to be d. 24,10 = 30.8 pixels.

[0098] Calculation process: Assume the time window length W = 60 frames. The distance between two pairs of feature points (p = 10, p = 11) in three frames (f = 24, 25, 26) within the window is as follows (unit: pixels): d 24,10 =30.8, d 24,11 =45.2d 25,10 =31.5, d 25,11 =45.0d 26,10 =31.9, d 26,11 =45.3, for example, the average number of effective feature point pairs within the window P = 50 pairs.

[0099] Calculate the inner sum when f = 25 (for examples p = 10, 11 only): |d 25,10 -d 24,10 |+|d 25,11 -d 24,11 |=|31.5-30.8|+|45.0-45.2|=0.7+|-0.2|=0.7+0.2=0.9, for example, the sum calculated for all pairs of P=50 when f=25 is

[0100] Calculate the inner sum when f = 26 (for examples p = 10, 11 only): |d 26,10 -d 25,10 |+|d 26,11 -d 25,11 |=|31.9-31.5|+|45.3-45.0|=0.4+0.3=0.7, for example, the sum calculated for all pairs of P=50 when f=26 is

[0101] For example, the inner sums calculated for all frames from f=2 to W=60 within the entire window are then summed together to obtain the total.

[0102]

[0103] Calculate the spatiotemporal rate of change Q w :

[0104]

[0105] Q w =25.0;

[0106] The result shows that within the 60-frame time window, the sum of the absolute values ​​of the distance changes between all tracked feature point pairs is 25.0 pixels on average per frame.

[0107] Based on the displacement standard deviation sequence and spatiotemporal change rate Q calculated in the first two steps and arranged in time window order, w The sequence needs to integrate these two indicators describing different aspects of the shaking into a comprehensive metric. Specifically, the time window alignment of the two sequences needs to be determined. Since both are calculated based on the same time window length W and the same sliding step size, their window numbers or timestamps (e.g., the end frame number of the window) are one-to-one. For each time window w, the standard deviation value S corresponding to that window is extracted from the displacement standard deviation sequence. w Simultaneously, the rate of change value Q corresponding to this window is extracted from the spatiotemporal rate of change sequence. w , take these two values ​​S w and Q w Combine them into a single binary tuple (or two-dimensional vector), denoted as (S w Q w This pair represents the sway spatiotemporal structure measure of the time window w, which simultaneously includes the sway amplitude within that time period (based on the displacement standard deviation S). w (manifested) and the rate of change of the swaying structure (by the spatiotemporal rate of change Q) w The information (reflected) is used to calculate the tuples (S) for all time windows. w Q w Arranged in window order, a complete time series of oscillating spatiotemporal structure measurements is generated.

[0108] The steps for obtaining the comparison results of the sway index threshold are as follows:

[0109] Based on the displacement standard deviation and spatiotemporal change rate in the swaying spatiotemporal structure measurement, the preset reference values ​​for normal swaying displacement amplitude and swaying frequency of the port machinery equipment cable are called to generate a set of displacement amplitude reference values ​​and swaying frequency reference values.

[0110] Based on the displacement standard deviation sequence and spatiotemporal change rate sequence of the swaying spatiotemporal structure measurement, each item is compared to see if the displacement standard deviation exceeds the reference value of normal swaying displacement amplitude and if the spatiotemporal change rate exceeds the reference value of swaying frequency. Items that meet any of the excess conditions are marked, and swaying index threshold comparison results are generated.

[0111] Specifically, based on the swaying spatiotemporal structure metric generated in the previous step, namely the displacement standard deviation S arranged according to time windows... w and the rate of change of spacetime Q w The sequence of binary tuples (S) w Q w This requires calling pre-set reference benchmarks for subsequent comparisons. These benchmarks are thresholds that distinguish between normal cable sway and potential abnormal sway, specifically including "reference values ​​for normal cable sway displacement amplitude" and "reference values ​​for sway structure change rate." The process for obtaining these two reference values ​​is as follows: First, collect a large amount of historical video data of port machinery equipment under clearly defined normal and safe operating conditions, covering different loads, wind speeds (within allowable ranges), and operation types. Apply all the previous steps to this normal operating condition data to calculate the corresponding S. w and Q w Time series, and then, for S under normal operating conditions w Perform statistical analysis on the sequence and calculate its mean. and standard deviation Set "Normal sway displacement amplitude reference value" A ref This is an upper limit that can cover most normal fluctuations; for example, it can be set as the mean plus three standard deviations. Example: By analyzing 500 normal operating condition time windows, S is obtained. w The mean is 1.8 pixels and the standard deviation is 0.5 pixels, then A ref =1.8 + 3 × 0.5 = 3.3 pixels. Similarly, for Q under normal operating conditions... w Perform statistical analysis on the sequence and calculate its mean. and standard deviation Set the "reference value for the rate of change of the swaying structure" Q ref This is the upper limit under normal circumstances; for example, it is also set to the mean plus three standard deviations. Example: By analyzing the same 500 normal operating condition time windows, Q is obtained. w The mean is 22.0 pixels per frame, and the standard deviation is 6.0 pixels per frame. Therefore, Q... ref =22.0 + 3 × 6.0 = 40.0 pixels / frame distance, these two calculated reference values ​​A ref and Q refThe data is stored (e.g., in a system configuration file) and retrieved (read) in the current step to generate the displacement amplitude reference value A. ref Reference value Q for the rate of change of the swaying structure ref For future use.

[0112] Based on the reference value A of the normal swaying displacement obtained in the previous step. ref Reference value Q for the rate of change of the swaying structure ref And the swaying spatiotemporal structure metric sequence (i.e., (S)) calculated in the aforementioned steps. w Q w For a given time series, each item in the series needs to be compared item by item to determine if it exceeds the normal range. Specifically, this involves iterating through each time window w in the time series and extracting the corresponding oscillation spatiotemporal structure metric tuple (S). w Q w For each pair of tuples, perform the following two checks: The first check is to adjust the displacement standard deviation S. w Reference value A for normal swaying displacement amplitude ref Compare and determine whether condition S is satisfied. w >A ref The second check is to measure the spatiotemporal rate of change Q. w Reference value Q of the rate of change of the swaying structure ref Compare and determine whether condition Q is met. w Q ref Next, based on the results of these two checks, the current time window w is marked. The marking rule is: as long as either (or both) of the above two conditions are met, i.e., S... w Exceeded A ref Or Q w Exceeded Q ref If the time window w is not met, it is marked as "out of limit". If neither condition is met, it is marked as "normal". For example, for a window w = 10, the calculated (S 10 Q 10 ) = (3.5, 35.0), while the reference value is A. ref =3.3, Q ref =40.0, because S 10 =3.5>A ref =3.3, although Q 10 =35.0≯Q ref =40.0, but the first out-of-limit condition is met, so the window w=10 is marked as "out of limit" for all (S) in the sequence. w Q wRepeat this comparison and labeling process, collecting the labeling results for each time window (e.g., "out of limit" or "normal", or represented by Boolean values ​​True / False) in the order of the time windows to generate a sequence of sway index threshold comparison results.

[0113] The steps to obtain the cable sway warning status are as follows:

[0114] Based on the displacement standard deviation sequence and spatiotemporal rate of change sequence in the comparison results of the sway index threshold, extract the displacement standard deviation sequence and spatiotemporal rate of change sequence within the current time window, and obtain the preset displacement amplitude reference value and sway frequency reference value.

[0115] The normalized early warning index is calculated using the following formula:

[0116]

[0117] Among them, E w S is a normalized early warning index. w Q is the standard deviation of displacement (meters). w Let A be the rate of change in spacetime (m / s). ref and Q ref This is a preset reference value;

[0118] Based on the normalized early warning index, determine whether it is an early warning state, count the number of consecutive exceedance windows, and establish a cable sway early warning state.

[0119] Specifically, based on the sway index threshold comparison result sequence obtained in the previous step, and the displacement standard deviation S used in this process... w Sequence and spatiotemporal rate of change Q w For a sequence, given the current time window w to be evaluated, the first step is to extract the specific S corresponding to that window. w Value and Q w In addition, it is necessary to obtain (or confirm that) the preset reference value required for calculating the normalized early warning index, namely the "displacement amplitude reference value" A, again. ref And "swaying frequency reference value" F ref A ref The setting method has been detailed in previous steps (e.g., by analyzing a large number of S under normal operating conditions). w The values ​​were statistically analyzed and set as follows. ), ensuring that its unit is consistent with S w Consistent (e.g., converting pixels to meters via camera calibration), in the example A ref = 0.066 meters, "Swaying Frequency Reference Value" F refThis needs to be set based on the upper limit of the typical swaying frequency of the cable under normal operating conditions. This can be obtained by: analyzing the trajectory data of characteristic points under normal operating conditions, using signal processing methods such as Fast Fourier Transform (FFT) to extract the main swaying frequencies, statistically analyzing the distribution of these frequencies, and setting F... ref This represents the upper limit of the distribution (e.g., the 95th percentile or the mean plus a certain number of standard deviations), in Hertz (Hz). For example, if the analysis finds that the normal swaying frequency is generally no more than 0.5 Hz, then F is set as... ref =0.5Hz, these are the extracted and acquired current window S w Q w Value and preset reference value A ref F ref This will be used in the next step of calculating the normalized early warning index.

[0120] formula: The advantage of the formula is that it aims to account for the two key dimensions of swaying—amplitude (via the standard deviation of displacement S)—... w Relative to reference value A ref (reflected by the degree of change) and dynamic characteristics / structural change rate (through the spatiotemporal change rate Q) w Relative to its reference value Q ref (Or reflected by some frequency-related reference quantity) – integrated into a single, normalized early warning index E w In this method, the deviation between the two dimensions is combined by using the square root of the sum of squares (similar to calculating Euclidean distance), making E... w The value can more comprehensively reflect the abnormality of the overall swaying state; a larger E value... w The value indicates that the swaying deviates significantly from the normal state in terms of amplitude or dynamic complexity, which facilitates the setting of a unified warning threshold.

[0121] S w The steps for obtaining the parameter are as follows: This parameter represents the standard deviation of the displacement vectors of all tracked feature points within the current time window w. Its value is calculated through the aforementioned steps, and the unit needs to be converted from pixels to meters (m) using camera calibration parameters. This value reflects the magnitude and inconsistency of the sway amplitude within the window. For example, the standard deviation of the displacement of the current window w can be calculated as S. w = 0.08 meters.

[0122] Q w The steps for obtaining the parameter are as follows: This parameter represents the spatiotemporal rate of change within the current time window w. Its value is calculated through the aforementioned steps, and the unit needs to be converted from pixels / frame to meters / second (m / s). This value reflects the average rate of change of the feature point cloud structure within the window. For example, the spatiotemporal rate of change of the current window w is Q obtained through calculation and conversion. w = 0.15 m / s.

[0123] A ref The steps for obtaining the parameter are as follows: This parameter is a preset reference value (threshold) for the normal sway displacement amplitude, and the unit is seconds (S). w The same, in meters (m), its setting method is based on a large number of normal operating conditions S w Statistical analysis of the data (e.g., μ+3σ), as described in the preceding steps, for example, setting A... ref = 0.066 meters.

[0124] Q ref The steps for obtaining the parameter are as follows: This parameter is a preset reference value (threshold) for the rate of change of the swaying structure, and its unit is Q. w The same, in meters per second (m / s), is used, and its setting method is based on a large number of normal operating conditions for Q. w Statistical analysis of data, such as Q ref = 0.40 m / s.

[0125] Calculation process: Using parameter value: S w =0.08m, Q w =0.15m / s, A ref =0.066m, Q ref =0.40m / s.

[0126]

[0127] E w ≈1.269;

[0128] The result indicates that the normalized early warning index E of the current time window w is... w Approximately 1.269. This value is a comprehensive measure indicating the degree to which the current swaying state deviates from a preset normal range. E w A value greater than 1 generally means that the swaying exceeds the normal reference threshold in at least one dimension, whether in amplitude or dynamic characteristics. Specifically, a value of 1.269 indicates that the current swaying state has deviated from the normal baseline, and its overall severity is approximately 1.269 times the baseline threshold. This index will be used for the final warning status determination.

[0129] Based on the normalized early warning index E calculated in the previous step w (e.g. E) w (≈1.269), it is necessary to determine whether the current shaking has reached the level that requires issuing a warning, and to establish the final warning state. First, a warning index threshold E is set. threshold This threshold defines E w The setting for determining the level of vibration as abnormal should be based on the value of E. wDistribution analysis under normal and known abnormal shaking conditions, combined with a trade-off between early warning sensitivity and false alarm rate requirements in practical applications, due to E w The way it's constructed ensures that a value near 1 corresponds to a single metric reaching a threshold; typically, E can be... threshold Set it to a value slightly greater than 1, for example, set E. threshold =1.2, Setting process: Select historical data containing both clearly abnormal shaking (requiring warning) and normal shaking, and calculate the corresponding E. w Sequence, plot E w The receiver operating characteristic (ROC) curve was used to select an E curve that could achieve a high detection rate at an acceptable false alarm rate. w The value was used as a threshold, and E was set after analysis. threshold =1.2, then, calculate E in the current window. w Value (1.269) and E threshold (1.2) Compare, if E w ≥E threshold If the current window is determined to be a "potential warning" window (in this example, 1.269 ≥ 1.2, which meets the condition), a continuous judgment mechanism is introduced to avoid false alarms triggered by brief disturbances. This requires counting the number of time windows in which the "potential warning" state occurs consecutively and setting a threshold N for the number of consecutive windows exceeding the limit. consecutive This indicates how many consecutive windows need to be in a potential warning state before a formal warning is triggered. This threshold is set based on the duration of the transient disturbances that need to be filtered out. For example, if you want to ignore disturbances with a duration of less than 6 seconds, and the time window D... w =3 seconds, then N can be set consecutive =6 / 3+1=3 windows (ensuring 6 seconds of coverage), maintain a counter in real time to record the number of potential warning windows that have appeared consecutively, if the current window E w ≥E threshold If E w <E threshold The counter is reset to zero if and only if the value of the counter reaches or exceeds N. consecutive (For example, when the condition reaches 3), the system's cable sway warning status is set to "warning" status; otherwise, the status remains "normal".

[0130] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for early warning of wire rope swaying based on visual perception in port machinery equipment, characterized in that, Includes the following steps: Based on the input video frame sequence of the port machinery equipment, calculate the temporal change value of pixel brightness between consecutive frames and the spatial contrast value of the pixel neighborhood within a single frame, define the bounding box of the connected region, and obtain the significant dynamic region. Based on the significant dynamic region, boundary pixels are extracted in the corresponding region of the current frame image, and the position coordinates of the pixels in the image coordinate system are recorded to obtain a single-frame feature point coordinate set. Based on the single-frame feature point coordinate set, the corresponding feature point coordinates are tracked and matched between consecutive video frames to establish a shaking feature point sequence. Based on the shaking feature point sequence, the displacement vector magnitude of each feature point in the sequence between adjacent time frames is calculated, and the pairwise distance between feature points within the same frame is calculated to obtain frame-by-frame shaking feature parameters. Based on the frame-by-frame shaking feature parameters, the standard deviation of the displacement vector magnitude of all feature points is calculated within a time window to generate a shaking spatiotemporal structure metric. Based on the aforementioned swaying spatiotemporal structure measurement, the preset reference values ​​for normal swaying displacement amplitude and swaying frequency of the port machinery equipment cable are compared to obtain the swaying index threshold comparison result. Based on the swaying index threshold comparison result, a cable swaying early warning state is established.

2. The method for early warning of wire rope swaying based on visual perception in port machinery equipment according to claim 1, characterized in that, The steps for obtaining the significant dynamic region are as follows: Based on the input video frame sequence of the port machinery equipment, calculate the absolute difference in brightness of each pixel (i,j) between frame t and frame t-1 to generate a time variation value matrix. At the same time, calculate the standard deviation of brightness in the 3×3 neighborhood of each pixel (i,j) in frame t to generate a spatial contrast value matrix. Thus, the time variation value matrix and the spatial contrast value matrix are obtained. Based on the time-varying value matrix and the spatial contrast value matrix, the dynamic significance is calculated; Based on the selected pixels, morphological closing operations are used to aggregate adjacent pixels to form connected regions. The minimum bounding rectangle of each connected region is extracted as the bounding box to generate significant dynamic regions.

3. The method for early warning of wire rope swaying based on visual perception in port machinery equipment according to claim 1, characterized in that, The steps for obtaining the single-frame feature point coordinate set are as follows: Based on the bounding box coordinate range of the significant dynamic region, the corresponding bounding box interior region in the current frame image is traversed, the pixels are scanned row by row and the gradient magnitude difference with the neighboring pixels is detected, the pixels with gradient magnitude exceeding the preset gradient threshold are filtered, and a candidate boundary pixel set is generated. Based on the candidate boundary pixel set, calculate the density distribution value of candidate points in a 5×5 neighborhood around each pixel, retain pixels with density distribution values ​​greater than a preset density threshold, remove invalid candidate points in isolated or low-density areas, and generate a valid boundary pixel set. Based on the set of effective boundary pixels, according to the rules for defining the origin of the image coordinate system, the integer pairs of horizontal and vertical coordinates of each effective boundary pixel are recorded point by point to form a single-frame feature point coordinate set.

4. The method for early warning of wire rope swaying based on visual perception in port machinery equipment according to claim 1, characterized in that, The steps for obtaining the shaking feature point sequence are as follows: Based on the current frame coordinate point set of the single frame feature point coordinate set, the feature point coordinate sets of two adjacent frames are compared frame by frame. The coordinate offset of each feature point in the neighborhood range of the next frame is calculated. Matching point pairs with the same offset direction and smaller than the preset maximum displacement threshold are selected to generate a set of matching feature point pairs. Based on the set of matching feature points, the distribution of motion vector direction angles of all matching pairs is statistically analyzed, the median of the direction angles is calculated as the reference value of the main motion trend, and matching pairs whose direction angles deviate from the median by more than 15 degrees are removed to generate a set of effective matching feature points. Based on the set of effective matching feature points, the feature points are classified according to their unique identifiers. The coordinate values ​​of each feature point in different frames are recorded in time-stamp order and integrated into a continuous coordinate data chain that increases in time. A shaking feature point sequence is then established, where the unique identifier of the feature point is the initial frame appearance sequence number.

5. The method for early warning of wire rope swaying based on visual perception in port machinery equipment according to claim 1, characterized in that, The steps for obtaining the frame-by-frame shaking feature parameters are as follows: Based on the unique identifier of each feature point in the shaking feature point sequence, the coordinate data between two adjacent frames are traversed, the absolute value of the Euclidean distance between the feature point coordinates of the current frame and the corresponding feature point coordinates of the previous frame is calculated, and recorded as the displacement vector size, generating a set of displacement vector sizes. Based on the coordinates of the feature points in the current frame, traverse all the unique identifiers of the feature points, calculate the Euclidean distance between every two feature points in the same frame, arrange them in the pairing order of the unique identifiers of the feature points, and generate a set of pairwise distances. Based on the set of displacement vector magnitudes and the set of pairwise distances, the displacement vector magnitude data and pairwise distance data of all feature points in the same frame are merged according to the frame number and arranged into a displacement-distance parameter matrix indexed by the frame number to generate frame-by-frame shaking feature parameters.

6. The method for early warning of wire rope swaying based on visual perception in port machinery equipment according to claim 1, characterized in that, The steps for obtaining the swaying spatiotemporal structure metric are as follows: Based on the set of displacement vector magnitudes in the frame-by-frame shaking feature parameters, the data segments are divided according to the preset time window length, the standard deviation of all displacement vector magnitudes in each time window is calculated, and a displacement standard deviation sequence is generated. Based on the pairwise distance set in the frame-by-frame shaking feature parameters, the ratio of the sum of the absolute values ​​of the distance differences between feature points between adjacent frames within each time window to the length of the time window is calculated to generate the spatiotemporal change rate. Based on the displacement standard deviation sequence and the spatiotemporal change rate, the two are aligned according to the time window number and merged into a binary tuple to generate a swaying spatiotemporal structure metric.

7. The method for early warning of wire rope swaying based on visual perception in port machinery equipment according to claim 1, characterized in that, The steps for obtaining the comparison result of the sway index threshold are as follows: Based on the displacement standard deviation and spatiotemporal change rate in the swaying spatiotemporal structure measurement, the preset reference values ​​for normal swaying displacement amplitude and swaying frequency of the port machinery equipment cable are called to generate a set of displacement amplitude reference values ​​and swaying frequency reference values. Based on the displacement standard deviation sequence and spatiotemporal change rate sequence of the swaying spatiotemporal structure measurement, each item is compared to see if the displacement standard deviation exceeds the reference value of normal swaying displacement amplitude and if the spatiotemporal change rate exceeds the reference value of swaying frequency. Items that meet any of the excess conditions are marked, and swaying index threshold comparison results are generated.

8. The method for early warning of wire rope swaying based on visual perception in port machinery equipment according to claim 1, characterized in that, The steps for obtaining the cable sway warning status are as follows: Based on the displacement standard deviation sequence and spatiotemporal rate of change sequence in the comparison results of the sway index threshold, the displacement standard deviation sequence and spatiotemporal rate of change sequence within the current time window are extracted, and preset displacement amplitude reference value and sway frequency reference value are obtained. Calculate the normalized early warning index; Based on the normalized early warning index, determine whether it is an early warning state, count the number of consecutive over-limit windows, and establish a cable sway early warning state.

Images