A detection method and system for abnormal operation of a crane trolley frame

Abstract

The present invention discloses a method and system for detecting abnormal operation of a crane trolley frame. The method includes: installing a target on the crane trolley frame; acquiring a monitoring video image; detecting the monitoring video image based on an object detection model to obtain the target area; obtaining the coordinates of the target corner points based on the sub-pixel accuracy method and the coordinate transformation method; obtaining the motion trajectory of the target corner points on the spatial axis and the vibration signal on the time axis; and judging whether the crane trolley frame operates abnormally according to the motion trajectory of the target corner points on the spatial axis and the vibration signal on the time axis. The technical solution provided by the present invention realizes the real-time monitoring and accurate detection of the operation state of the trolley frame, can significantly improve the accuracy of abnormal detection, comprehensively evaluate and real-time perceive the operation state of the trolley frame, and give an abnormal warning in time, so as to maximize the stable operation of the equipment and avoid production interruption and safety risks caused by sudden failures.

Description

Technical Field

[0001] The present invention relates to the field of crane detection, and particularly to a method and system for detecting abnormal operation of a crane trolley frame. Background Art

[0002] With the continuous growth of the global port throughput, the working intensity and load pressure of cranes are also increasing continuously. As one of the core components of a crane, the stability of the working state of the trolley becomes particularly important. However, at present, the maintenance of the crane trolley frame still relies on traditional passive shutdown maintenance. This method has low maintenance efficiency, cannot predict the potential risks of the equipment in advance, and cannot obtain the operating state and health status of the equipment in real time. This not only affects the continuity of port operations but also may pose safety hazards. Summary of the Invention

[0003] In view of the above deficiencies in the current technology, the present invention provides a method for detecting abnormal operation of a crane trolley frame. By installing a target on the crane trolley frame, using a target detection model to detect the target image, determining the approximate position of the target corner points, and using the sub-pixel accuracy method and coordinate transformation method to obtain the coordinates of the target corner points, and by analyzing the movement trajectory and vibration signal of the target corner points, it is judged whether there is abnormal deviation or failure of the trolley frame.

[0004] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0005] A method for detecting abnormal operation of a crane trolley frame includes the following steps:

[0006] Install a target including corner points on the crane trolley frame;

[0007] Obtain a monitoring video image of the position of the crane trolley frame including the target;

[0008] Based on the target detection model, detect the monitoring video image to obtain the target area;

[0009] According to the target area, based on the sub-pixel accuracy method and coordinate transformation method, obtain the coordinates of the target corner points;

[0010] According to the coordinates of the target corner points, obtain the movement trajectory of the target corner points on the spatial axis and the vibration signal on the time axis;

[0011] According to the movement trajectory of the target corner points on the spatial axis and the vibration signal on the time axis, judge whether the crane trolley frame is operating abnormally.

[0012] According to one aspect of the present invention, the target including corner points is a high-contrast double-sided checkerboard target.

[0013] According to one aspect of the present invention, the detection of the surveillance video image by the target detection model to obtain the target area includes:

[0014] Using the YOLO, YOLOv8 or improved YOLOv8 target detection model to detect the surveillance video image and identify the target area.

[0015] According to one aspect of the present invention, the improved YOLOv8 model includes:

[0016] The Backbone layer extracts the features of the surveillance video image based on the convolutional module, C2f module, and SPPF module;

[0017] The Neck layer uses the features extracted by the Backbone layer to perform multi-scale feature fusion based on upsampling, feature splicing, convolutional module, C2f module, and BiFPN structure to obtain the fused features;

[0018] The Head layer detects the target area in the image according to the fused features.

[0019] According to one aspect of the present invention, the obtaining of the coordinates of the target corner points based on the sub-pixel accuracy method and the coordinate transformation method according to the target area includes:

[0020] Analyze the gray-scale change of the pixels in the target area by the sub-pixel accuracy method, and estimate the corner point position by the interpolation method to obtain the pixel coordinates of the target corner points;

[0021] Obtain the camera parameters and pose information of the surveillance camera;

[0022] Use the coordinate transformation method to inversely solve the accurate world coordinates of the target corner points.

[0023] According to one aspect of the present invention, the analyzing of the gray-scale change of the pixels in the target area by the sub-pixel accuracy method, and the estimating of the corner point position by the interpolation method to obtain the pixel coordinates of the target corner points includes:

[0024] Extract the rectangular area image at the position where the corner points are located from the target area;

[0025] Generate the gray-scale image of the rectangular area image;

[0026] Use the sobar operator to calculate the gradient values of each point in the horizontal direction of the gray-scale image;

[0027] Select the point with the largest gradient value and determine its position of the single-pixel edge point in the rectangular area;

[0028] Starting from the single-pixel edge point, select two adjacent points in the horizontal or vertical direction and fit them with a quadratic polynomial curve;

[0029] Substitute the gradient values of a single-pixel edge point and its two adjacent points, along with their corresponding pixel coordinates, into the quadratic polynomial curve equation for solution to obtain the precise coordinates of the target point in the horizontal or vertical direction.

[0030] According to one aspect of the present invention, the method for inversely solving the precise world coordinates of the target corner points by using the coordinate transformation method includes:

[0031] Based on the inverse solution PnP method, use the camera parameters and pose information of the monitoring camera, as well as the pixel coordinates of the target corner points, to inversely solve the precise world coordinates of the target corner points.

[0032] According to one aspect of the present invention, the method for obtaining the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis based on the coordinates of the target corner point includes:

[0033] Detect and process consecutive frames in the video image to obtain several trajectory points of the target corner point;

[0034] Based on the several trajectory points of the target corner point, obtain the motion trajectory of the target corner point on the spatial axis;

[0035] Obtain the deviation between the actual position of the target corner point and its ideal trajectory;

[0036] Based on the deviation between the actual position of the target corner point and its ideal trajectory, decompose the vibration signal of the target corner point on the time axis.

[0037] According to one aspect of the present invention, the method for determining whether the crane trolley frame is operating abnormally based on the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis includes:

[0038] If the deviation between the actual position of the target corner point and its ideal trajectory exceeds the preset offset threshold, the crane trolley frame is operating abnormally;

[0039] Perform time-domain and frequency-domain statistical feature analysis on the vibration signal of the target corner point on the time axis to determine whether the crane trolley frame is operating abnormally.

[0040] A crane trolley frame operation abnormality detection system, based on the above-mentioned crane trolley frame operation abnormality detection method, includes:

[0041] An installation module for installing a target including corner points on the crane trolley frame;

[0042] A collection module for acquiring a monitoring video image of the position of the crane trolley frame including the target;

[0043] A detection module for detecting the monitoring video image based on the target detection model to obtain the target area;

[0044] A coordinate transformation module, configured to obtain the coordinates of the target corner points based on the sub-pixel accuracy method and the coordinate transformation method according to the target area;

[0045] An analysis module, configured to obtain the motion trajectory of the target corner points on the spatial axis and the vibration signal on the time axis according to the coordinates of the target corner points;

[0046] A judgment module, configured to judge whether the crane trolley frame is operating abnormally according to the motion trajectory of the target corner points on the spatial axis and the vibration signal on the time axis.

[0047] Advantages of the implementation of the present invention:

[0048] The present invention provides a method for detecting abnormal operation of a crane trolley frame. By installing a target on the crane trolley frame, detecting the target image using a target detection model to determine the approximate position of the target corner points, and using the sub-pixel accuracy method and the coordinate transformation method to obtain the coordinates of the target corner points, it is judged whether there is abnormal deviation or fault of the trolley frame by analyzing the motion trajectory and vibration signal of the target corner points.

[0049] This method constructs a multi-dimensional state evaluation system, realizes real-time monitoring and accurate detection of the operation state of the trolley frame, can not only significantly improve the accuracy of abnormal detection, but also comprehensively evaluate and real-time perceive the operation state of the trolley frame, give an abnormal warning in time, so as to maximize the guarantee of the stable operation of the equipment and avoid production interruption and safety risks caused by sudden failures. Description of the Drawings

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to these drawings.

[0051] Figure 1 It is a flowchart of a method for detecting abnormal operation of a crane trolley frame according to the present invention;

[0052] Figure 2 It is a schematic structural diagram of a crane according to the present invention;

[0053] Figure 3 It is a schematic diagram of target detection according to the present invention;

[0054] Figure 4 It is a general processing flowchart of a method for detecting abnormal operation of a crane trolley frame according to the present invention;

[0055] Figure 5 It is an improved YOLOv8 model architecture according to the second embodiment of the present invention.

[0056] Reference numerals: 1, front girder of crane; 2, high-definition camera; 3, trolley frame; 4, double-sided target; 5, PLC; 6, server; 7, electrical room; 8, rear girder of crane. Specific implementation mode

[0057] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0058] Embodiment 1

[0059] As Figure 1 shown, a method for detecting abnormal operation of a crane trolley frame includes the following steps:

[0060] S1: Install a target including corner points on the crane trolley frame.

[0061] In practical applications, as Figure 2 shown, high-definition cameras are deployed on the front girder and rear girder of the crane. The high-definition cameras are high-frame-rate cameras, and the installation angle is the horizontal direction, covering the moving area of the trolley frame. A double-sided target including corner points is installed on the crane trolley frame, and the target is a checkerboard target with high contrast.

[0062] The high-definition cameras on the front girder and rear girder can collect images of the double-sided target on the trolley frame in real time from the front and rear ends, which can effectively avoid the loss of the double-sided target caused by the change of the trolley frame position. In addition, the high-definition cameras have the ability to collect high-frame-rate videos, providing data support for subsequent acquisition of the vibration signals of the target corner points.

[0063] There is a PLC and a server in the electrical room of the crane. The server is connected to the high-definition camera and the PLC. The server is connected to the high-definition camera through optical fiber or 5G communication for real-time transmission of image data.

[0064] S2: Obtain the monitoring video image of the position of the crane trolley frame including the target.

[0065] The high-definition cameras deployed on the front girder and rear girder monitor the working state of the trolley frame in real time. By obtaining the images of the trolley frame from the front and rear ends, the loss of the target caused by equipment limitations can be avoided, thus ensuring a clear picture. At the same time, the video data collected by the high-definition cameras will be transmitted to the server, and the PLC will send the signal for controlling the movement of the trolley frame to the server.

[0066] The server in the electrical room receives the data transmitted by the PLC. The PLC signal contains information about the moving distance of the trolley frame, so the server can obtain the current position of the trolley frame. Based on the current position, the server then adjusts the focal length of the high-definition camera for focusing to ensure a clear video of the double-sided target is captured.

[0067] S3: Detect the surveillance video image based on the object detection model to obtain the target area.

[0068] Use the pre-trained object detection model to quickly locate the approximate position of the upper corner points of the double-sided target in the image and mark the area.

[0069] In practical applications, object detection models such as YOLO, YOLOv8, other improved versions of YOLO, SSD, Faster R-CNN, etc. can be used to detect and recognize the surveillance video image and frame the area where the target is located.

[0070] For example, the YOLOv8 model can be used for detection. Specifically, the YOLOv8 model architecture includes:

[0071] (1) Backbone layer, which extracts the features of the surveillance video image based on the convolutional module, C2f module, and SPPF module.

[0072] (2) Neck layer, which uses the features extracted by the Backbone layer and performs multi-scale feature fusion based on upsampling, feature splicing, convolutional module, C2f module, and FPN+PAN structure to obtain the fused features. The Neck layer adopts the FPN+PAN structure, where FPN represents the Feature Pyramid Network and PAN represents the Path Aggregation Network. Through bidirectional feature transfer, the full fusion of feature maps at different scales is achieved, retaining both the semantic information of the high layer and using the localization information of the low layer, thus significantly improving the performance of object detection.

[0073] (3) Head layer, which detects and locates the area where the target is located in the image according to the fused features.

[0074] The training of the YOLOv8 model includes:

[0075] (1) Collect the videos recorded during the operation of the trolley frame with the target, and calibrate the corner positions of the double-sided target in the videos to generate the training database.

[0076] (2) Divide the training database into a training set and a test set in a ratio of 8:2. Use the training set to train the YOLOv8 model to obtain the object detection model and evaluate its performance through the test set.

[0077] (3) By adjusting the hyperparameters of the model, the object detection model with the best detection effect is selected as the final version to be used in the actual working scenario.

[0078] S4: According to the target area, based on the sub-pixel accuracy method and the coordinate transformation method, obtain the coordinates of the target corner points.

[0079] Specifically, step S4 includes:

[0080] S41: Analyze the gray-scale change of the pixels in the target area using the sub-pixel accuracy method, and estimate the corner point position using the interpolation method to obtain the pixel coordinates of the target corner points.

[0081] Step S41 includes:

[0082] S411: Extract the rectangular area image of the corner point position from the target area.

[0083] In practical applications, a rectangular area of 5×5 pixels can be extracted.

[0084] S412: Generate the gray-scale image of the rectangular area image.

[0085] S413: Use the sobar operator to calculate the gradient value of each point in the horizontal direction of the gray-scale image.

[0086] The sobar operator is: .

[0087] S414: Select the point with the maximum gradient value and determine its position as the single-pixel edge point within the rectangular area.

[0088] S415: Starting from the single-pixel edge point, select two adjacent points in the horizontal or vertical direction and perform fitting using the quadratic polynomial curve.

[0089] Starting from the single-pixel edge point X, select two adjacent points in the horizontal direction and , and on this basis, use the quadratic polynomial curve to perform fitting.

[0090] S416: Substitute the gradient values and corresponding pixel coordinates of the single-pixel edge point and its two adjacent points into the quadratic polynomial curve equation for solution to obtain the precise coordinates of the target point in the horizontal or vertical direction.

[0091] Substitute the gradient values and corresponding pixel coordinates of the above three points into the equation for solution to determine the precise coordinate x of the target point in the horizontal direction i :

[0092] .

[0093] Similarly, the exact coordinate y of the target in the vertical direction can be determined. i 。

[0094] Among them, (x i , y i ) represents the position of the target point, i is the i-th pixel point, x is the horizontal coordinate of the edge pixel point, and f(x) is the gradient value corresponding to the horizontal position of the pixel point.

[0095] S42: Obtain the camera parameters and pose information of the surveillance camera.

[0096] S43: Use the coordinate transformation method to inversely solve the exact world coordinates of the target corner points.

[0097] Step S43 includes:

[0098] Based on the inverse solution PnP method, use the camera parameters and pose information of the surveillance camera and the pixel coordinates of the target corner points to inversely solve the exact world coordinates of the target corner points.

[0099] The PnP (Perspective-n-Point) method is a method for solving the motion of 3D to 2D point pairs, that is, given the coordinates of n 3D points in the world coordinate system and the coordinates of these points in the image (or pixel) coordinate system, and at the same time knowing the internal parameter matrix of the camera, solve the pose of the camera coordinate system relative to the world coordinate system. Here, the pose usually includes the rotation matrix R and the translation vector t. Conversely, given the camera parameters, camera pose information, and camera coordinate system information, the world coordinate system information can also be inversely solved.

[0100] The specific formula of the PnP method is:

[0101] ,

[0102] ,

[0103] ,

[0104] ,

[0105] .

[0106] Among them, s is the scaling factor or the depth of the target point, that is, the Z coordinate of this 3D point in the camera coordinate system; (u i , v i , 1) is the homogeneous coordinate of the target point in the pixel coordinate system; (Xc, Yc, Zc) is the coordinate of the target point in the camera coordinate system; (X i , Y i, (Z) is the coordinate of the target point in the world coordinate system, and Z is the position where the current trolley frame is located; K is the internal parameter matrix of the camera, fx and fy are the parameters of the camera focal length, and c x and c y are the offsets of the principal point of the camera; the external parameter matrix of the camera describes the position of the camera in the world coordinate with two parameters, the rotation matrix R and the translation vector t, corresponding to the rotation and translation of the camera in the world coordinate system respectively; r *1 and r *2 and r *3 represent the directions of the x, y, and z axes of the pixel coordinate system in the world coordinate respectively, and the translation vector t is the coordinate of the camera in the world coordinate system.

[0107] Substitute the known camera parameters, camera pose information, and target corner pixel coordinate information into the above formula for solution, and the world coordinate information of the target corner can be inversely solved.

[0108] S5: According to the coordinates of the target corner, obtain the motion trajectory of the target corner on the spatial axis and the vibration signal on the time axis.

[0109] Specifically, step S5 includes:

[0110] S51: Detect and process consecutive frames in the video image to obtain several trajectory points of the target corner.

[0111] S52: According to several trajectory points of the target corner, obtain the motion trajectory of the target corner on the spatial axis.

[0112] S53: Obtain the deviation between the actual position of the target corner and its ideal trajectory.

[0113] S54: According to the deviation between the actual position of the target corner and its ideal trajectory, decompose the vibration signal of the target corner on the time axis.

[0114] As Figure 3 shown, the vibration signal of the target on the time axis comes from decomposing the motion trajectory of the corner on the target on the spatial axis through the formula, and the decomposition formula is: .

[0115] Among them, ρ represents the deviation vector between the actual position and the ideal position of a certain corner on the target, and x and y respectively represent the offsets of this point in the horizontal and vertical directions of the trolley frame, that is, the vibration signals in the corresponding directions.

[0116] S6: According to the motion trajectory of the target corner on the spatial axis and the vibration signal on the time axis, judge whether the crane trolley frame is operating abnormally.

[0117] Specifically, step S6 includes:

[0118] S61: If the deviation between the actual position of the target corner point and its ideal trajectory exceeds the preset offset threshold, the crane trolley frame will operate abnormally.

[0119] At a certain position of the trolley frame, the ideal position of the target corner point is the ideal world coordinate after ignoring external interference, while the actual position is its true world coordinate. After knowing the world coordinates of the corner points on the target, the deviation distance between the actual position of the target corner point on the current trolley frame and its ideal trajectory can be compared to determine whether the deviation exceeds the set offset threshold, thereby determining whether the trolley frame has abnormal offset.

[0120] S62: Perform time-domain and frequency-domain statistical feature analysis on the vibration signal of the target corner point over the time axis to determine whether the crane trolley frame operates abnormally.

[0121] Based on the vibration signals of the target corner point in the x and y axis directions obtained in step S54, by analyzing the time-domain and frequency-domain statistical features, it can be determined whether there are faults in the trolley frame devices. Time-domain analysis mainly extracts features such as the mean, variance, and kurtosis of the signal to characterize the fluctuation characteristics and trend changes of the vibration signal; frequency-domain analysis can reveal the main frequency components and their energy distribution of the signal through power spectrum analysis.

[0122] For example, the server can use the power spectral density of the vibration signal to determine whether there is an abnormality in the components on the trolley frame, taking the vibration signal in the horizontal direction as an example:

[0123] (1) Window the signal using a Hamming window:

[0124] .

[0125] (2) Perform a fast Fourier transform on the windowed signal:

[0126] .

[0127] (3) Calculate the power spectral density of the signal:

[0128] ,

[0129] .

[0130] (4) Analyze the power spectral density of the signal to check whether the power of a certain frequency component is abnormally high.

[0131] In the above formula, x[n] is the discrete vibration signal sequence in the x-axis direction, n is the sequence index, N is the total number of sequences, x i is the offset of the corner point in the horizontal direction at a certain moment, w[n] is the selected window function, xw [n] is the sequence of vibration signals after windowing; X[k] is the complex frequency-domain signal, k is the discrete frequency index, △f is the frequency resolution, fs is the total frequency, and PSD[k] is the power of the signal in the frequency domain.

[0132] If the power of the vibration signal is abnormally high at a certain frequency, it indicates that there may be a fault in the device on the trolley frame.

[0133] If the analysis result shows that the trolley frame has an abnormal offset or there may be a fault in the trolley frame components, the server will determine that the status is abnormal and send an alarm signal to notify the staff to handle it in time. If there is no abnormality and the trolley frame stops working, the system will stop detection to ensure the normal progress of the operation.

[0134] Figure 4 The overall processing flow of this method is shown as follows. In practical applications, the image data at the target on the crane trolley frame can be collected in real time for detection to determine whether there is an abnormal offset or fault in the trolley frame, which is convenient for timely processing.

[0135] The beneficial effect of this embodiment is as follows: By installing a target on the crane trolley frame, this method uses the target detection model to detect the target image, determines the approximate position of the target corner points, and uses the sub-pixel accuracy method and coordinate transformation method to obtain the coordinates of the target corner points. By analyzing the motion trajectory and vibration signals of the target corner points, it is judged whether there is an abnormal offset or fault in the trolley frame.

[0136] This method constructs a multi-dimensional state evaluation system, realizes the real-time monitoring and accurate detection of the running state of the trolley frame, not only can significantly improve the accuracy of abnormal detection, but also can comprehensively evaluate and real-time perceive the running state of the trolley frame, give an early warning of abnormalities in time, so as to maximize the guarantee of the stable operation of the equipment and avoid production interruption and safety risks caused by sudden failures. This method solves the problems of low detection accuracy and insufficient real-time performance in traditional methods, can effectively improve the detection efficiency and reliability, provides a strong guarantee for the safe and stable operation of crane equipment, and provides strong technical support for the safety and reliability of port operations.

[0137] Embodiment 2

[0138] As Figure 1 shown, a method for detecting abnormal operation of a crane trolley frame includes the following steps:

[0139] S1: Install a target including corner points on the crane trolley frame.

[0140] In practical applications, as Figure 2As shown, high-definition cameras with high frame rates are deployed on the front and rear girders of the crane. They are installed at a horizontal viewing angle to cover the moving area of the trolley frame. A double-sided target with corner points is installed on the crane trolley frame. The target is a high-contrast checkerboard target.

[0141] High-definition cameras on the front and rear beams capture real-time images of the double-sided target on the trolley frame from both ends, effectively preventing loss of the double-sided target due to changes in the trolley frame's position. Furthermore, the high-definition cameras' high-frame-rate video capture capability provides data support for subsequent acquisition of vibration signals from the target's corners.

[0142] The crane's electrical room houses a programmable logic controller (PLC) and server, which are connected to the HD camera and PLC. The server and HD camera are connected via optical fiber or 5G communication for real-time image data transmission.

[0143] S2: Acquire a monitoring video image of the crane trolley frame position including the target.

[0144] High-definition cameras deployed on the front and rear beams monitor the trolley's operating status in real time. By capturing images of the trolley from both the front and rear ends, they avoid target loss due to equipment limitations and ensure clear images. Simultaneously, the video data captured by the HD cameras is transmitted to a server, and the PLC sends signals to the server to control the trolley's movement.

[0145] A server in the electrical room receives data from the PLC. The PLC signal contains information about the distance the trolley has moved, allowing the server to determine its current location. Based on this location, the server then adjusts the focus of the HD camera to ensure clear video of the double-sided target.

[0146] S3: Detect the surveillance video image based on the target detection model to obtain the target area.

[0147] Use the pre-trained object detection model to quickly locate the approximate position of the corner points on the two-sided target in the image and mark the area.

[0148] In practical applications, target detection models such as YOLO, YOLOv8, other improved versions of YOLO, SSD, and Faster R-CNN can be used to detect and identify surveillance video images and frame the target area.

[0149] For example, an improved YOLOv8 model can be used for detection. Specifically, the improved YOLOv8 model architecture includes:

[0150] (1) Backbone layer, which extracts the features of the monitored video image based on convolutional modules, C2f modules, and SPPF modules.

[0151] (2) Neck layer, which uses the features extracted by the Backbone layer to perform multi-scale feature fusion based on upsampling, feature concatenation, convolutional modules, C2f modules, and BiFPN structure to obtain fused features.

[0152] The original YOLOv8 model's Neck layer uses the FPN (Feature Pyramid Network) + PAN (Path Aggregation Network) structure. In this method, as Figure 5 shown, the BiFPN (Bidirectional Feature Pyramid Network) structure is used to replace the original FPN + PAN structure. BiFPN includes efficient bidirectional cross-scale connections, simplified network structures, weighted feature fusion, and bidirectional feature fusion, which can achieve more efficient and accurate multi-scale feature fusion, thereby improving the detection accuracy of the model.

[0153] (3) Head layer, which detects and locates the area where the target is located in the image according to the fused features.

[0154] The training of the improved YOLOv8 model includes:

[0155] (1) Collect the videos recorded during the operation of the trolley frame equipped with the target, and calibrate the corner positions of the double-sided target in the videos to generate a training database.

[0156] (2) Divide the training database into a training set and a test set according to a ratio of 8:2. Use the training set to train the improved YOLOv8 model to obtain a target detection model, and evaluate its performance through the test set.

[0157] (3) By adjusting the hyperparameters of the model, select the target detection model with the best detection effect as the final version used in the actual working scenario.

[0158] S4: According to the target area, based on the sub-pixel accuracy method and coordinate transformation method, obtain the coordinates of the target corner points.

[0159] Specifically, step S4 includes:

[0160] S41: Analyze the gray-scale change of the pixels in the target area using the sub-pixel accuracy method, and estimate the corner position using the interpolation method to obtain the pixel coordinates of the target corner points.

[0161] Step S41 includes:

[0162] S411: Extract the rectangular region image at the position of the corner point from the target area.

[0163] In practical applications, a rectangular region of 5×5 pixels can be extracted.

[0164] S412: Generate a grayscale image of the rectangular region image.

[0165] S413: Use the sobar operator to calculate the gradient values of each point in the horizontal direction of the grayscale image.

[0166] The sobar operator is: .

[0167] S414: Select the point with the maximum gradient value and determine the position of its single-pixel edge point within the rectangular region.

[0168] S415: Starting from the single-pixel edge point, select two adjacent points in the horizontal or vertical direction and perform fitting using a quadratic polynomial curve. [[ID=2�]]

[0169] Starting from this single-pixel edge point X, select two adjacent points in the horizontal direction and , and on this basis, use the quadratic polynomial curve to perform fitting.

[0170] S416: Substitute the gradient values of the single-pixel edge point and its two adjacent points and their corresponding pixel coordinates into the quadratic polynomial curve equation for solution to obtain the exact coordinates of the target point in the horizontal or vertical direction.

[0171] Substitute the gradient values of the above three points and their corresponding pixel coordinates into the equation for solution to determine the exact coordinates of the target point in the horizontal direction :

[0172] .

[0173] Similarly, the exact coordinates of the target in the vertical direction can be determined .

[0174] Among them, (x i , y i ) represents the position of the target point, i is the i-th pixel point, x is the horizontal coordinate of the edge pixel point, and f(x) is the gradient value corresponding to the horizontal position of the pixel point.

[0175] S42: Obtain the camera parameters and pose information of the monitoring camera.

[0176] S43: Use the coordinate transformation method to inversely solve the exact world coordinates of the target corner point.

[0177] Step S43 includes:

[0178] Based on the inverse solution PnP method, using the camera parameters and pose information of the monitoring camera and the pixel coordinates of the target corner points, the exact world coordinates of the target corner points are inversely solved.

[0179] The PnP (Perspective-n-Point) method is a method for solving the motion of 3D to 2D point pairs, that is, given the coordinates of n 3D points in the world coordinate system and the coordinates of these points in the image (or pixel) coordinate system, and at the same time knowing the internal parameter matrix of the camera, the pose of the camera coordinate system relative to the world coordinate system is solved. Here, the pose usually includes the rotation matrix R and the translation vector t. Conversely, given the camera parameters, camera pose information, and camera coordinate system information, the world coordinate system information can also be inversely solved.

[0180] The specific formula of the PnP method is:

[0181] ,

[0182] ,

[0183] ,

[0184] ,

[0185] .

[0186] Among them, s is the scaling factor or the depth of the target point, that is, the Z coordinate of this 3D point in the camera coordinate system; (u i , v i , 1) is the homogeneous coordinate of the target point in the pixel coordinate system; (Xc, Yc, Zc) is the coordinate of the target point in the camera coordinate system; (X i , Y i , Z) is the coordinate of the target point in the world coordinate system, Z is the position where the current trolley frame is located; K is the internal parameter matrix of the camera, fx and fy are the parameters of the camera focal length, c x , c y are the offsets of the principal point of the camera; the external parameter matrix of the camera describes the position of the camera in the world coordinate with two parameters, the rotation matrix R and the translation vector t, corresponding to the rotation and translation of the camera in the world coordinate system respectively; r *1 , r *2 , r *3 in the rotation matrix R respectively represent the directions of the x, y, and z axes of the pixel coordinate system in the world coordinate, and the translation vector t is the coordinate of the camera in the world coordinate system.

[0187] Substitute the known camera parameters, camera pose information, and target corner pixel coordinate information into the above formula for solution, and the world coordinate information of the target corner can be inversely solved.

[0188] S5: According to the coordinates of the target corners, obtain the motion trajectory of the target corners on the spatial axis and the vibration signal on the time axis.

[0189] Specifically, step S5 includes:

[0190] S51: Detect and process consecutive frames in the video image to obtain several trajectory points of the target corners.

[0191] S52: According to several trajectory points of the target corners, obtain the motion trajectory of the target corners on the spatial axis.

[0192] S53: Obtain the deviation between the actual position of the target corners and their ideal trajectory.

[0193] S54: According to the deviation between the actual position of the target corners and their ideal trajectory, decompose the vibration signal of the target corners on the time axis.

[0194] As Figure 3 shown, the vibration signal of the target on the time axis comes from decomposing the motion trajectory of the corners on the target on the spatial axis through a formula, and the decomposition formula is: .

[0195] Among them, ρ represents the deviation vector between the actual position and the ideal position of a certain corner on the target, and x and y respectively represent the offsets of this point in the horizontal and vertical directions of the trolley frame, that is, the vibration signals in the corresponding directions.

[0196] S6: According to the motion trajectory of the target corners on the spatial axis and the vibration signal on the time axis, determine whether the crane trolley frame is operating abnormally.

[0197] Specifically, step S6 includes:

[0198] S61: If the deviation between the actual position of the target corners and their ideal trajectory exceeds the preset offset threshold, the crane trolley frame has an abnormal operation.

[0199] At a certain position of the trolley frame, the ideal position of the target corner is the ideal world coordinate after ignoring external interference, and the actual position is its true world coordinate. After knowing the world coordinates of the corners on the target, the deviation distance between the actual position of the target corners on the current trolley frame and their ideal trajectory can be compared to determine whether this deviation exceeds the set offset threshold, so as to determine whether the trolley frame has an abnormal offset.

[0200] S62: Analyze the statistical characteristics in the time domain and frequency domain of the vibration signal of the target corner point over the time axis to determine whether the crane trolley frame is operating abnormally.

[0201] Based on the vibration signals of the target corner point in the x and y axis directions obtained in step S54, by analyzing the statistical characteristics in the time domain and frequency domain, it is possible to determine whether there are faults in the trolley frame components. Time domain analysis mainly extracts features such as the mean, variance, and kurtosis of the signal to characterize the fluctuation characteristics and trend changes of the vibration signal; frequency domain analysis can reveal the main frequency components of the signal and their energy distribution through power spectrum analysis.

[0202] For example, the server can use the power spectral density of the vibration signal to determine whether there are abnormalities in the components on the trolley frame. Taking the vibration signal in the horizontal direction as an example: For example:

[0203] (1) Window the signal using a Hamming window:

[0204] .

[0205] (2) Perform a fast Fourier transform on the windowed signal:

[0206] .

[0207] (3) Calculate the power spectral density of the signal:

[0208] ,

[0209] .

[0210] (4) Analyze the power spectral density of the signal to check if the power of a certain frequency component is abnormally high.

[0211] In the above formula, x[n] is the discrete vibration signal sequence in the x-axis direction, n is the sequence index, N is the total number of sequences, x i is the offset of the corner point in the horizontal direction at a certain moment, w[n] is the selected window function, and x w [n] is the windowed vibration signal sequence; X[k] is the complex frequency domain signal, k is the discrete frequency index, △f is the frequency resolution, fs is the total frequency, and PSD[k] is the power of the signal in the frequency domain.

[0212] If the power of the vibration signal is abnormally high at a certain frequency, it indicates that there may be a fault in the components on the trolley frame.

[0213] If the analysis result indicates that the trolley frame has an abnormal offset or there may be a fault in the trolley frame components, the server will determine that the status is abnormal, send an alarm signal, and notify the staff to handle it in a timely manner. If there is no abnormality and the trolley frame stops working, the system will stop detection to ensure the normal progress of the operation.

[0214] Figure 4 The overall processing flow of this method is shown as follows. In practical applications, the image data at the target on the crane trolley frame can be collected in real time for detection to determine whether there is an abnormal offset or fault in the trolley frame, facilitating timely handling.

[0215] The beneficial effect of this embodiment is that the object detection model in this method applies an improved YOLOv8 detection model. The Neck layer uses the BiFPN structure to replace the original FPN + PAN structure, achieving more efficient and accurate multi-scale feature fusion, thereby improving the detection accuracy of the model.

[0216] Embodiment Three

[0217] A crane trolley frame operation abnormal detection system, based on the crane trolley frame operation abnormal detection method as described in Embodiment One or Two, includes:

[0218] An installation module, used to install a target including corner points on the crane trolley frame;

[0219] An acquisition module, used to obtain the monitoring video image of the position of the crane trolley frame including the target;

[0220] A detection module, used to detect the monitoring video image based on the object detection model to obtain the target area;

[0221] A coordinate transformation module, used to obtain the coordinates of the target corner points based on the sub-pixel accuracy method and the coordinate transformation method according to the target area;

[0222] An analysis module, used to obtain the motion trajectory of the target corner points on the spatial axis and the vibration signal on the time axis according to the coordinates of the target corner points;

[0223] A judgment module, used to judge whether the crane trolley frame operates abnormally according to the motion trajectory of the target corner points on the spatial axis and the vibration signal on the time axis.

[0224] Embodiment Four

[0225] A computer program product includes a computer program, and when the computer program is executed, it implements the steps of the crane trolley frame operation abnormal detection method as described in Embodiment One or Two.

[0226] Embodiment Five

[0227] A readable storage medium stores a computer program as described in Embodiment 4, and when the computer program is executed, it implements the steps of the abnormal operation detection method for the crane trolley frame as described in Embodiment 1 or 2.

[0228] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for detecting abnormal operation of a crane trolley frame, characterized in that, It includes the following steps: Install a target including corner points on the crane trolley frame; Obtain a monitoring video image of the position of the crane trolley frame including the target; Detect the monitoring video image based on the target detection model to obtain the target area; Based on the target area, use the sub-pixel accuracy method and the coordinate transformation method to obtain the coordinates of the target corner points; According to the coordinates of the target corner points, obtain the motion trajectory of the target corner points on the spatial axis and the vibration signal on the time axis; According to the motion trajectory of the target corner points on the spatial axis and the vibration signal on the time axis, determine whether the crane trolley frame is operating abnormally; Among them, the obtaining the motion trajectory of the target corner points on the spatial axis and the vibration signal on the time axis according to the coordinates of the target corner points includes: Detect and process consecutive frames in the video image to obtain several trajectory points of the target corner points; According to several trajectory points of the target corner points, obtain the motion trajectory of the target corner points on the spatial axis; Obtain the deviation between the actual position of the target corner point and its ideal trajectory; According to the deviation between the actual position of the target corner point and its ideal trajectory, decompose the vibration signal of the target corner point on the time axis.

2. The abnormal operation detection method of the crane trolley frame according to claim 1, characterized in that, The target including corner points is a high-contrast double-sided checkerboard target.

3. The abnormal operation detection method of the crane trolley frame according to claim 1, characterized in that The detecting the monitoring video image based on the target detection model to obtain the target area includes: Use the YOLO, YOLOv8 or improved YOLOv8 target detection model to detect the monitoring video image and identify the target area.

4. The abnormal running detection method of the crane trolley frame according to claim 3, characterized in that, The improved YOLOv8 model includes: Backbone layer, which extracts features of the monitoring video image based on the convolutional module, C2f module, and SPPF module; Neck layer, which uses the features extracted by the Backbone layer and performs multi-scale feature fusion based on upsampling, feature splicing, convolutional module, C2f module, and BiFPN structure to obtain the fused features; Head layer, which detects the target area in the image according to the fused features.

5. The abnormal operation detection method of the crane trolley frame according to claim 1, characterized in that, The obtaining the coordinates of the target corner points based on the sub-pixel accuracy method and the coordinate transformation method according to the target area includes: Use the sub-pixel accuracy method to analyze the gray-scale change of pixels in the target area, and use the interpolation method to estimate the corner point position to obtain the pixel coordinates of the target corner points; Obtain the camera parameters and pose information of the monitoring camera; Use the coordinate transformation method to inversely solve the accurate world coordinates of the target corner points.

6. The abnormal operation detection method of the crane trolley frame according to claim 5, characterized in that, The using the sub-pixel accuracy method to analyze the gray-scale change of pixels in the target area and using the interpolation method to estimate the corner point position to obtain the pixel coordinates of the target corner points includes: Extract the rectangular area image at the position where the corner points are located from the target area; Generate a grayscale image of the rectangular area image; Use the sobar operator to calculate the gradient value of each point in the horizontal direction of the grayscale image; Select the point with the largest gradient value and determine its position of the single-pixel edge point in the rectangular area; Starting from the single-pixel edge point, select two adjacent points in the horizontal or vertical direction and fit them with a quadratic polynomial curve; Substitute the gradient values of the single-pixel edge point and its two adjacent points and the corresponding pixel coordinates into the quadratic polynomial curve equation for solution to obtain the accurate coordinates of the target point in the horizontal or vertical direction.

7. The abnormal operation detection method of the crane trolley frame according to claim 5, wherein, The method for inversely solving the accurate world coordinates of the target corner points by using the coordinate transformation method includes: Based on the inverse solution PnP method, using the camera parameters and pose information of the monitoring camera and the pixel coordinates of the target corner points, the accurate world coordinates of the target corner points are inversely solved.

8. The abnormal running detection method of the crane trolley frame according to claim 1, characterized in that, The method for judging whether the crane trolley frame operates abnormally according to the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis includes: If the deviation between the actual position of the target corner point and its ideal trajectory exceeds the preset offset threshold, the crane trolley frame has an abnormal operation; Perform time-domain and frequency-domain statistical feature analysis on the vibration signal of the target corner point on the time axis to judge whether the crane trolley frame operates abnormally.

9. An abnormal operation detection system for a crane trolley frame, characterized in that, Based on the method for detecting abnormal operation of the crane trolley frame according to any one of claims 1 to 8, it includes: An installation module for installing a target including corner points on the crane trolley frame; An acquisition module for acquiring a monitoring video image of the position of the crane trolley frame including the target; A detection module for detecting the monitoring video image based on the target detection model to obtain the target area; A coordinate transformation module for obtaining the coordinates of the target corner points according to the target area based on the sub-pixel accuracy method and the coordinate transformation method; An analysis module for obtaining the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis according to the coordinates of the target corner point; A judgment module for judging whether the crane trolley frame operates abnormally according to the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis.

Images