Method and system for detecting abnormal operation of crane trolley frame

Abstract

The invention discloses a method and a system for detecting abnormal operation of a crane trolley frame. The method comprises the following steps: mounting a target on the crane trolley frame; acquiring a monitoring video image; detecting the monitoring video image based on a target detection model to obtain a target area; based on a sub-pixel precision method and a coordinate transformation method, obtaining coordinates of the target angular point; acquiring a motion track of the target angular point on a space axis and a vibration signal of the target angular point on a time axis; and judging whether the crane trolley frame runs abnormally or not according to the motion trail of the target angular point on the space axis and the vibration signal on the time axis. According to the technical scheme provided by the invention, real-time monitoring and accurate detection of the running state of the trolley frame are realized, the accuracy of anomaly detection can be remarkably improved, the running state of the trolley frame can be comprehensively evaluated and perceived in real time, and anomaly early warning is carried out in time, so that stable running of equipment is guaranteed to the greatest extent, and the working efficiency is improved. And production interruption and safety risks caused by sudden faults are avoided.

Description

Technical Field

[0001] The present invention relates to the field of crane detection, and in particular to a method and system for detecting abnormal operation of a crane trolley frame. Background Art

[0002] As the throughput of global ports continues to grow, the working intensity and load pressure of cranes are also increasing. As one of the core components of the crane, the stability of the working state of the trolley becomes particularly important. However, the maintenance of the crane trolley frame still relies on the traditional passive shutdown maintenance. This maintenance method is inefficient, unable to predict the potential risks of the equipment in advance, and unable to obtain the operating status and health of the equipment in real time. This not only affects the continuity of port operations, but also may bring safety hazards. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the current technology, the present invention provides a method for detecting abnormal operation of a crane trolley frame, by installing a target on the crane trolley frame, detecting the target image using a target detection model, determining the approximate position of the target corner point, and obtaining the coordinates of the target corner point using a sub-pixel precision method and a coordinate transformation method. By analyzing the motion trajectory and vibration signal of the target corner point, it is determined whether the trolley frame has abnormal deviation or fault.

[0004] To achieve the above object, the embodiments of the present invention adopt the following technical solutions: A method for detecting abnormal operation of a crane trolley frame comprises the following steps: Installing a target including corner points on a crane trolley frame; Acquire a surveillance video image of the crane trolley frame position including the target; Detect surveillance video images based on the target detection model to obtain the target area; According to the target area, the coordinates of the target corner points are obtained based on the sub-pixel precision method and coordinate transformation method; According to the coordinates of the target corner point, the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis are obtained; Based on the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis, it is determined whether the crane trolley frame is operating abnormally.

[0005] According to one aspect of the present invention, the target including the corner points is a high-contrast double-sided checkerboard target.

[0006] According to one aspect of the present invention, the detecting of the surveillance video image based on the target detection model to obtain the target area includes: Use YOLO, YOLOv8, or improved YOLOv8 target detection models to detect surveillance video images and identify target areas.

[0007] According to one aspect of the present invention, the improved YOLOv8 model includes: Backbone layer, based on convolution module, C2f module and SPPF module, performs surveillance video image feature extraction; The Neck layer uses the features extracted by the Backbone layer to perform multi-scale feature fusion based on upsampling, feature concatenation, convolution module, C2f module, and BiFPN structure to obtain fused features; The head layer detects the target area in the image based on the fusion features.

[0008] According to one aspect of the present invention, obtaining the coordinates of the target corner points based on the target area, based on the sub-pixel precision method and the coordinate transformation method includes: The grayscale changes of pixels in the target area are analyzed using the sub-pixel precision method, and the corner point positions are estimated using the interpolation method to obtain the pixel coordinates of the target corner points; Get the camera parameters and posture information of the surveillance camera; The coordinate transformation method is used to inversely solve the precise world coordinates of the target corner points.

[0009] According to one aspect of the present invention, the method of analyzing the grayscale change of pixels in the target area by using a sub-pixel precision method and estimating the corner point position by using an interpolation method to obtain the pixel coordinates of the target corner point includes: Extracting a rectangular area image where the corner points are located from the target area; Generate a grayscale image of a rectangular area image; The sobar operator is used to calculate the gradient value of each point in the horizontal direction of the grayscale image; Select the point with the largest gradient value and determine its single-pixel edge point position within the rectangular area; Starting from a single pixel edge point, two adjacent points in the horizontal or vertical direction are selected and fitted using a quadratic polynomial curve; Substitute the gradient values ​​of a single pixel edge point and its two adjacent points and the corresponding pixel coordinates into the quadratic polynomial curve equation for solution to obtain the precise coordinates of the target point in the horizontal or vertical direction.

[0010] According to one aspect of the present invention, the method of using a coordinate transformation method to inversely resolve the precise world coordinates of the target corner points includes: Based on the inverse PnP method, the precise world coordinates of the target corner points are inversely solved using the camera parameters and posture information of the surveillance camera and the pixel coordinates of the target corner points.

[0011] According to one aspect of the present invention, obtaining the motion trajectory of the target corner point on the space axis and the vibration signal on the time axis according to the coordinates of the target corner point includes: Detect and process the continuous frames in the video image to obtain several trajectory points of the target corner points; According to a number of trajectory points of the target corner point, the motion trajectory of the target corner point on the spatial axis is obtained; Obtain the deviation between the actual position of the target corner point and its ideal trajectory; According to the deviation between the actual position of the target corner point and its ideal trajectory, the vibration signal of the target corner point on the time axis is decomposed.

[0012] According to one aspect of the present invention, judging whether the crane trolley frame is operating abnormally according to the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis includes: If the deviation between the actual position of the target corner point and its ideal trajectory exceeds the preset offset threshold, the crane trolley frame will operate abnormally; Perform time domain and frequency domain statistical feature analysis on the vibration signal of the target corner point on the time axis to determine whether the crane trolley frame is operating abnormally.

[0013] A crane trolley frame operation abnormality detection system, based on the crane trolley frame operation abnormality detection method as described above, comprises: A mounting module for mounting the target including corner points on a crane trolley frame; An acquisition module, used to acquire a monitoring video image of the position of the crane trolley frame including the target; A detection module is used to detect the surveillance video image based on the target detection model and obtain the target area; A coordinate transformation module is used to obtain the coordinates of the target corner points according to the target area based on the sub-pixel precision method and the coordinate transformation method; The analysis module is used to obtain the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis according to the coordinates of the target corner point; The judgment module is used to judge whether the crane trolley frame is operating abnormally according to the motion trajectory of the target corner point on the space axis and the vibration signal on the time axis.

[0014] Advantages of the present invention: The present invention provides a method for detecting abnormal operation of a crane trolley frame. A target is installed on the crane trolley frame, a target detection model is used to detect the target image, the approximate position of the target corner point is determined, the coordinates of the target corner point are obtained using a sub-pixel precision method and a coordinate transformation method, and the motion trajectory and vibration signal of the target corner point are analyzed to determine whether the trolley frame has abnormal deviation or fault.

[0015] This method constructs a multi-dimensional status assessment system, which realizes real-time monitoring and accurate detection of the operating status of the trolley frame. It can not only significantly improve the accuracy of abnormal detection, but also comprehensively evaluate and perceive the operating status of the trolley frame in real time, and issue abnormal warnings in time, thereby maximizing the stable operation of the equipment and avoiding production interruptions and safety risks caused by sudden failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of a method for detecting abnormal operation of a crane trolley frame according to the present invention; Figure 2 This is a schematic diagram of the crane structure of the present invention; Figure 3 This is a schematic diagram of target detection according to the present invention; Figure 4 It is an overall processing flow chart of a method for detecting abnormal operation of a crane trolley frame according to the present invention; Figure 5 This is the improved YOLOv8 model architecture described in Example 2 of the present invention.

[0018] Figure numerals: 1. front beam of crane; 2. high-definition camera; 3. trolley frame; 4. double-sided target; 5. PLC; 6. server; 7. electrical room; 8. rear beam of crane. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Embodiment 1 like Figure 1 As shown, a method for detecting abnormal operation of a crane trolley frame comprises the following steps: S1: Install the target including the corner points on the crane trolley frame.

[0021] In practical applications, such as Figure 2As shown, high-definition cameras are deployed on the front and rear beams of the crane. The high-definition cameras are high-frame rate cameras and are installed at a horizontal viewing angle to cover the moving area of ​​the trolley frame. Double-sided targets including corner points are installed on the crane trolley frame. The targets are high-contrast checkerboard targets.

[0022] The high-definition cameras on the front and rear beams can collect images of the double-sided targets on the trolley frame in real time from the front and rear ends, which can effectively avoid the loss of the double-sided targets caused by the change of the position of the trolley frame. In addition, the high-definition cameras have high-frame rate video acquisition capabilities, which provides data support for the subsequent acquisition of vibration signals of the target corners.

[0023] The crane electrical room is equipped with a PLC and a server, which are connected to the HD camera and PLC. The server and the HD camera are connected via optical fiber or 5G communication to transmit image data in real time.

[0024] S2: Acquire a monitoring video image of the crane trolley frame position including the target.

[0025] The high-definition cameras deployed on the front and rear beams monitor the working status of the trolley frame in real time. By acquiring images of the trolley frame from the front and rear ends, the target loss caused by equipment limitations can be avoided, thus ensuring a clear picture. At the same time, the video data collected by the high-definition camera will be transmitted to the server, and the PLC will send the signal to control the movement of the trolley frame to the server.

[0026] The server in the electrical room receives data from the PLC. The PLC signal contains information about the distance the trolley has moved, so the server can obtain the current location of the trolley. Based on the current location, the server then adjusts the focal length of the high-definition camera to focus and ensure that a clear double-sided target video is captured.

[0027] S3: Detect the surveillance video image based on the target detection model to obtain the target area.

[0028] Use the pre-trained object detection model to quickly locate the approximate position of the corner points on the two-sided target in the image and mark the area.

[0029] In practical applications, target detection models such as YOLO, YOLOv8, other improved versions of YOLO, SSD, and Faster R-CNN can be used to detect and identify surveillance video images and frame the target area.

[0030] For example, the YOLOv8 model can be used for detection. Specifically, the YOLOv8 model architecture includes: (1) Backbone layer: extracts surveillance video image features based on convolution module, C2f module and SPPF module.

[0031] (2) The Neck layer uses the features extracted by the Backbone layer to perform multi-scale feature fusion based on upsampling, feature concatenation, convolution module, C2f module, and FPN+PAN structure to obtain fused features. The Neck layer adopts the FPN+PAN structure, where FPN stands for Feature Pyramid Network and PAN stands for Path Aggregation Network. Through bidirectional feature transmission, it achieves full fusion of feature maps of different scales, retaining high-level semantic information and utilizing low-level positioning information, thereby significantly improving the performance of target detection.

[0032] (3) Head layer: detects the target area in the positioning image based on the fused features.

[0033] The training of the YOLOv8 model includes: (1) Collect videos recorded during the operation of the small vehicle frame equipped with targets, and calibrate the corner points of the double-sided targets in the videos to generate a training database.

[0034] (2) The training database is divided into a training set and a test set in a ratio of 8:2. The YOLOv8 model is trained using the training set to obtain the object detection model, and its performance is evaluated using the test set.

[0035] (3) By adjusting the model’s hyperparameters, the target detection model with the best detection effect is selected as the final version used in the actual work scenario.

[0036] S4: According to the target area, the coordinates of the target corner points are obtained based on the sub-pixel precision method and the coordinate transformation method.

[0037] Specifically, step S4 includes: S41: Analyze the grayscale changes of pixels in the target area using a sub-pixel precision method, and estimate the corner point positions using an interpolation method to obtain the pixel coordinates of the target corner points.

[0038] Step S41 includes: S411: Extracting a rectangular area image where corner points are located from the target area.

[0039] In practical applications, a rectangular area of ​​5×5 pixels can be extracted.

[0040] S412: Generate a grayscale image of the rectangular area image.

[0041] S413: Calculate the gradient value of each point in the grayscale image in the horizontal direction using the sobar operator.

[0042] The sobar operator is: .

[0043] S414: Select the point with the largest gradient value and determine its single-pixel edge point position within the rectangular area.

[0044] S415: Starting from the single-pixel edge point, two adjacent points in the horizontal or vertical direction are selected and fitted using a quadratic polynomial curve.

[0045] Starting from the single pixel edge point X, select two adjacent points in the horizontal direction and On this basis, the quadratic polynomial curve Perform the fitting.

[0046] S416: Substitute the gradient values ​​of the single-pixel edge point and its two adjacent points and the corresponding pixel coordinates into the quadratic polynomial curve equation for solution to obtain the precise coordinates of the target point in the horizontal or vertical direction.

[0047] Substitute the gradient values ​​and corresponding pixel coordinates of the above three points into the equation to solve, so as to determine the precise horizontal coordinates x of the target point i : .

[0048] Similarly, the precise vertical coordinate y of the target can be determined i .

[0049] Among them, (x i ,y i ) represents the position of the target point, i is the i-th pixel point, x is the horizontal coordinate of the edge pixel point, and f(x) is the gradient value corresponding to the horizontal position of the pixel point.

[0050] S42: Obtain camera parameters and posture information of the surveillance camera.

[0051] S43: Use the coordinate transformation method to inversely solve the precise world coordinates of the target corner points.

[0052] Step S43 includes: Based on the inverse PnP method, the precise world coordinates of the target corner points are inversely solved using the camera parameters and posture information of the surveillance camera and the pixel coordinates of the target corner points.

[0053] The PnP (Perspective-n-Point) method is a method for solving the motion of 3D to 2D point pairs. That is, the coordinates of n 3D points in the world coordinate system and the coordinates of these points in the image (or pixel) coordinate system are known. At the same time, the intrinsic parameter matrix of the camera is known, and the position and posture of the camera coordinate system relative to the world coordinate system are solved. The position and posture here usually include the rotation matrix R and the translation vector t. Conversely, if the camera parameters, camera attitude information, and camera coordinate system information are known, the world coordinate system information can also be inversely solved.

[0054] The specific formula of the PnP method is: , , ,

[0055] ,

[0056] .

[0057] Where s is the scaling factor or the depth of the target point, that is, the Z coordinate of the 3D point in the camera coordinate system; (u i , v i , 1) is the homogeneous coordinate of the target point in the pixel coordinate system; (Xc, Yc, Zc) is the coordinate of the target point in the camera coordinate system; (Xc, Yc, Zc) is the homogeneous coordinate of the target point in the pixel coordinate system; i , Y i , Z) is the coordinate of the target point in the world coordinate system, Z is the current position of the small frame; K is the internal parameter matrix of the camera, fx, fy are the parameters of the camera focal length, c x 、c y is the principal point offset of the camera; the external parameter matrix of the camera describes the position of the camera in the world coordinate system and has two parameters, the rotation matrix R and the translation vector t, which correspond to the rotation and translation of the camera in the world coordinate system respectively; r in the rotation matrix R *1 、r *2 、r *3 They respectively represent the directions of the x, y, and z axes of the pixel coordinate system in the world coordinate system, and the translation vector t is the coordinate of the camera in the world coordinate system.

[0058] Substituting the known camera parameters, camera posture information and target corner pixel coordinate information into the above formula for solving, the world coordinate information of the target corner point can be inversely solved.

[0059] S5: According to the coordinates of the target corner point, the motion trajectory of the target corner point on the space axis and the vibration signal on the time axis are obtained.

[0060] Specifically, step S5 includes: S51: Detect and process the continuous frames in the video image to obtain several trajectory points of the target corner points.

[0061] S52: Obtain the motion trajectory of the target corner point on the spatial axis according to a plurality of trajectory points of the target corner point.

[0062] S53: Obtain the deviation between the actual position of the target corner point and its ideal trajectory.

[0063] S54: Decomposing the vibration signal of the target corner point on the time axis according to the deviation between the actual position of the target corner point and its ideal trajectory.

[0064] like Figure 3 As shown in the figure, the vibration signal of the target on the time axis comes from the motion trajectory of the corner point on the target on the space axis decomposed by the formula. The decomposition formula is: .

[0065] Among them, ρ represents the deviation vector between the actual position of a corner point on the target and the ideal position, and x and y represent the offset of the point in the horizontal and vertical directions of the trolley frame, that is, the vibration signal in the corresponding direction.

[0066] S6: Determine whether the crane trolley frame is operating abnormally based on the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis.

[0067] Specifically, step S6 includes: S61: If the deviation between the actual position of the target corner point and its ideal trajectory exceeds the preset offset threshold, the crane trolley frame operates abnormally.

[0068] At a certain position of the trolley frame, the ideal position of the target corner point is the ideal world coordinate after ignoring external interference, while the actual position is its real world coordinate. After the world coordinate of the corner point on the target is known, the deviation distance between the actual position of the target corner point on the current trolley frame and its ideal trajectory can be compared to determine whether the deviation exceeds the set offset threshold, thereby determining whether the trolley frame has an abnormal offset.

[0069] S62: Perform time domain and frequency domain statistical feature analysis on the vibration signal of the target corner point on the time axis to determine whether the crane trolley frame is operating abnormally.

[0070] According to the vibration signal of the target corner point in the x and y axis directions obtained in step S54, by analyzing the statistical characteristics in the time domain and frequency domain, it can be determined whether the trolley frame device has a fault. The time domain analysis mainly extracts the mean, variance, kurtosis and other characteristics of the signal to characterize the fluctuation characteristics and trend changes of the vibration signal; the frequency domain analysis can reveal the main frequency components of the signal and its energy distribution through power spectrum analysis.

[0071] For example, the server can use the power spectrum density of the vibration signal to determine whether there is an abnormality in the components on the trolley frame. For example: (1) Use the Hamming window to window the signal: .

[0072] (2) Perform fast Fourier transform on the windowed signal: .

[0073] (3) Calculate the power spectral density of the signal: ,

[0074] .

[0075] (4) Analyze the power spectral density of the signal to see whether the power of a certain frequency component is abnormally high.

[0076] In the above formula, x[n] is the discrete vibration signal sequence in the x-axis direction, n is the sequence index, N is the total number of sequences, and x i is the horizontal offset of the corner point at a certain moment, w[n] is the selected window function, x w [n] is the windowed vibration signal sequence; X[k] is the complex frequency domain signal, k is the discrete frequency index, △f is the frequency resolution, fs is the total frequency, and PSD[k] is the power of the signal in the frequency domain.

[0077] If the power of the vibration signal at a certain frequency is abnormally high, it indicates that there may be a fault in the device on the trolley frame.

[0078] If the analysis results show that the trolley frame has an abnormal deviation or a trolley frame component may be faulty, the server will determine that the status is abnormal and send an alarm signal to notify the staff to handle it in time. If there is no abnormality and the trolley frame stops working, the system will stop detection to ensure the normal operation.

[0079] Figure 4 The overall processing flow of the method is shown. In practical applications, the image data of the target on the crane trolley frame can be collected in real time for detection to determine whether the trolley frame has abnormal deviation or failure, so as to facilitate timely processing.

[0080] The beneficial effects of this embodiment are as follows: this method installs a target on a crane trolley frame, detects the target image using a target detection model, determines the approximate position of the target corner point, obtains the coordinates of the target corner point using a sub-pixel precision method and a coordinate transformation method, and determines whether the trolley frame has abnormal offset or fault by analyzing the motion trajectory and vibration signal of the target corner point.

[0081] This method constructs a multi-dimensional status assessment system, realizes real-time monitoring and accurate detection of the trolley frame's operating status, and can not only significantly improve the accuracy of abnormal detection, but also comprehensively evaluate and perceive the trolley frame's operating status in real time, and issue abnormal warnings in a timely manner, thereby maximizing the guarantee of stable operation of the equipment and avoiding production interruptions and safety risks caused by sudden failures. This method solves the problems of low detection accuracy and lack of real-time performance in traditional methods, and can effectively improve detection efficiency and reliability, providing a strong guarantee for the safe and stable operation of crane equipment, and providing strong technical support for the safety and reliability of port operations.

[0082] Embodiment 2 like Figure 1 As shown, a method for detecting abnormal operation of a crane trolley frame comprises the following steps: S1: Install the target including the corner points on the crane trolley frame.

[0083] In practical applications, such as Figure 2 As shown, high-definition cameras are deployed on the front and rear beams of the crane. The high-definition cameras are high-frame rate cameras and are installed at a horizontal viewing angle to cover the moving area of ​​the trolley frame. Double-sided targets including corner points are installed on the crane trolley frame. The targets are high-contrast checkerboard targets.

[0084] The high-definition cameras on the front and rear beams can collect images of the double-sided targets on the trolley frame in real time from the front and rear ends, which can effectively avoid the loss of the double-sided targets caused by the change of the position of the trolley frame. In addition, the high-definition cameras have high-frame rate video acquisition capabilities, which provides data support for the subsequent acquisition of vibration signals of the target corners.

[0085] The crane electrical room is equipped with a PLC and a server, which are connected to the HD camera and PLC. The server and the HD camera are connected via optical fiber or 5G communication to transmit image data in real time.

[0086] S2: Acquire a monitoring video image of the crane trolley frame position including the target.

[0087] The high-definition cameras deployed on the front and rear beams monitor the working status of the trolley frame in real time. By acquiring images of the trolley frame from the front and rear ends, the target loss caused by equipment limitations can be avoided, thus ensuring a clear picture. At the same time, the video data collected by the high-definition camera will be transmitted to the server, and the PLC will send the signal to control the movement of the trolley frame to the server.

[0088] The server in the electrical room receives data from the PLC. The PLC signal contains information about the distance the trolley has moved, so the server can obtain the current location of the trolley. Based on the current location, the server then adjusts the focal length of the high-definition camera to focus and ensure that a clear double-sided target video is captured.

[0089] S3: Detect the surveillance video image based on the target detection model to obtain the target area.

[0090] Use the pre-trained object detection model to quickly locate the approximate position of the corner points on the two-sided target in the image and mark the area.

[0091] In practical applications, target detection models such as YOLO, YOLOv8, other improved versions of YOLO, SSD, and Faster R-CNN can be used to detect and identify surveillance video images and frame the target area.

[0092] For example, an improved YOLOv8 model may be used for detection. Specifically, the improved YOLOv8 model architecture includes: (1) Backbone layer: extracts surveillance video image features based on convolution module, C2f module and SPPF module.

[0093] (2) The Neck layer uses the features extracted by the Backbone layer to perform multi-scale feature fusion based on upsampling, feature concatenation, convolution module, C2f module, and BiFPN structure to obtain fused features.

[0094] The original YOLOv8 model Neck layer uses FPN (Feature Pyramid Network) + PAN (Path Aggregation Network) structure. In this method, Figure 5 As shown in the figure, the Bidirectional Feature Pyramid Network (BiFPN) structure is used to replace the original FPN+PAN structure. BiFPN includes efficient bidirectional cross-scale connections, simplified network structure, weighted feature fusion and bidirectional feature fusion, which can achieve more efficient and accurate multi-scale feature fusion, thereby improving the detection accuracy of the model.

[0095] (3) Head layer: detects the target area in the positioning image based on the fused features.

[0096] The training of the improved YOLOv8 model includes: (1) Collect the videos recorded during the operation of the trolley frame equipped with the target, and calibrate the corner positions of the double-sided target in the videos to generate a training database.

[0097] (2) Divide the training database into a training set and a test set at a ratio of 8:2. Use the training set to train the improved YOLOv8 model to obtain an object detection model, and evaluate its performance through the test set.

[0098] (3) By adjusting the hyperparameters of the model, screen out the object detection model with the best detection effect as the final version used in the actual working scenario.

[0099] S4: Based on the target area, use the sub-pixel accuracy method and the coordinate transformation method to obtain the coordinates of the target corner points.

[0100] Specifically, step S4 includes: S41: Use the sub-pixel accuracy method to analyze the gray-scale change of the pixels in the target area, and use the interpolation method to estimate the corner position to obtain the pixel coordinates of the target corner points.

[0101] Step S41 includes: S411: Extract the rectangular area image at the position of the corner point from the target area.

[0102] In practical applications, a rectangular area of 5×5 pixels can be extracted.

[0103] S412: Generate the gray-scale image of the rectangular area image.

[0104] S413: Use the sobar operator to calculate the gradient value of each point in the horizontal direction of the gray-scale image.

[0105] The sobar operator is: .

[0106] S414: Select the point with the maximum gradient value and determine its position as a single-pixel edge point within the rectangular area.

[0107] S415: Starting from the single-pixel edge point, select two adjacent points in the horizontal or vertical direction and fit them using a quadratic polynomial curve.

[0108] Starting from this single-pixel edge point X, select two adjacent points in the horizontal direction and , and on this basis, use the quadratic polynomial curve for fitting.

[0109] S416: Substitute the gradient values and corresponding pixel coordinates of the single-pixel edge point and its two adjacent points into the quadratic polynomial curve equation for solution to obtain the precise coordinates of the target point in the horizontal or vertical direction.

[0110] Substitute the gradient values ​​and corresponding pixel coordinates of the above three points into the equation to solve it, so as to determine the precise horizontal coordinates of the target point. : .

[0111] Similarly, the precise vertical coordinates of the target can be determined. .

[0112] Among them, (x i ,y i ) represents the position of the target point, i is the i-th pixel point, x is the horizontal coordinate of the edge pixel point, and f(x) is the gradient value corresponding to the horizontal position of the pixel point.

[0113] S42: Obtain camera parameters and posture information of the surveillance camera.

[0114] S43: Use the coordinate transformation method to inversely solve the precise world coordinates of the target corner points.

[0115] Step S43 includes: Based on the inverse PnP method, the precise world coordinates of the target corner points are inversely solved using the camera parameters and posture information of the surveillance camera and the pixel coordinates of the target corner points.

[0116] The PnP (Perspective-n-Point) method is a method for solving the motion of 3D to 2D point pairs. That is, the coordinates of n 3D points in the world coordinate system and the coordinates of these points in the image (or pixel) coordinate system are known. At the same time, the intrinsic parameter matrix of the camera is known, and the position and posture of the camera coordinate system relative to the world coordinate system are solved. The position and posture here usually include the rotation matrix R and the translation vector t. Conversely, if the camera parameters, camera attitude information, and camera coordinate system information are known, the world coordinate system information can also be inversely solved.

[0117] The specific formula of the PnP method is: ,

[0118] ,

[0119] , ,

[0120] .

[0121] Where s is the scaling factor or the depth of the target point, that is, the Z coordinate of the 3D point in the camera coordinate system; (u i , v i, 1) is the homogeneous coordinate of the target point in the pixel coordinate system; (Xc, Yc, Zc) is the coordinate of the target point in the camera coordinate system; (Xc, Yc, Zc) is the homogeneous coordinate of the target point in the pixel coordinate system; i , Y i , Z) is the coordinate of the target point in the world coordinate system, Z is the current position of the small frame; K is the internal parameter matrix of the camera, fx, fy are the parameters of the camera focal length, c x 、c y is the principal point offset of the camera; the external parameter matrix of the camera describes the position of the camera in the world coordinate system and has two parameters, the rotation matrix R and the translation vector t, which correspond to the rotation and translation of the camera in the world coordinate system respectively; r in the rotation matrix R *1 、r *2 、r *3 They respectively represent the directions of the x, y, and z axes of the pixel coordinate system in the world coordinate system, and the translation vector t is the coordinate of the camera in the world coordinate system.

[0122] Substituting the known camera parameters, camera posture information and target corner pixel coordinate information into the above formula for solving, the world coordinate information of the target corner point can be inversely solved.

[0123] S5: According to the coordinates of the target corner point, the motion trajectory of the target corner point on the space axis and the vibration signal on the time axis are obtained.

[0124] Specifically, step S5 includes: S51: Detect and process the continuous frames in the video image to obtain several trajectory points of the target corner points.

[0125] S52: Obtain the motion trajectory of the target corner point on the spatial axis according to a plurality of trajectory points of the target corner point.

[0126] S53: Obtain the deviation between the actual position of the target corner point and its ideal trajectory.

[0127] S54: Decomposing the vibration signal of the target corner point on the time axis according to the deviation between the actual position of the target corner point and its ideal trajectory.

[0128] like Figure 3 As shown in the figure, the vibration signal of the target on the time axis comes from the motion trajectory of the corner point on the target on the space axis decomposed by the formula. The decomposition formula is: .

[0129] Among them, ρ represents the deviation vector between the actual position of a corner point on the target and the ideal position, and x and y represent the offset of the point in the horizontal and vertical directions of the trolley frame, that is, the vibration signal in the corresponding direction.

[0130] S6: Determine whether the crane trolley frame is operating abnormally based on the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis.

[0131] Specifically, step S6 includes: S61: If the deviation between the actual position of the target corner point and its ideal trajectory exceeds the preset offset threshold, the crane trolley frame operates abnormally.

[0132] At a certain position of the trolley frame, the ideal position of the target corner point is the ideal world coordinate after ignoring external interference, while the actual position is its real world coordinate. After the world coordinate of the corner point on the target is known, the deviation distance between the actual position of the target corner point on the current trolley frame and its ideal trajectory can be compared to determine whether the deviation exceeds the set offset threshold, thereby determining whether the trolley frame has an abnormal offset.

[0133] S62: Perform time domain and frequency domain statistical feature analysis on the vibration signal of the target corner point on the time axis to determine whether the crane trolley frame is operating abnormally.

[0134] According to the vibration signal of the target corner point in the x and y axis directions obtained in step S54, by analyzing the statistical characteristics in the time domain and frequency domain, it can be determined whether the trolley frame device has a fault. The time domain analysis mainly extracts the mean, variance, kurtosis and other characteristics of the signal to characterize the fluctuation characteristics and trend changes of the vibration signal; the frequency domain analysis can reveal the main frequency components of the signal and its energy distribution through power spectrum analysis.

[0135] For example, the server can use the power spectrum density of the vibration signal to determine whether there is an abnormality in the components on the trolley frame. For example: (1) Use the Hamming window to window the signal: .

[0136] (2) Perform fast Fourier transform on the windowed signal: .

[0137] (3) Calculate the power spectral density of the signal: ,

[0138] .

[0139] (4) Analyze the power spectral density of the signal to see whether the power of a certain frequency component is abnormally high.

[0140] In the above formula, x[n] is the discrete vibration signal sequence in the x-axis direction, n is the sequence index, N is the total number of sequences, and x iis the offset of the corner point in the horizontal direction at a certain moment, w[n] is the selected window function, and x w [n] is the vibration signal sequence after windowing; X[k] is the complex frequency domain signal, k is the discrete frequency index, △f is the frequency resolution, fs is the total frequency, and PSD[k] is the power of the signal in the frequency domain.

[0141] If the power of the vibration signal is abnormally high at a certain frequency, it indicates that there may be a fault in the device on the trolley frame.

[0142] If the analysis result shows that the trolley frame has abnormal offset or there may be a fault in the trolley frame components, the server will determine that the status is abnormal and send an alarm signal to notify the staff to handle it in time. If there is no abnormality and the trolley frame stops working, the system will stop detection to ensure the normal progress of the operation.

[0143] Figure 4 The overall processing flow of this method is shown. In practical applications, the image data at the target point on the crane trolley frame can be collected in real time for detection to determine whether there is abnormal offset or fault in the trolley frame, so as to facilitate timely processing.

[0144] The beneficial effect of this embodiment is that the object detection model in this method applies an improved YOLOv8 detection model. The Neck layer uses the BiFPN structure to replace the original FPN+PAN structure to achieve more efficient and accurate multi-scale feature fusion, thereby improving the detection accuracy of the model.

[0145] Embodiment Three A crane trolley frame operation abnormality detection system, based on the crane trolley frame operation abnormality detection method described in Embodiment One or Two, includes: An installation module for installing a target including corner points on the crane trolley frame; A collection module for obtaining the monitoring video image of the position of the crane trolley frame including the target; A detection module for detecting the monitoring video image based on the object detection model to obtain the target area; A coordinate transformation module for obtaining the coordinates of the target corner points based on the sub-pixel accuracy method and the coordinate transformation method according to the target area; An analysis module for obtaining the motion trajectory of the target corner points on the spatial axis and the vibration signal on the time axis according to the coordinates of the target corner points; A judgment module for judging whether the crane trolley frame operates abnormally according to the motion trajectory of the target corner points on the spatial axis and the vibration signal on the time axis.

[0146] Embodiment Four A computer program product comprises a computer program, wherein when the computer program is executed, the steps of the method for detecting abnormal operation of a crane trolley frame as described in Embodiment 1 or 2 are implemented.

[0147] Embodiment 5 A readable storage medium stores a computer program as described in Example 4, and when the computer program is executed, the steps of the crane trolley frame operation abnormality detection method as described in Example 1 or 2 are implemented.

[0148] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for detecting abnormal operation of a crane trolley frame, characterized in that: The following steps are involved: Installing a target including corner points on a crane trolley frame; Acquire a surveillance video image of the crane trolley frame position including the target; Detect surveillance video images based on the target detection model to obtain the target area; According to the target area, the coordinates of the target corner points are obtained based on the sub-pixel precision method and coordinate transformation method; According to the coordinates of the target corner point, the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis are obtained; Based on the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis, it is determined whether the crane trolley frame is operating abnormally.

2. The method for detecting abnormal operation of a crane trolley frame according to claim 1, characterized in that: The target including the corner points is a high-contrast double-sided checkerboard target.

3. The method for detecting abnormal operation of a crane trolley frame according to claim 1, characterized in that: The detecting of the surveillance video image based on the target detection model to obtain the target area includes: Use YOLO, YOLOv8, or improved YOLOv8 target detection models to detect surveillance video images and identify target areas.

4. The method for detecting abnormal operation of a crane trolley frame according to claim 3 is characterized in that: The improved YOLOv8 model includes: Backbone layer, based on convolution module, C2f module and SPPF module, performs surveillance video image feature extraction; The Neck layer uses the features extracted by the Backbone layer to perform multi-scale feature fusion based on upsampling, feature concatenation, convolution module, C2f module, and BiFPN structure to obtain fused features; The head layer detects the target area in the image based on the fusion features.

5. The method for detecting abnormal operation of a crane trolley frame according to claim 1, characterized in that: The method of obtaining the coordinates of the target corner points according to the target area based on the sub-pixel precision method and the coordinate transformation method includes: The sub-pixel precision method is used to analyze the grayscale changes of pixels in the target area, and the interpolation method is used to estimate the corner point position to obtain the pixel coordinates of the target corner point; Get the camera parameters and posture information of the surveillance camera; The coordinate transformation method is used to inversely solve the precise world coordinates of the target corner points.

6. The method for detecting abnormal operation of a crane trolley frame according to claim 5, characterized in that: The method of analyzing the grayscale changes of pixels in the target area by using the sub-pixel precision method and estimating the corner point positions by using the interpolation method to obtain the pixel coordinates of the target corner points includes: Extracting a rectangular area image where the corner points are located from the target area; Generate a grayscale image of a rectangular area image; The sobar operator is used to calculate the gradient value of each point in the horizontal direction of the grayscale image; Select the point with the largest gradient value and determine its single-pixel edge point position within the rectangular area; Starting from a single pixel edge point, two adjacent points in the horizontal or vertical direction are selected and fitted using a quadratic polynomial curve; Substitute the gradient values ​​of a single pixel edge point and its two adjacent points and the corresponding pixel coordinates into the quadratic polynomial curve equation for solution to obtain the precise coordinates of the target point in the horizontal or vertical direction.

7. The method for detecting abnormal operation of a crane trolley frame according to claim 5, characterized in that: The method of using the coordinate transformation method to inversely solve the precise world coordinates of the target corner points includes: Based on the inverse PnP method, the precise world coordinates of the target corner points are inversely solved using the camera parameters and posture information of the surveillance camera and the pixel coordinates of the target corner points.

8. The method for detecting abnormal operation of a crane trolley frame according to claim 1, characterized in that: The step of obtaining the motion trajectory of the target corner point on the space axis and the vibration signal on the time axis according to the coordinates of the target corner point comprises: Detect and process the continuous frames in the video image to obtain several trajectory points of the target corner points; According to a number of trajectory points of the target corner point, the motion trajectory of the target corner point on the spatial axis is obtained; Obtain the deviation between the actual position of the target corner point and its ideal trajectory; According to the deviation between the actual position of the target corner point and its ideal trajectory, the vibration signal of the target corner point on the time axis is decomposed.

9. The method for detecting abnormal operation of a crane trolley frame according to claim 8, characterized in that: The method of judging whether the crane trolley frame is operating abnormally according to the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis includes: If the deviation between the actual position of the target corner point and its ideal trajectory exceeds the preset offset threshold, the crane trolley frame will operate abnormally; Perform time domain and frequency domain statistical feature analysis on the vibration signal of the target corner point on the time axis to determine whether the crane trolley frame is operating abnormally.

10. A crane trolley frame operation abnormality detection system, characterized in that: The method for detecting abnormal operation of a crane trolley frame according to any one of claims 1 to 9 comprises: A mounting module for mounting the target including corner points on a crane trolley frame; An acquisition module, used to acquire a monitoring video image of the position of the crane trolley frame including the target; A detection module is used to detect the surveillance video image based on the target detection model and obtain the target area; A coordinate transformation module is used to obtain the coordinates of the target corner points according to the target area based on the sub-pixel precision method and the coordinate transformation method; The analysis module is used to obtain the motion trajectory of the target corner point on the spatial axis and the vibration signal on the time axis according to the coordinates of the target corner point; The judgment module is used to judge whether the crane trolley frame is operating abnormally according to the motion trajectory of the target corner point on the space axis and the vibration signal on the time axis.

Images