Method and device for tracking and drop detection of a suspended object based on multi-sensor fusion

Abstract

The application discloses a method and device for tracking and drop detection of a hanging object based on multi-sensor fusion. The method comprises the following steps: monitoring the main and auxiliary hoist revolutions through an encoder to adjust the vertical rotation angle of a camera, and ensuring that the visual angle covers the hook and the hanging object; using a six-axis sensor on the hook to determine its motion state, if it is in motion, analyzing the displacement of the image SIFT feature points to calculate the background displacement, and using the mean square deviation to divide the background and the motion area, and combining the encoder data to determine the position of the hook and the hanging object; based on the visual judgment result and the six-axis sensor information, applying the PID control logic to adjust the speed and angle of the camera in real time, so that the hanging object is kept in the center of the visual angle; when the tower crane moves and the hanging object is stationary, the camera rotation speed is predicted and adjusted according to the motion law of the tower crane, so that the continuous tracking of the hanging object is realized, and whether the hanging object falls is determined according to the multiple positions of the hanging object. Through the method of the application, the precise real-time tracking of the hanging object can be realized, and the method can work effectively even in a dynamic environment or in an abnormal situation.

Description

Technical Field

[0001] This invention relates to the field of construction safety monitoring technology, and more specifically to a method and device for tracking and detecting falling objects based on multi-sensor fusion. Background Technology

[0002] Tower crane operations are an indispensable part of construction, primarily used for material hoisting in high-rise buildings and large structures. They accomplish various construction tasks by suspending heavy objects at high altitudes and precisely moving them to designated locations.

[0003] Accurately determining the position and status changes of the suspended load during operation is crucial, especially at heights, in complex environments, or when visibility is obstructed. Traditional monitoring methods rely primarily on the operator's visual observation and experience, which is not only inefficient but also prone to misjudgment, particularly in poor weather conditions or when surrounded by dense buildings. These limitations result in higher safety hazards at construction sites, such as the risk of accidental collisions or falls of the suspended load.

[0004] Therefore, it is necessary to design a new method to achieve accurate real-time tracking of suspended objects, which can work effectively even in dynamic environments or under abnormal conditions. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and device for tracking and detecting falling objects based on multi-sensor fusion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for tracking and detecting falling objects based on multi-sensor fusion, comprising:

[0007] The encoder obtains the rotation speed of the main winch and the auxiliary winch, and adjusts the vertical rotation angle of the camera according to the rotation speed. The camera is then adjusted according to the vertical rotation angle so that the camera's field of view accurately covers the hook and the suspended object.

[0008] Acquire information from the six-axis sensor on the hook;

[0009] The information from the six-axis sensor is used to determine whether the hook / load is in motion.

[0010] When the hook / load is in motion, the SIFT feature shift of two adjacent frames is analyzed and statistically processed to obtain the background displacement. After aligning one frame with the background of another frame, the mean square error of the corresponding region is calculated.

[0011] The region is divided into a background region and a motion region based on the mean square error, and the position of the hook and the position of the suspended object are determined by combining the information obtained by the encoder to obtain a visual judgment result.

[0012] The position of the suspended object in the camera's field of vision is determined based on the visual judgment result;

[0013] The camera is rotated, and based on the information from the six-axis sensor on the hook, the rotation speed and angle of the camera are adjusted in real time using PID control logic to ensure that the suspended object is always kept in the center of the camera's field of view, so as to reconfirm the visual judgment result.

[0014] Determine whether the suspended object has fallen based on multiple visual assessments.

[0015] The further technical solution is as follows: the rotating camera, based on the information from the six-axis sensor on the hook, uses PID control logic to adjust the rotation speed and angle of the camera in real time to ensure that the suspended object always remains in the center of the camera's field of view, so as to reconfirm the visual judgment result, and also includes:

[0016] If the suspended object is detected to be stationary while the tower crane is moving, the rotation speed of the camera is predicted and adjusted according to the movement pattern of the tower crane to track the suspended object, and the visual judgment result is determined again. If the position information of the suspended object remains unchanged, the action of the camera is determined by referring to the state at the previous moment to track the suspended object, and the visual judgment result is determined again. If the position information of the suspended object is missing, the camera is adjusted according to the cause of the missing position information to track the suspended object, and the visual judgment result is determined again.

[0017] The further technical solution is as follows: The method involves obtaining the rotation speeds of the main and auxiliary winches via an encoder, adjusting the vertical rotation angle of the camera based on the rotation speeds, and adjusting the camera according to the vertical rotation angle to ensure that the camera's field of view accurately covers the hook and the suspended object, including:

[0018] The encoder obtains the rotation speed of the main winch and the auxiliary winch, and the conversion factor is used to calculate the position of the hook and the trolley.

[0019] Based on the positions of the hook and the trolley, the angle of vertical rotation of the camera is determined using trigonometric functions. The direction of the camera is then adjusted according to the calculated vertical rotation angle so that the camera's field of view accurately covers the hook and the suspended object.

[0020] The further technical solution is as follows: The background displacement is obtained by analyzing the SIFT feature shift of two adjacent frames and performing statistical processing. After aligning one frame with the background of another frame, the mean square error of the corresponding region is calculated, including:

[0021] Extract two consecutive frames of image data from the images captured by the camera to obtain two adjacent frames of images;

[0022] The SIFT algorithm is applied to each of the two adjacent frames to extract their respective feature vectors;

[0023] Compare the matching distances between all feature vectors in two adjacent frames of images, determine the best matching pair of feature points as a pair, obtain the pairing vector, and determine the relative displacement of the pairing vector;

[0024] The relative displacement values ​​of the paired vectors are statistically processed, and the mode average of the remaining relative displacement values ​​is calculated after removing outliers.

[0025] The position of one frame is adjusted based on the mode average of the remaining relative displacement values ​​to obtain the adjusted two frames.

[0026] Calculate the mean square error of the corresponding regions in the two adjusted images.

[0027] The further technical solution is as follows: The process of dividing the region into a background region and a motion region based on the mean square error, and determining the hook position and the position of the suspended object by combining the information obtained from the encoder, to obtain a visual judgment result, includes:

[0028] For the mean squared error of each region, determine whether the mean squared error is less than a set threshold.

[0029] If the mean square error is less than a set threshold, the region corresponding to the mean square error is divided into the background region.

[0030] The regions are merged based on proximity relationships;

[0031] The fusion does not occur in the middle of the movement area.

[0032] A further technical solution is as follows: after determining whether the mean square error of each region is less than a set threshold, the method further includes:

[0033] If the mean square error is not less than a set threshold, then the region corresponding to the mean square error is divided into a motion region;

[0034] The regions are merged based on proximity relationships;

[0035] Remove motion regions that are not in the middle to obtain the removal results;

[0036] The position of the hook is determined based on the rejection results and the position information obtained by the encoder, and the position of the suspended object is determined based on the position of the hook to obtain a visual judgment result.

[0037] The further technical solution is as follows: The rotating camera, based on the information from the six-axis sensor on the hook, uses PID control logic to adjust the rotation speed and angle of the camera in real time to ensure that the suspended object always remains in the center of the camera's field of view, so as to re-determine the visual judgment result, including:

[0038] The camera is rotated again, and based on the information from the six-axis sensor on the hook and the distance from the hook to the camera, the required rotation speed of the camera is calculated to correct the camera angle, so as to ensure that the suspended object remains in the center of the camera's field of view.

[0039] When the suspended object deviates from the center of the camera's field of view again, the deviation distance of the suspended object is obtained through image processing algorithm, and PID control is applied to dynamically adjust the rotation speed of the camera to compensate for the deviation distance, ensuring that the suspended object always stays in the center of the camera's field of view, so as to re-determine the visual judgment result.

[0040] The further technical solution is as follows: Predicting and adjusting the rotation speed of the camera based on the tower crane's motion pattern to track the suspended object, and then re-determining the visual judgment result, includes:

[0041] By combining historical motion data and a decision tree model, the real-time motion state of the camera under PID control is predicted to track the suspended object, determine its position, and then re-determine the visual judgment result.

[0042] The further technical solution is as follows: The method of combining historical motion data and a decision tree model to predict the real-time motion state of the camera under PID control to track the suspended object, determine the position of the suspended object, and further determine the visual judgment result includes:

[0043] The camera's angular velocity, angular acceleration, and direction data are recorded in real time to obtain the collected data.

[0044] The CART algorithm is used to construct a decision tree regression model based on the collected data to simulate changes in speed and direction;

[0045] By inputting the angular acceleration and direction of the previous moment into the decision tree regression model, the velocity and direction of the current moment are predicted to obtain the prediction result;

[0046] The camera's control strategy is adjusted based on the prediction results to ensure precise and smooth camera turning in order to track the suspended object and determine its position, thereby reconfirming the visual judgment result.

[0047] The present invention also provides a device for tracking and detecting falling objects based on multi-sensor fusion, comprising:

[0048] A pre-positioning unit is used to obtain the rotation speed of the main winch and the auxiliary winch through an encoder, and adjust the vertical rotation angle of the camera according to the rotation speed, so that the camera's field of view accurately covers the hook and the suspended object.

[0049] The acquisition unit is used to acquire information from the six-axis sensor on the hook.

[0050] The motion state determination unit is used to determine whether the hook / load is in motion based on the information from the six-axis sensor.

[0051] The analysis unit is used to analyze the SIFT feature shift of two adjacent frames of images and statistically process them to obtain the background displacement when the hook / load is in motion. After adjusting one frame of image to align with the background of another frame, the mean square error of the corresponding region is calculated.

[0052] The visual determination unit is used to divide the region into a background region and a motion region based on the mean square error, and to determine the position of the hook and the position of the suspended object by combining the information obtained by the encoder, so as to obtain the visual determination result.

[0053] A position determination unit is used to determine the position of the suspended object in the camera's field of vision based on the visual determination result;

[0054] The PID control unit is used to rotate the camera. Based on the information from the six-axis sensor on the hook, the PID control logic is used to adjust the rotation speed and angle of the camera in real time to ensure that the suspended object is always kept in the center of the camera's field of view, so as to reconfirm the visual judgment result.

[0055] The drop determination unit is used to determine whether the suspended object has fallen based on multiple visual judgment results.

[0056] The advantages of this invention compared to existing technologies are as follows: This invention uses an encoder to obtain the rotation speed of the main and auxiliary winches in real time to adjust the vertical rotation angle of the camera, ensuring that the viewing angle accurately covers the hook and the suspended object. It also utilizes information from a six-axis sensor on the hook to determine whether the hook / suspended object is in motion. When motion is detected, the system analyzes the SIFT feature shift of two adjacent frames to statistically determine the background displacement. After adjusting the image alignment, it calculates the mean square error to distinguish between the background and the motion area. Combined with encoder data, it determines the position of the hook and the suspended object, obtaining a preliminary visual judgment result. Subsequently, PID control logic is used to adjust the camera angle and speed in real time based on feedback from the six-axis sensor, keeping the suspended object in the center of the viewing angle. Multiple visual confirmations are performed, and finally, based on the multiple judgment results, it is determined whether the suspended object has fallen. This method can work effectively even in dynamic environments or abnormal conditions, achieving accurate real-time tracking of the suspended object.

[0057] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0058] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating the multi-sensor fusion-based method for tracking and detecting falling objects provided in an embodiment of the present invention.

[0060] Figure 2 This is a schematic diagram of a sub-process of the multi-sensor fusion-based suspended object tracking and fall detection method provided in an embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram of a sub-process of the multi-sensor fusion-based suspended object tracking and fall detection method provided in an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of a sub-process of the multi-sensor fusion-based suspended object tracking and fall detection method provided in an embodiment of the present invention;

[0063] Figure 5 This is a schematic diagram of a sub-process of the multi-sensor fusion-based suspended object tracking and fall detection method provided in an embodiment of the present invention;

[0064] Figure 6 A schematic diagram of the camera view provided in an embodiment of the present invention;

[0065] Figure 7 This is a schematic diagram of the velocity and acceleration curves of a camera provided in an embodiment of the present invention;

[0066] Figure 8 This is a schematic block diagram of a suspended object tracking and fall detection device based on multi-sensor fusion provided in an embodiment of the present invention;

[0067] Figure 9 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0070] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0071] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0072] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating the multi-sensor fusion-based method for tracking and detecting falling objects provided in this invention. The method is applied to a server that interacts with a camera, two encoders, and a six-axis sensor. Through multi-sensor fusion, it achieves accurate real-time tracking of the suspended object. First, the encoders and the six-axis sensor acquire the winch rotation speed and hook motion state, respectively, and adjust the camera's viewing angle accordingly to ensure target coverage. When object movement is detected, the SIFT (Scale-Invariant Feature Transform) algorithm is used to analyze the displacement of image feature points to identify the background and motion area, and the object's position is determined by combining the mean square error. During dynamic adjustment, the PID (Proportional-Integral-Derivative) control logic adjusts the camera speed and angle in real time based on the six-axis sensor data to keep the object centered in the field of view. Even if image interruption or network lag causes position loss, tracking can be predicted and resumed based on the previous state. In addition, by combining historical data analysis with decision tree models to predict the movement trend of tower cranes, the camera control strategy is further optimized to ensure that the position of the hoisted object can still be accurately tracked in complex or abnormal situations, thereby effectively ensuring operational safety.

[0073] This embodiment's method utilizes hardware consisting of two encoders, a six-axis sensor, a 2K high-definition camera, a tower crane central control system, and other necessary auxiliary devices for detection. Each of the main and auxiliary winches is equipped with an encoder, whose function is to monitor the rotation speed of the hook and trolley motors, respectively. This allows for precise calculation of the hook's actual position. The six-axis sensor, mounted on the hook, captures its attitude changes and rotation during movement. The 2K high-definition camera is located near the boom, below the tower crane cab, and adjusts its viewing angle as the boom rotates. Its primary purpose is to track the suspended object and detect any potential slippage. The tower crane central control system collects various control information about the tower crane operation and feeds this data back to the processing algorithm to ensure operational safety and efficiency. The auxiliary system includes components such as a wireless communication module and a high-performance computer, providing support for the overall system operation. This configuration aims to improve the safety and accuracy of tower crane operations while leveraging advanced sensing technology and image processing capabilities to enhance monitoring of the working environment.

[0074] Figure 1 This is a flowchart illustrating the multi-sensor fusion-based method for tracking and detecting falling objects provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S180.

[0075] S110. Obtain the rotation speed of the main winch and the auxiliary winch through the encoder, and adjust the vertical rotation angle of the camera according to the rotation speed. Adjust the camera according to the vertical rotation angle so that the camera's field of view accurately covers the hook and the suspended object.

[0076] In this embodiment, the rotation speeds of the main and auxiliary winches are obtained through an encoder, and the positions of the hook and trolley are calculated using a conversion factor. Based on the positions of the hook and trolley, the angle of vertical rotation of the camera is determined using trigonometric functions. The direction of the camera is adjusted according to the calculated vertical rotation angle so that the camera's field of view accurately covers the hook and the suspended object.

[0077] Please see Figure 6 Before the system is started, the camera's field of view and the position of the tower crane hook are both random and unknown. To ensure accurate detection and tracking of the hook and the object it is lifting, the camera's field of view must first be adjusted to cover both the hook and the object. Since the camera is mounted near the operator's cab under the boom and rotates with the boom, this can be achieved simply by adjusting the camera's tilt angle.

[0078] Read the current revolution count from the encoders on the main winch and auxiliary winch respectively. and These two values ​​represent the number of revolutions the motor has made since the last reset.

[0079] The conversion factors between the stroke and revolution of the main hoist and the auxiliary hoist are known to be respectively... and Based on this information, the vertical height of the hook can be calculated. (Assuming the hook is at its highest point with a height of 0) and the position of the trolley along the boom direction. (Assume the position of the trolley when it is at the innermost part of the boom is 0).

[0080] The vertical tilt angle θ that the camera should be adjusted to precisely align with the hook is calculated using trigonometric functions. This employs the inverse operation of the tangent function, i.e. Using this formula, we can determine the angle at which the camera needs to be tilted up or down so that its field of view can cover the location of the hook.

[0081] Based on the calculated angle θ, the camera is adjusted accordingly. After adjustment, the camera is locked in this position to ensure it can be continuously and stably aligned with the hook, providing clear and accurate visual information for subsequent detection and tracking.

[0082] By following the steps above, the problem of uncertain camera and hook positions during the initial system startup can be effectively solved, ensuring that the entire system can operate in optimal condition and improving the safety and efficiency of the operation.

[0083] S120, Obtain information from the six-axis sensor on the hook.

[0084] In this embodiment, the six-axis sensor information includes data such as the acceleration and angular velocity of the hook.

[0085] Acceleration and angular velocity data of the hook are collected in real time via a sensor interface. Six-axis sensors are typically equipped with digital output interfaces (such as I2C or SPI) for direct connection to the control system for data transmission. Due to potential noise interference in the real-world environment, filtering algorithms (such as Kalman filters) are applied to process the raw data and improve signal quality.

[0086] S130. Determine whether the hook / load is in motion based on the information from the six-axis sensor.

[0087] In this embodiment, acceleration data is used to determine whether the hook is stationary or in motion. If the acceleration value exceeds a preset threshold, the hook is considered to be moving. By combining acceleration and angular velocity data, the position change and rotation angle change of the hook can be calculated through integral calculation.

[0088] S140. When the hook / load is in motion, the SIFT feature shift of two adjacent frames is analyzed and statistically processed to obtain the background displacement. After aligning one frame with the background of another frame, the mean square error of the corresponding region is calculated.

[0089] In this embodiment, the mean square error of a region refers to the average value of the squared brightness difference of each corresponding region between two frames of images after adjustment, calculated block by block using a step window of size 16×16 pixels, in order to identify changing regions and stable background regions in the image.

[0090] In this embodiment, step S140 aims to determine the background displacement by processing two adjacent frames of images when the hook / load is in motion, and to identify the hook and load by adjusting the position of one frame of image to align with the background of another frame and calculating the mean square error of the corresponding area.

[0091] In one embodiment, please refer to Figure 2 The above-mentioned step S140 may include steps S141 to S146.

[0092] S141. Extract two consecutive frames of image data from the images captured by the camera to obtain two adjacent frames of images.

[0093] First, two consecutive frames are extracted from the video stream captured by the camera as the basis for analysis. This step ensures that there is a sufficient time interval (e.g., 1 second) to capture changes in the movement of the object, while maintaining sufficient correlation for feature matching.

[0094] Specifically, two consecutive frames are extracted from the video stream, arranged in chronological order. Choosing an appropriate frame interval is crucial: if the interval is too short, there may not be enough difference between the two frames to distinguish moving objects from the background; if the interval is too long, it may lead to inaccurate tracking or loss of target. Typically, this interval is adjusted according to the specific scene and requirements.

[0095] S142. Apply the SIFT algorithm to each of the two adjacent frames to extract their respective feature vectors.

[0096] In this embodiment, the Scale Invariant Feature Transform (SIFT) algorithm is applied to both frames of images. The SIFT algorithm can detect and describe local features that are robust to changes in rotation, scaling, and illumination. After processing each frame of the image using the SIFT algorithm, a set of feature points and their corresponding feature vectors are generated.

[0097] Specifically, the SIFT algorithm is a local feature descriptor algorithm used in image processing. It identifies key points in an image and generates a unique descriptor (i.e., feature vector) for each key point. In this embodiment, the SIFT algorithm is applied to each frame of the image to find key points and descriptors that remain unchanged even under different viewpoints, scales, or lighting conditions. This is done to ensure accurate identification and matching of these feature points even when the hook / load is moving.

[0098] S143. Compare the matching distances between all feature vectors in two adjacent frames of images, and determine the best matching pair of feature points as a pair to obtain a pairing vector, and determine the relative displacement of the pairing vector.

[0099] In this embodiment, the pairing vector refers to a vector that describes the relative positional relationship between corresponding feature points in two frames by comparing the positional differences between them.

[0100] Relative displacement refers to the specific value and direction of the positional change of the same feature point in two consecutive frames of images due to the movement of the object or the camera.

[0101] The matching distances between all feature vectors in two adjacent frames are compared. The best matching point pair is typically determined by the ratio of the nearest neighbor to the second nearest neighbor distance. This finds the most closely matched pair of feature points as a pair, and then determines their relative displacement. This process helps identify which feature points have changed due to background pixel movement.

[0102] Once the feature vectors of two frames are obtained, the next step is to compare their similarity. This is typically done using the nearest neighbor to second nearest neighbor distance ratio method. For each feature point, the closest feature point in the other frame is found as the matching point. In this way, we can obtain a series of paired feature points and calculate the relative displacement between each pair.

[0103] S144. Perform statistical processing on the relative displacement values ​​of the paired vectors, remove outliers, and calculate the mode average of the remaining relative displacement values.

[0104] In this embodiment, statistical analysis is performed on the relative displacement values ​​of all paired vectors obtained in the previous step to remove outliers that significantly deviate from the majority value. Then, the mode mean of the remaining relative displacement values ​​is calculated, which represents the general movement of background pixels between two frames.

[0105] Specifically, considering the potential for outliers or noise interference in practical operations, statistical analysis of the relative displacement values ​​of all paired vectors is necessary. First, outliers are removed, and then the mode mean of the remaining relative displacement values ​​is calculated. Here, the "mode mean" refers to the average of the most frequently occurring displacement values. This method helps filter out incorrect matches caused by noise or other unexpected factors.

[0106] S145. Adjust the position of one of the frames based on the mode average of the remaining relative displacement values ​​to obtain the adjusted two frames.

[0107] In this embodiment, based on the mode average obtained in the previous step, i.e. the average displacement of the background pixels, the position of one of the frames is adjusted so that the background parts of the two frames are aligned as much as possible.

[0108] Specifically, based on the mode and mean calculated in the previous step, one of the image frames is translated to align the two frames as closely as possible. This step is crucial because it directly affects the accuracy of the subsequent mean squared error calculation. By aligning the background, moving objects can be more clearly distinguished from the stationary background.

[0109] S146. Calculate the mean square error of the corresponding regions in the two adjusted images.

[0110] In this embodiment, the final step uses a 16×16 pixel stepping window to calculate the mean squared error block by block on the adjusted two frames. Mean squared error is a metric that measures the degree of difference between two images of the same size. In this specific application scenario, it helps identify which areas have changed (such as the movement of a hook or load) and which areas have remained unchanged (such as the background). A smaller mean squared error means that the area is likely to be part of the background, while a larger mean squared error suggests that an object is moving. In other words, areas with smaller mean squared errors are considered background because they show almost no change between the two frames; while areas with larger mean squared errors indicate potentially moving objects, such as hooks and loads.

[0111] The method described in this embodiment can not only effectively track the movement of the hook and the suspended load, but also provide stable and reliable performance in complex environments. This method combines computer vision technology and advanced image processing algorithms, demonstrating the powerful potential of modern technology in the fields of automated monitoring and security inspection.

[0112] Through the steps described above, the system can effectively distinguish between background and foreground objects (hook and load), providing accurate and reliable tracking results even in complex environments. This method not only improves detection accuracy but also enhances the system's robustness and adaptability.

[0113] S150. Divide the region into a background region and a motion region according to the mean square error, and determine the position of the hook and the position of the suspended object by combining the information obtained by the encoder, so as to obtain a visual judgment result.

[0114] In this embodiment, the visual determination result includes the position of the hook and the position of the suspended object.

[0115] By analyzing the degree of variation (i.e., mean squared error) in different regions of an image, the background and foreground (moving objects) can be distinguished, and data from other sensors (such as encoders) can be combined to accurately identify the position of the hook and the object it carries.

[0116] In one embodiment, please refer to Figure 3 The above-mentioned step S150 may include steps S151 to S158.

[0117] S151. For the mean square error of each region, determine whether the mean square error is less than a set threshold.

[0118] In this embodiment, a threshold is predefined to determine which areas are considered as a "static" background and which areas are considered as a moving foreground.

[0119] S152. If the mean square error is less than a set threshold, the region corresponding to the mean square error is divided into a background region.

[0120] In this embodiment, when the MSE of a sub-region is lower than a set threshold, it is marked as a background region, which means that this region does not change much between frames and is likely to be a fixed background part.

[0121] S153. Merge the regions based on proximity relationships.

[0122] In this embodiment, connectivity algorithms (such as connected component labeling) are used to merge adjacent regions that both belong to the background, forming larger, more continuous background blocks, thereby simplifying subsequent processing. This merging operation helps reduce the complexity of subsequent processing and improves the accuracy of background recognition.

[0123] S154, Fusion is not in the middle of the motion area.

[0124] In this embodiment, small moving regions located at the edges or off-center of the image are identified and merged with other regions of similar nature. This helps reduce noise interference.

[0125] In the processing, in addition to considering the merging of background areas, it is also necessary to pay attention to moving areas located at the edges of the image or in non-critical positions, and to perform corresponding processing to optimize the background model.

[0126] S155. If the mean square error is not less than a set threshold, then the region corresponding to the mean square error is divided into a motion region.

[0127] In this embodiment, if the mean square error of a certain region is equal to or greater than a set threshold, then the region is considered to contain moving objects and is therefore marked as a moving region; if the mean square error of a certain region is equal to or greater than a set threshold, it indicates that there are significant changes in the region, such as a moving object passing through, and therefore it is classified as a moving region.

[0128] S156. Merge the regions according to their proximity.

[0129] In this embodiment, the processing method is similar to that for background areas, but the purpose here is to merge adjacent motion areas to make the moving object more complete and easier to track. Specifically, based on the principle of proximity, adjacent sub-regions identified as motion areas are merged to more accurately capture and describe the behavior patterns of moving targets.

[0130] S157. Remove motion regions that are not in the middle to obtain the removal results.

[0131] In this embodiment, the elimination result refers to the motion region remaining after eliminating the motion region that is not in the middle.

[0132] Remove motion regions deemed unimportant, especially those located at the image edges, and focus on activity near the image center, as this is often where key actions occur. Specifically, to focus on the main areas of activity, remove motion regions located at the image edges or deemed unimportant, retaining only the core motion regions as the basis for further analysis.

[0133] S158. Determine the position of the hook based on the rejection result and the position information obtained by the encoder, and determine the position of the suspended object based on the position of the hook to obtain a visual judgment result.

[0134] In this embodiment, the exact position of the hook is located using the motion area information processed as described above and the position data provided by the encoder.

[0135] The position of the suspended object is inferred from the position of the hook, and a visual judgment result containing detailed position information of the hook and the suspended object is finally generated.

[0136] Finally, by combining the processed results with data from the encoder, the position of the hook is precisely determined. Once the hook position is clear, the specific position of the load can be inferred based on the physical relationship between the hook and the load (direct suspension or connection via rope), ultimately generating a visual determination result that includes the position information of both the hook and the load.

[0137] The entire process integrates image processing technology and sensor data fusion methods to achieve intelligent analysis and judgment of the state of objects in specific scenarios.

[0138] S160. Determine the position of the suspended object in the camera's vision based on the visual judgment result.

[0139] In this embodiment, the visual judgment results obtained above are converted into specific coordinate information to clarify the specific position of the suspended object in the camera's field of view, which is crucial for accurate positioning in the automation system.

[0140] Specifically, the visual judgment results obtained through image processing and algorithm analysis are converted into specific coordinate information to determine the exact position of the suspended object within the camera's field of view. This process is crucial for the precise positioning of automated systems.

[0141] First, based on the visual judgment results obtained in the previous steps (e.g., distinguishing between background and moving areas by calculating the mean square error and identifying the positions of the hook and the suspended object), these results are converted into position data in coordinate form. This typically involves mapping from pixel coordinates to physical coordinates. Using this coordinate information, the positional deviation of the suspended object relative to the camera center can be accurately determined. This is fundamental for subsequent control logic to adjust the camera orientation so that the suspended object is repositioned in the center of the field of view.

[0142] S170. Rotate the camera and, based on the information from the six-axis sensor on the hook, use PID control logic to adjust the camera's rotation speed and angle in real time to ensure that the suspended object remains in the center of the camera's field of view, so as to reconfirm the visual judgment result.

[0143] In this embodiment, data from a six-axis sensor on the hook and PID control logic are used to dynamically adjust the camera's orientation, ensuring the suspended load remains centered in the camera's field of view. This is a cyclic feedback process designed to overcome the problem of load position shift caused by tower crane movement.

[0144] In one embodiment, please refer to Figure 4 The above-mentioned step S180 may include steps S181 to S182.

[0145] S181. Rotate the camera and, based on the information from the six-axis sensor on the hook and the distance from the hook to the camera, calculate the required rotation speed of the camera to correct the camera angle, ensuring that the suspended object remains in the center of the camera's field of view.

[0146] In this embodiment, after the position of the suspended object is first obtained, the system attempts to move the object to the center of the camera's field of view based on the current visual judgment result. At this time, considering that visual judgment may be lagging, the system will also combine the speed and position information provided by the six-axis sensor on the hook to make preliminary corrections to the camera's rotation.

[0147] Based on the visual assessment, the position of the suspended object in the camera's field of view is determined. The camera is then rotated again to center the object within the camera's field of view. Because the visual assessment of the suspended object has a lag, after the initial position is determined, the rotation speed and angle of the camera are corrected using the speed and position data from a six-axis sensor. The speed of the six-axis sensor and the rotation speed of the camera can be approximated as... ,in v is the camera rotation speed, v is the speed detected by the six-axis sensor, and r is the distance from the hook to the camera.

[0148] Therefore, it can be seen that using the formula This is used to approximate the rotational speed the camera should have, where v is the speed detected by the six-axis sensor and r is the distance from the hook to the camera. This step helps compensate for errors caused by visual delay.

[0149] S182. When the suspended object deviates from the center of the camera's field of view again, the deviation distance of the suspended object is obtained through an image processing algorithm, and the rotation speed of the camera is dynamically adjusted by PID control to compensate for the deviation distance, ensuring that the suspended object always stays in the center of the camera's field of view, so as to re-determine the visual judgment result.

[0150] In this embodiment, when the tower crane's movement causes the suspended load to deviate from the center of the camera's field of view again, PID control logic is used for compensation. Specifically, the application of PID control logic is as follows:

[0151] Proportional (P) control: The rotation speed of the camera is adjusted directly based on the distance (Δx) of the suspended object deviating from the center of the camera.

[0152] Integral (I) control: Accumulates position deviations over a period of time to eliminate long-term steady-state errors.

[0153] Differential (D) control: predicts future trends based on the rate of change of position deviation and responds quickly to impending changes.

[0154] PID tuning formula is By adjusting the three coefficients To optimize the control effect, among which, This is the proportional coefficient of the proportional term. It determines the degree to which the control output is directly adjusted based on the current error (i.e., the deviation Δx between the current position of the load and the target position). A larger value indicates a more accurate and efficient adjustment. A higher value will make the system respond faster, but if set too high, it may cause system instability or oscillation. This is the proportionality coefficient of the integral term. The function of the integral term is to accumulate past errors to help eliminate long-standing steady-state errors. This is achieved by adjusting... A higher value can increase the system's responsiveness to small, persistent errors. However, if Setting it too high may cause system overshoot or instability. It is the summation of errors over time. It reflects the history of the system's failure to correct errors in a timely manner, and helps to eventually eliminate any remaining static errors. It is the proportionality constant of the differential term. The differential term takes into account the rate of change of the error (i.e., the speed at which the error changes). This allows for the prediction of future error trends and the making of adjustments in advance. This helps improve the system's response speed and reduce overshoot. However, if... If the value is too high, it may amplify the effects of noise, leading to a decrease in system performance; This indicates the rate at which the error changes over time. A rapidly changing error means that the system state is changing quickly, and more aggressive adjustment measures may be needed to correct this change.

[0155] As the tower crane continues to move, the load may deviate from the center of the camera again. Therefore, it is necessary to continuously repeat the above process, that is, dynamically monitor the deviation distance Δx of the load in the image and adjust the rotation speed of the camera accordingly to ensure that the load remains in the center of the camera's field of view.

[0156] After each adjustment, the position of the suspended object needs to be reassessed, and adjustments should continue based on the new visual judgment results until satisfactory positioning accuracy is achieved. The position of the suspended object is then reassessed, mainly by executing steps S140~S150, to reconfirm the visual judgment results.

[0157] In this way, even in complex and dynamic environments, the suspended object can be effectively tracked stably within the camera's field of view, providing a reliable position reference for subsequent operations.

[0158] Subsequently, each time the position of the suspended object is acquired, the rotation speed of the camera is corrected using PID to ensure that the suspended object is always located near the center of the image.

[0159] In step S180, when it is detected that the suspended object is stationary while the tower crane is moving, and the suspended object is stationary in the image while the tower crane is in motion, the rotation speed of the camera is predicted and adjusted according to the movement law of the tower crane to track the suspended object, and the visual judgment result is determined again. When the position information of the suspended object remains unchanged, and the image captured by the camera is interrupted or the network is slowed down, causing the position of the suspended object to be temporarily lost, or the position of the suspended object in the image captured by the camera stops updating, the action of the camera is determined by referring to the state at the previous moment to track the suspended object, and the visual judgment result is determined again. If the position information of the suspended object is missing, the camera is adjusted according to the cause of the missing position to track the suspended object, and the visual judgment result is determined again.

[0160] Specifically, when the suspended object is detected to be stationary in the image while the tower crane is in motion, the rotation speed of the camera is predicted and adjusted according to the movement pattern of the tower crane to track the suspended object, and the visual judgment result is determined again. If the position information of the suspended object cannot be obtained due to image transmission interruption or network delay, or if the position of the suspended object does not change in consecutive image frames, the action of the camera is determined by referring to the state at the previous moment to track the suspended object, and the visual judgment result is determined again. If the position of the suspended object is lost but the tower crane is still moving, the rotation speed of the last recorded camera is used to run the camera.

[0161] In this embodiment, the above-mentioned content addresses several complex situations that may be encountered during the tracking of suspended objects, including the object's stagnation in the image, image interruption, or loss of position due to network lag.

[0162] When the load remains stationary in the image while the tower crane is in motion, the system will predict and adjust the camera's rotation speed based on the tower crane's motion patterns. This ensures that even if the load temporarily stops moving, the camera can still dynamically adjust its own trajectory according to the tower crane's movements, thereby maintaining effective tracking of the load.

[0163] If an image interruption or network lag causes a temporary loss of the suspended object's position information, the system will refer to the previous state (such as angular velocity, angular acceleration, etc.) to determine the camera's next action. This minimizes tracking errors caused by information loss. If the system finds that the suspended object's position in the image is no longer changing, but the tower crane is still moving, it will use the last recorded camera rotation speed to continue operating the camera in an attempt to recapture the suspended object's position.

[0164] In one embodiment, the above-mentioned process of predicting and adjusting the rotation speed of the camera based on the movement law of the tower crane to track the suspended object and then re-determining the visual judgment result includes:

[0165] By combining historical motion data and a decision tree model, the real-time motion state of the camera under PID control is predicted to track the suspended object, determine its position, and then re-determine the visual judgment result.

[0166] In one embodiment, please refer to Figure 5 The system combines historical motion data and a decision tree model to predict the real-time motion state of the camera under PID control in order to track the suspended object, determine the position of the suspended object, and re-determine the visual judgment result, including steps S181' to S184'.

[0167] S181' Records the required angular velocity, angular acceleration, and direction data of the camera in real time to obtain the collected data.

[0168] In this embodiment, the camera's angular velocity, angular acceleration, and direction are recorded in real time, meeting specific requirements. This data forms the basis for building a predictive model used to analyze the camera's current and future motion trends.

[0169] S182' Utilize the CART algorithm to construct a decision tree regression model based on the collected data to simulate changes in speed and direction.

[0170] In this embodiment, a decision tree regression model is constructed using the CART algorithm based on the data collected in step S181'. This model can simulate the changing patterns of the camera's speed and direction, which is crucial for predicting its future actions.

[0171] S183' By inputting the angular acceleration and direction of the previous moment into the decision tree regression model, the velocity and direction of the current moment are predicted to obtain the prediction result.

[0172] In this embodiment, the prediction result refers to the velocity and direction at the current moment predicted using a decision tree regression model.

[0173] By inputting the angular acceleration and direction from the previous moment into the decision tree regression model, the system can predict the velocity and direction at the current moment. This process helps the system make more accurate real-time decisions to adapt to the constantly changing environment.

[0174] S184' Adjust the control strategy of the camera according to the prediction result to ensure that the camera turns accurately and smoothly to track the suspended object and determine the position of the suspended object in order to re-determine the visual judgment result.

[0175] In this embodiment, based on the prediction results obtained from the decision tree regression model, the system adjusts the camera's control strategy to ensure that it can turn accurately and smoothly, continuously track the suspended object, and finally determine the exact position of the suspended object to reconfirm the result of the visual judgment.

[0176] In case of abnormal situations during camera rotation, such as image interruption or network lag causing the suspended object to lose its position for a period of time or its position in the image to stop updating, the system will determine the camera's action based on the previous state. The specific strategy is as follows:

[0177] If the load is lost while the tower crane is still in motion: In this situation, the camera will continue to operate at the rotation speed of the previous moment in the hope of re-capturing the position of the load.

[0178] If the suspended object appears stationary in the image, but the tower crane is still moving, the system will predict and adjust the camera's rotation speed based on the tower crane's motion pattern to ensure it can adapt to the tower crane's movement and maintain effective tracking of the suspended object.

[0179] To more precisely control the camera's movement, the system employs a method based on historical data and machine learning models to predict the camera's rotation speed. This process follows the principles outlined below, and the resulting graph is shown in the image. Figure 7 As shown:

[0180] Oscillation Approaching Target Speed: Considering the camera's operation under PID control and the influence of the actual tower crane's movement, its rotation speed often gradually approaches the desired speed in an oscillating manner. In other words, the change in rotation speed is not linear, but rather achieved through the back-and-forth oscillation of angular acceleration.

[0181] Slow-amplitude changes: Under the control of the tower crane, the speed of the camera usually changes slowly, which may be an increase, a decrease or an overall constant.

[0182] Based on the above principles, the current rotation speed of the camera depends not only on the rotation speed at the previous moment, but also on the angular acceleration at the previous moment.

[0183] To implement the above strategy, the system takes the following steps:

[0184] Dynamic recording: During normal operation, the system dynamically records twenty sets of angular velocity and angular acceleration values ​​and their direction information within the three seconds preceding the current moment. This data provides the foundation for subsequent model building.

[0185] Building a decision tree regression model: Using the collected data, a decision tree regression model was built using the CART algorithm. This model aims to simulate the changing patterns of camera speed and orientation over time, which is crucial for predicting future actions.

[0186] Prediction and Adjustment: By inputting the angular acceleration and direction from the previous moment into the decision tree regression model, the system can predict the current velocity and direction. Based on this prediction, the system can adjust the camera's control strategy in a timely manner to ensure precise and smooth turning, continuous tracking of the suspended object, and determination of its exact position.

[0187] This method effectively combines real-time data analysis, machine learning techniques (such as decision tree regression models), and traditional PID control strategies, thereby significantly improving the robustness and accuracy of the system. Even in challenging situations such as image interruption or network lag, it can efficiently complete the task of tracking suspended objects.

[0188] This method not only improves the robustness and accuracy of camera tracking systems but also effectively addresses challenging issues such as image interruptions and network lag, ensuring efficient target tracking under various conditions. By integrating real-time data analysis, machine learning model prediction, and traditional PID control techniques, this method significantly enhances the overall performance of the system.

[0189] S190. Determine whether the suspended object has fallen based on multiple visual judgment results.

[0190] In this embodiment, multiple visual determination results refer to the position information of the suspended object determined by rotating the camera at least twice.

[0191] First, multiple visual assessments confirm that the hook is in the air and not placed on the ground or other supported surface. This can be determined by detecting the hook's height. If the hook height exceeds a preset threshold, it is considered to be in the air.

[0192] After confirming that the hook is in the air, the next crucial step is to observe whether the vertical distance between the load and the hook has increased significantly. Normally, during hoisting, the vertical distance between the load and the hook should remain relatively stable or follow an expected trend as the hook rises or falls (e.g., the distance decreases when the hook rises and may increase when it falls). However, if a sudden and significant increase in the vertical distance between the load and the hook is observed, and this increase is beyond what can be explained by normal operation, it indicates that the load has detached from the hook and begun to fall freely.

[0193] The method in this embodiment uses an encoder and a six-axis sensor to assist in visual positioning, tracking the position of the object suspended below the hook, and performing fall detection analysis based on this. The main advantage of this method is its versatility, i.e., it does not require prior learning or constraints on specific types of suspended objects.

[0194] By fusing multiple sensors, including encoders and six-axis sensors, the system can track the specific position of the load below the hook in real time. Regardless of the shape or size of the load, as long as it can be visually identified and its motion can be indirectly reflected by data from the six-axis sensors and encoders, it can be effectively tracked.

[0195] The advantage of this method is that it does not depend on the specific type of the object being lifted, and there is no need to extract features and train on various possible objects beforehand, which greatly improves the adaptability and flexibility of the system.

[0196] After obtaining continuous data on the position of the suspended load, the system can build a model of the changes in the position of the load relative to the hook. When an abnormal change in the position of the load relative to the hook is detected (such as a sudden increase in vertical distance), it can be determined that the load may have fallen.

[0197] Fall detection not only considers changes in the distance between the suspended load and the hook, but also comprehensively analyzes parameters such as the load's trajectory and speed. For example, under normal circumstances, the load will move with the hook; if the load's speed is found to significantly exceed the hook's own speed, or if there are signs of free fall, it further confirms the possibility of a fall.

[0198] In summary, this method, by combining the advantages of multiple sensors, achieves effective tracking of different types of suspended objects and can promptly detect instances of objects falling, demonstrating strong application value and wide applicability. This method is particularly suitable for complex and variable working environments, such as tower crane operations on construction sites.

[0199] The aforementioned multi-sensor fusion-based method for tracking and detecting falling objects uses an encoder to obtain real-time information on the rotation of the main and auxiliary winches to adjust the vertical rotation angle of the camera, ensuring accurate coverage of the hook and the object. It then utilizes information from a six-axis sensor on the hook to determine if the hook / object is in motion. When motion is detected, the system analyzes the SIFT feature shift of two adjacent frames to statistically determine background displacement. After image alignment, it calculates the mean square error to distinguish between the background and the moving area. Combined with encoder data, it determines the position of the hook and object, obtaining a preliminary visual judgment result. Subsequently, PID control logic adjusts the camera angle and speed in real-time based on feedback from the six-axis sensor, keeping the object centered in the field of view. Multiple visual confirmations are performed, and finally, based on the multiple judgment results, it is determined whether the object has fallen. This method works effectively even in dynamic environments or abnormal conditions, achieving accurate real-time tracking of the object.

[0200] Figure 8 This is a schematic block diagram of a multi-sensor fusion-based suspended object tracking and fall detection device 300 provided in an embodiment of the present invention. Figure 8As shown, corresponding to the above-described multi-sensor fusion-based suspended object tracking and fall detection method, the present invention also provides a multi-sensor fusion-based suspended object tracking and fall detection device 300. This multi-sensor fusion-based suspended object tracking and fall detection device 300 includes a unit for executing the above-described multi-sensor fusion-based suspended object tracking and fall detection method, and the device can be configured in a server. Specifically, please refer to... Figure 8 The multi-sensor fusion-based suspended object tracking and fall detection device 300 includes a pre-positioning unit 301, an acquisition unit 302, a motion state determination unit 303, an analysis unit 304, a vision judgment unit 305, a position determination unit 306, a PID control unit 307, and a fall determination unit 308.

[0201] The pre-positioning unit 301 is used to acquire the rotation speed of the main winch and the auxiliary winch through the encoder, and adjust the vertical rotation angle of the camera according to the rotation speed, so that the camera's field of view accurately covers the hook and the suspended object; the acquisition unit 302 is used to acquire the six-axis sensor information on the hook; the motion state determination unit 303 is used to determine whether the hook / suspended object is in motion according to the six-axis sensor information; the analysis unit 304 is used to analyze the SIFT feature shift of two adjacent frames of images and statistically process it to obtain the background displacement when the hook / suspended object is in motion, adjust one frame of image to align with the background of another frame, and calculate the mean square of the corresponding region. The system comprises: a visual judgment unit 305, used to divide the region into a background region and a motion region based on the mean square error, and to determine the position of the hook and the position of the suspended object by combining the information obtained by the encoder, so as to obtain a visual judgment result; a position determination unit 306, used to determine the position of the suspended object in the camera's vision based on the visual judgment result; a PID control unit 307, used to rotate the camera and, based on the information from the six-axis sensor on the hook, to adjust the rotation speed and angle of the camera in real time using PID control logic to ensure that the suspended object always remains in the center of the camera's field of view, so as to re-determine the visual judgment result; and a drop determination unit 308, used to determine whether the suspended object has fallen based on multiple visual judgment results.

[0202] In one embodiment, the system described above further includes:

[0203] The adjustment unit is used to predict and adjust the rotation speed of the camera according to the movement law of the tower crane if the suspended object is detected to be stationary while the tower crane is moving, so as to track the suspended object and re-determine the visual judgment result; if the position information of the suspended object remains unchanged, the camera's action is determined by referring to the state of the previous moment to track the suspended object and re-determine the visual judgment result; if the position information of the suspended object is missing, the camera is adjusted according to the cause of the missing position information to track the suspended object and re-determine the visual judgment result.

[0204] In one embodiment, the pre-positioning unit 301 is used to obtain the rotational speeds of the main winch and the auxiliary winch through an encoder, and to calculate the positions of the hook and the trolley using a conversion factor;

[0205] Based on the positions of the hook and the trolley, the angle of vertical rotation of the camera is determined using trigonometric functions. The direction of the camera is then adjusted according to the calculated vertical rotation angle so that the camera's field of view accurately covers the hook and the suspended object.

[0206] In one embodiment, the analysis unit 304 includes:

[0207] The image extraction subunit is used to extract two consecutive frames of image data from the images captured by the camera to obtain two adjacent frames of images; the feature extraction subunit is used to apply the SIFT algorithm to each frame of the two adjacent frames of images to extract their respective feature vectors; the comparison subunit is used to compare the matching distances between all feature vectors in the two adjacent frames of images, and determine the most matching pair of feature points as a pair to obtain a pairing vector, and determine the relative displacement of the pairing vector; the processing subunit is used to perform statistical processing on the relative displacement values ​​of the pairing vectors, remove outliers, and calculate the mode mean of the remaining relative displacement values; the adjustment subunit is used to adjust the position of one of the frames of images according to the mode mean of the remaining relative displacement values ​​to obtain two adjusted frames of images; the calculation subunit is used to calculate the root mean square error of the corresponding regions of the two adjusted frames of images.

[0208] In one embodiment, the visual determination unit 305 includes:

[0209] The system includes a judgment subunit, used to determine whether the mean squared error of each region is less than a set threshold; a first division subunit, used to divide the region corresponding to the mean squared error into a background region if the mean squared error is less than the set threshold; a first fusion subunit, used to fuse the regions based on proximity; a second fusion subunit, used to fuse motion regions that are not in the middle; a second division subunit, used to divide the region corresponding to the mean squared error into a motion region if the mean squared error is not less than the set threshold; a third fusion subunit, used to fuse the regions based on proximity; a rejection subunit, used to reject motion regions that are not in the middle to obtain a rejection result; and a position determination subunit, used to determine the position of the hook based on the rejection result and the position information obtained by the encoder, and to determine the position of the suspended object based on the position of the hook to obtain a visual judgment result.

[0210] In one embodiment, the PID control unit 307 includes:

[0211] The angle calculation subunit is used to rotate the camera again. Based on the information from the six-axis sensor on the hook and the distance from the hook to the camera, it calculates the required rotation speed of the camera and corrects the camera angle to ensure that the suspended object remains in the center of the camera's field of view.

[0212] The compensation subunit is used to obtain the deviation distance of the suspended object through an image processing algorithm when the tower crane's movement causes the suspended object to deviate from the center of the camera's field of view again, and to dynamically adjust the camera's rotation speed using PID control to compensate for the deviation distance, ensuring that the suspended object always remains in the center of the camera's field of view, so as to re-determine the visual judgment result.

[0213] In one embodiment, the adjustment unit is used to combine historical motion data and a decision tree model to predict the real-time motion state of the camera under PID control, so as to track the suspended object, determine the position of the suspended object, and then re-determine the visual judgment result.

[0214] In one embodiment, the adjustment unit includes:

[0215] The system comprises the following subunits: a recording subunit for real-time recording of the camera's angular velocity, angular acceleration, and direction data as required; a model building subunit for constructing a decision tree regression model based on the collected data using the CART algorithm to simulate changes in velocity and direction; a prediction subunit for predicting the velocity and direction at the current moment by inputting the angular acceleration and direction from the previous moment into the decision tree regression model; and a control subunit for adjusting the camera's control strategy based on the prediction results to ensure precise and smooth camera turning to track the suspended object and determine the object's position to reconfirm the visual judgment result.

[0216] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned multi-sensor fusion-based suspended object tracking and fall detection device 300 and its various units can be found in the corresponding descriptions in the foregoing method embodiments. For the sake of convenience and brevity, these details will not be repeated here.

[0217] The aforementioned multi-sensor fusion-based suspended object tracking and fall detection device 300 can be implemented as a computer program, which can be used in, for example... Figure 9 It runs on the computer device shown.

[0218] Please see Figure 9 , Figure 9 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0219] See Figure 9 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0220] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a method for tracking and detecting falling objects based on multi-sensor fusion.

[0221] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0222] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a method for tracking and detecting falling objects based on multi-sensor fusion.

[0223] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0224] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps:

[0225] The encoder obtains the rotation speed of the main winch and the auxiliary winch, and adjusts the vertical rotation angle of the camera according to the rotation speed. The camera is then adjusted according to the vertical rotation angle so that the camera's field of view accurately covers the hook and the suspended object.

[0226] The process involves: acquiring information from a six-axis sensor on the hook; determining whether the hook / load is in motion based on the six-axis sensor information; when the hook / load is in motion, analyzing the SIFT feature shift of two adjacent frames and statistically processing it to obtain the background displacement; aligning one frame with the background of another frame and calculating the mean square error of the corresponding region; dividing the region into background and motion regions based on the mean square error, and determining the hook and load positions using information from the encoder to obtain a visual judgment result; determining the position of the load in the camera's field of vision based on the visual judgment result; rotating the camera and adjusting its rotation speed and angle in real time using PID control logic based on the six-axis sensor information on the hook to ensure the load remains centered in the camera's field of view, to re-determine the visual judgment result; and finally, determining whether the load has fallen based on multiple visual judgment results.

[0227] In one embodiment, when the processor 502 implements the step of rotating the camera and adjusting the camera's rotation speed and angle in real time using PID control logic based on the information from the six-axis sensor on the hook to ensure that the suspended object always remains in the center of the camera's field of view, in order to re-determine the visual judgment result, the processor 502 specifically implements the following steps:

[0228] If the suspended object is detected to be stationary while the tower crane is moving, the rotation speed of the camera is predicted and adjusted according to the movement pattern of the tower crane to track the suspended object, and the visual judgment result is determined again. If the position information of the suspended object remains unchanged, the action of the camera is determined by referring to the state at the previous moment to track the suspended object, and the visual judgment result is determined again. If the position information of the suspended object is missing, the camera is adjusted according to the cause of the missing position information to track the suspended object, and the visual judgment result is determined again.

[0229] In one embodiment, when the processor 502 implements the steps of obtaining the rotation speeds of the main and auxiliary winches through the encoder, adjusting the vertical rotation angle of the camera according to the rotation speeds, and adjusting the camera according to the vertical rotation angle so that the camera's field of view accurately covers the hook and the suspended object, the processor 502 specifically implements the following steps:

[0230] The encoder obtains the rotation speed of the main winch and the auxiliary winch, and the conversion coefficient is used to calculate the position of the hook and the trolley. Based on the position of the hook and the trolley, the angle of vertical rotation of the camera is determined by trigonometric function relationship. The direction of the camera is adjusted according to the calculated vertical rotation angle so that the camera's field of view accurately covers the hook and the suspended object.

[0231] In one embodiment, when the processor 502 performs the steps of analyzing the SIFT feature shift of two adjacent frames and statistically processing to obtain the background displacement, adjusting one frame image to align with the background of another frame, and then calculating the mean square error of the corresponding region, the specific implementation is as follows:

[0232] Two consecutive frames of image data are extracted from the images captured by the camera to obtain two adjacent frames. The SIFT algorithm is applied to each of the two adjacent frames to extract its respective feature vector. The matching distances between all feature vectors in the two adjacent frames are compared, and the best-matching pair of feature points is determined as a pair to obtain a pairing vector. The relative displacement of the pairing vector is then determined. Statistical processing is performed on the relative displacement values ​​of the pairing vectors, outliers are removed, and the mode mean of the remaining relative displacement values ​​is calculated. The position of one of the frames is adjusted based on the mode mean of the remaining relative displacement values ​​to obtain two adjusted frames. The root mean square error of the corresponding regions in the two adjusted frames is calculated.

[0233] In one embodiment, when the processor 502 performs the step of dividing the region into a background region and a motion region based on the mean square error, and determining the position of the hook and the position of the suspended object in combination with the information obtained by the encoder to obtain a visual judgment result, the processor 502 specifically implements the following steps:

[0234] For the mean squared error of each region, determine whether the mean squared error is less than a set threshold; if the mean squared error is less than the set threshold, divide the region corresponding to the mean squared error into a background region; merge the regions according to the proximity relationship; merge the motion regions that are not in the middle.

[0235] In one embodiment, after implementing the step of determining whether the mean squared error for each region is less than a set threshold, the processor 502 further implements the following steps:

[0236] If the mean square error is not less than a set threshold, the region corresponding to the mean square error is divided into motion regions; the regions are fused according to proximity relationships; motion regions not in the middle are removed to obtain removal results; the position of the hook is determined according to the removal results and the position information obtained by the encoder, and the position of the suspended object is determined according to the position of the hook to obtain a visual judgment result.

[0237] In one embodiment, when the processor 502 implements the step of rotating the camera and adjusting the camera's rotation speed and angle in real time using PID control logic based on the information from the six-axis sensor on the hook to ensure that the suspended object always remains in the center of the camera's field of view, in order to re-determine the visual judgment result, the processor 502 specifically implements the following steps:

[0238] The camera is rotated, and based on the information from the six-axis sensor on the hook and the distance from the hook to the camera, the required rotation speed of the camera is calculated to correct the camera angle, ensuring that the suspended object remains in the center of the camera's field of view. When the tower crane's movement causes the suspended object to deviate from the center of the camera's field of view again, the deviation distance of the suspended object is obtained through image processing algorithms, and PID control is applied to dynamically adjust the camera's rotation speed to compensate for the deviation distance, ensuring that the suspended object always remains in the center of the camera's field of view, so as to re-determine the visual judgment result.

[0239] In one embodiment, when the processor 502 implements the step of predicting and adjusting the rotation speed of the camera based on the movement law of the tower crane to track the suspended object and then determining the visual judgment result again, the processor 502 specifically implements the following steps:

[0240] By combining historical motion data and a decision tree model, the real-time motion state of the camera under PID control is predicted to track the suspended object, determine its position, and then re-determine the visual judgment result.

[0241] In one embodiment, when the processor 502 implements the step of combining historical motion data and a decision tree model to predict the real-time motion state of the camera under PID control in order to track the suspended object and determine the position of the suspended object in order to re-determine the visual judgment result, the processor 502 specifically implements the following steps:

[0242] The camera's angular velocity, angular acceleration, and direction data are recorded in real time to obtain the collected data. A decision tree regression model is constructed based on the collected data using the CART algorithm to simulate changes in velocity and direction. The angular acceleration and direction at the previous moment are input into the decision tree regression model to predict the velocity and direction at the current moment to obtain the prediction result. The camera's control strategy is adjusted according to the prediction result to ensure that the camera turns accurately and smoothly to track the suspended object and determine the position of the suspended object to reconfirm the visual judgment result.

[0243] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0244] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0245] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0246] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0247] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0248] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0249] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for tracking and drop detection of a suspended object based on multi-sensor fusion, characterized in that, The method comprises the following steps: acquiring the number of rotations of the main winch and the auxiliary winch through an encoder, and adjusting the vertical rotation angle of the camera according to the number of rotations, and adjusting the camera according to the vertical rotation angle so that the visual angle of the camera accurately covers the hook and the hoisted object; acquiring the six-axis sensor information on the hook; determining whether the hook / hoisted object is in a motion state according to the six-axis sensor information; when the hook / hoisted object is in a motion state, analyzing the SIFT feature vectors of the adjacent two frames of images and obtaining the background displacement through statistical processing, calculating the mean square error of the corresponding region after aligning one frame of image with another frame of background; dividing the region into a background region and a motion region according to the mean square error, and determining the hook position and the hoisted object position in combination with the information acquired by the encoder to obtain a visual judgment result; determining the position of the hoisted object in the camera vision according to the visual judgment result; rotating the camera, adjusting the rotation speed and angle of the camera in real time according to the six-axis sensor information on the hook by using a PID control logic, ensuring that the hoisted object is always kept in the center of the camera visual angle, and determining the visual judgment result again; determining whether the hoisted object has fallen according to multiple visual judgment results. 2.The multi-sensor fusion based load tracking and drop detection method according to claim 1, wherein, The method of rotating the camera, adjusting the rotation speed and angle of the camera in real time according to the six-axis sensor information on the hook by using a PID control logic, ensuring that the hoisted object is always kept in the center of the camera visual angle, and determining the visual judgment result again, further comprises: if the hoisted object is detected to be stationary while the tower crane is moving, predicting and adjusting the rotation speed of the camera according to the motion law of the tower crane to track the hoisted object, determining the visual judgment result again; if the hoisted object position information is always unchanged, determining the action of the camera by referring to the state at the previous moment to track the hoisted object, determining the visual judgment result again; if the hoisted object position information is missing, adjusting the camera according to the reason causing the hoisted object position to be missing to track the hoisted object, and determining the visual judgment result again. 3.The multi-sensor fusion based load tracking and drop detection method according to claim 1, wherein, The method of acquiring the number of rotations of the main winch and the auxiliary winch through an encoder, adjusting the vertical rotation angle of the camera according to the number of rotations, and adjusting the camera according to the vertical rotation angle so that the visual angle of the camera accurately covers the hook and the hoisted object, comprises: acquiring the number of rotations of the main winch and the auxiliary winch through an encoder, and calculating the positions of the hook and the trolley by using a conversion coefficient; determining the angle of vertical rotation of the camera based on the positions of the hook and the trolley by using a trigonometric function relationship, and adjusting the direction of the camera according to the calculated vertical rotation angle so that the visual angle of the camera accurately covers the hook and the hoisted object. 4.The multi-sensor fusion based load tracking and drop detection method according to claim 1, wherein, The method of analyzing the SIFT feature vectors of the adjacent two frames of images and obtaining the background displacement through statistical processing, and calculating the mean square error of the corresponding region after aligning one frame of image with another frame of background, comprises: extracting two consecutive frames of image data from the images captured by the camera to obtain the adjacent two frames of images; extracting the feature vectors of each frame of image in the adjacent two frames of images by using a SIFT algorithm; Compare the matching distances between all feature vectors in the two adjacent frames of images, and determine the most matched pair of feature points as a matching pair to obtain a matching vector, and determine the relative displacement of the matching vector; Statistically process the relative displacement values of the matching vector, and calculate the mode average of the remaining relative displacement values after removing outliers; Adjust the position of one of the two frames of images according to the mode average of the remaining relative displacement values to obtain two adjusted frames of images; Calculate the mean square error of the corresponding regions of the two adjusted frames of images. 5.The multi-sensor fusion based load tracking and drop detection method according to claim 1, wherein, According to the mean square error, the region is divided into background region and motion region, and the position of the hook and the position of the hoisted object are determined in combination with the information obtained by the encoder to obtain a visual judgment result, including: For the mean square error of each region, it is judged whether the mean square error is less than a set threshold value; If the mean square error is less than the set threshold value, the region corresponding to the mean square error is divided into a background region; According to the adjacent relationship, the regions are fused; The motion region not in the middle is fused. 6.The multi-sensor fusion based load tracking and drop detection method according to claim 5, wherein, After the mean square error of each region is judged whether it is less than a set threshold value, it further includes: If the mean square error is not less than the set threshold value, the region corresponding to the mean square error is divided into a motion region; According to the adjacent relationship, the regions are fused; The motion region not in the middle is removed to obtain a removal result; According to the removal result and the position information obtained by the encoder, the position of the hook is judged, and the position of the hoisted object is determined according to the position of the hook to obtain a visual judgment result. 7.The multi-sensor fusion based load tracking and drop detection method according to claim 1, wherein, The rotating camera adjusts the rotating speed and angle of the camera in real time according to the six-axis sensor information on the hook by using PID control logic to ensure that the hoisted object always remains in the center of the camera view angle to determine the visual judgment result again, including: The rotating camera calculates the required rotating speed of the camera to correct the angle of the camera to ensure that the hoisted object remains in the center of the camera view angle according to the six-axis sensor information on the hook and the distance from the hook to the camera; When the hoisted object deviates from the center of the camera view angle again, the deviation distance of the hoisted object is obtained by an image processing algorithm, and the rotating speed of the camera is dynamically adjusted by applying PID control to compensate for the deviation distance, so that the hoisted object always remains in the center of the camera view angle to determine the visual judgment result again. 8.The multi-sensor fusion based load tracking and drop detection method according to claim 1, wherein, The rotating speed of the camera is predicted and adjusted according to the motion law of the tower crane to track the hoisted object to determine the visual judgment result again, including: The instantaneous motion state of the camera under PID control is predicted based on historical motion data and a decision tree model to track the hoisted object and determine the position of the hoisted object to determine the visual judgment result again. 9.The multi-sensor fusion based load tracking and drop detection method according to claim 8, wherein, The instantaneous motion state of the camera under PID control is predicted based on historical motion data and a decision tree model to track the hoisted object and determine the position of the hoisted object to determine the visual judgment result again, including: Real-time record the angular velocity, angular acceleration and direction data of the camera to obtain collected data; A decision tree regression model is constructed based on the collected data by using the CART algorithm to simulate the changes of speed and direction; The angular acceleration and direction of the previous moment are input into the decision tree regression model to predict the speed and direction of the current moment, so as to obtain a prediction result; According to the prediction result, the control strategy of the camera is adjusted to ensure accurate and smooth turning of the camera to track the suspended object and determine the position of the suspended object, so as to determine the visual judgment result again.

10. A suspended object tracking and drop detection device based on multi-sensor fusion, characterized in that, Comprise: A pre-positioning unit is configured to obtain the number of revolutions of the main hoist and the auxiliary hoist through an encoder, and adjust the vertical rotation angle of the camera according to the number of revolutions, and adjust the camera according to the vertical rotation angle, so that the visual angle of the camera accurately covers the hook and the suspended object; An acquisition unit is configured to acquire six-axis sensor information on the hook; A motion state determination unit is configured to determine whether the hook / suspended object is in a motion state according to the six-axis sensor information; An analysis unit is configured to analyze the SIFT feature vectors of adjacent two frames of images and obtain background displacement through statistical processing when the hook / suspended object is in a motion state, and calculate the mean square error of the corresponding region after aligning one frame of image with another frame of background; A visual judgment unit is configured to divide the region into a background region and a motion region according to the mean square error, and determine the position of the hook and the position of the suspended object according to the information obtained by the encoder, so as to obtain a visual judgment result; A position determination unit is configured to determine the position of the suspended object in the visual field of the camera according to the visual judgment result; A PID control unit is configured to rotate the camera, adjust the rotation speed and angle of the camera in real time according to the six-axis sensor information on the hook by using PID control logic, and ensure that the suspended object is always kept in the center of the visual angle of the camera, so as to determine the visual judgment result again; A falling determination unit is configured to determine whether the suspended object has fallen according to multiple visual judgment results.

Images