Method and device for monitoring and rectifying deviation of tower crane and tower crane system

Abstract

The invention provides a tower crane monitoring and deviation rectifying method and device and a tower crane system. The tower crane monitoring and deviation rectifying method comprises the steps that a first target image and a second target image shot by a shooting device are obtained; obtaining a target angular point set based on the first target image; based on the target angular point set and the second target image, obtaining an optical flow variable quantity; and outputting deviation compensation information based on the optical flow variable quantity in response to meeting a preset deviation correction condition. In the lifting hook operation process, a first target image and a second target image are collected in real time, deviation compensation information is generated under the condition that the optical flow variable quantity meets the deviation correction condition by calculating the optical flow variable quantity of angular points in the first target image and the second target image, and the deviation compensation information is converted into a control signal to drive a hydraulic rod. And finally, the pitching angle of the shooting device is automatically adjusted, so that the lifting hook is always stably positioned in the center of the shooting view, the lifting hook is prevented from instantly separating from the shooting view of the shooting device and running out of a monitoring picture, and user experience is improved.

Description

Technical Field

[0001] This application relates to the field of tower crane control technology, and in particular to a tower crane monitoring and correction method, device and tower crane system. Background Technology

[0002] With the development of artificial intelligence and the improvement of safety awareness, the use of AI technology to improve the safety of workers' working environment on construction sites is becoming increasingly common. For example, in tower crane operations, a visualization system can be used to track and collect images of the tower crane hook and display the surrounding environment and the status of the goods to the tower crane operator in real time.

[0003] In related technologies, tower cranes are equipped with camera devices on their main body. When the tower crane's boom moves, the camera device swings synchronously to track and capture the hook on the boom, keeping it within the monitoring screen of the visualization system. However, during the swinging process, the camera device may experience sudden large angle changes, deviating from the system's expected swing trajectory. This can cause the hook to momentarily leave the camera's field of view and disappear from the monitoring screen. In such cases, operators need to manually adjust the camera's shooting angle to retrieve the hook, impacting the user experience. Summary of the Invention

[0004] In view of the above, it is necessary to provide a tower crane monitoring and correction method, device and tower crane system that can achieve automatic correction and improve user experience.

[0005] The first aspect of this application provides a tower crane monitoring and correction method applied to a tower crane system. The tower crane system includes a tower crane body, an imaging device, and a drive device. The tower crane body has a tower boom and a hook. When the tower boom moves relative to the tower crane body, the imaging device swings synchronously and aligns with the hook. The tower crane monitoring and correction method includes: acquiring a first target image and a second target image captured by the imaging device, wherein the first target image was captured earlier than the second target image; obtaining a target corner point set based on the first target image, wherein the target corner point set includes multiple corner points, which are used to characterize the area corresponding to the hook in the first target image; obtaining an optical flow change based on the target corner point set and the second target image, wherein the optical flow change reflects the motion change of the corner points; and outputting deviation compensation information based on the optical flow change in response to satisfying a preset correction condition, wherein the deviation compensation information is used to control the drive device to drive the imaging device to swing so that the hook remains within the field of view of the imaging device.

[0006] In some embodiments, obtaining a target corner point set based on a first target image includes: performing corner point detection on the first target image to obtain an initial corner point set, and selecting multiple original corner point clusters from the initial corner point set; selecting regional corner point clusters from the multiple original corner point clusters, wherein the regional corner point clusters include spatial regional corner point clusters that satisfy spatial corner point conditions and background target corner point clusters that satisfy hook corner point conditions, the spatial regional corner point clusters being used to characterize the region corresponding to the activity space of the hook in the first target image, and the spatial regional corner point clusters being used to characterize the region corresponding to the outline of the hook in the first target image; and using the spatial regional corner point clusters and the background target corner point clusters as the target corner point set.

[0007] In some embodiments, the spatial corner condition includes: the candidate distance between the candidate region formed by the original corner cluster and the designated region of the first target image is less than a preset distance threshold.

[0008] In some embodiments, the hook corner point conditions include at least one of the following: the length and width of the minimum bounding rectangle of the candidate region formed by the original corner point cluster are within a preset ratio range; the angle of the included angle formed by the candidate region formed by the original corner point cluster is within a preset angle range; and the graphic features of the candidate region formed by the original corner point cluster match a preset hook template.

[0009] In some embodiments, the method further includes: inputting a first target image into a preset blurred corner detection model to obtain an effective corner cluster, wherein the blurred corner detection model is used to predict and extract corners from a blurred image; and using the spatial region corner cluster, the background target corner cluster, and the effective corner cluster as a target corner set.

[0010] In some embodiments, the optical flow change is obtained based on the target corner point set and the second target image, including: solving the motion vector of each corner point in the target corner point set based on the grayscale image of the second target image to obtain the optical flow of each corner point, so as to obtain the optical flow field corresponding to the spatial region corner point cluster, the background target corner point cluster and the effective corner point cluster; and weighting and summing the optical flow fields corresponding to the spatial region corner point cluster, the background target corner point cluster and the effective corner point cluster based on the weight coefficients corresponding to the spatial region corner point cluster, the background target corner point cluster and the effective corner point cluster, and the influence coefficient of the effective corner point cluster, combined with a preset change time, to obtain the optical flow change.

[0011] In some embodiments, the step of inputting the first target image into a preset blurred corner detection model is performed in response to the first target image satisfying a preset image blur condition; If the first target image does not meet the image blur condition, the influence coefficient is set to 0.

[0012] In some embodiments, the tower crane monitoring and correction method further includes: performing blur detection on the first target image to obtain the blur degree of the first target image; the image blur condition includes: the blur degree of the first target image is greater than a preset blur threshold.

[0013] In some embodiments, the tower crane monitoring and correction method further includes: setting the value of the influence coefficient based on the blur of the first target image.

[0014] In some embodiments, the correction condition includes: the change in optical flow is greater than a preset activation threshold.

[0015] In some embodiments, the correction condition further includes: during the capture time of the first target image and the second target image, the driving device does not drive the shooting device to swing under the control of the interactive terminal.

[0016] The second aspect of this application provides a tower crane monitoring and correction device, applied to a tower crane system. The tower crane system includes a tower crane body, a tower boom, a hook, a camera, and a drive device. When the tower boom moves relative to the tower crane body, the camera simultaneously swings and aligns with the hook. The tower crane monitoring and correction device includes: a data acquisition module for acquiring a first target image and a second target image captured by the camera, wherein the first target image was captured earlier than the second target image; a corner detection module for obtaining a target corner set based on the first target image, wherein the target corner set includes multiple corners, which characterize the area corresponding to the hook in the first target image; an optical flow calculation module for obtaining an optical flow change based on the target corner set and the second target image, wherein the optical flow change reflects the motion change of the corners; and a control output module for outputting deviation compensation information based on the optical flow change in response to satisfying preset correction conditions, wherein the deviation compensation information controls the drive device to swing the camera so that the hook remains within the field of view of the camera.

[0017] A third aspect of this application provides a tower crane system, which includes a tower crane body, a camera, and a drive device. The tower crane body has a boom and a hook. When the boom moves relative to the tower crane body, the camera swings synchronously and aligns with the hook. The control device includes: a processor for implementing one or more instructions; and a memory for storing multiple instructions, which are adapted to be loaded by the processor and executed as in the first aspect.

[0018] The tower crane monitoring and correction method, device, and system provided in this application acquire first and second target images in real time from the data stream of the shooting device during hook operation. By calculating the change in optical flow at corner points in the first and second target images, deviation compensation information is generated when the change in optical flow meets the correction conditions. The deviation compensation information is converted into a control signal to drive the hydraulic rod, ultimately automatically adjusting the pitch angle of the shooting device. This enables automatic monitoring of the hook. When the shooting device deviates from its swing trajectory due to a large angle change, the system triggers automatic correction, ensuring the hook remains stably centered in the lens's field of view. This prevents the hook from momentarily leaving the shooting device's field of view and exiting the monitoring frame, improving user experience. Furthermore, it reduces the risk of blurred monitoring image quality caused by the hook exiting the monitoring frame, ensuring the system can maintain automatic tracking of the hook and improving the stability of the hook operation system. Attached Figure Description

[0019] Figure 1 A schematic diagram of the tower crane system provided in this application.

[0020] Figure 2 This is a schematic diagram of the working state of the tower crane system provided in this application.

[0021] Figure 3 This is a schematic diagram of the monitoring screen of the tower crane system provided in this application under normal conditions.

[0022] Figure 4 This is a schematic diagram of the first process of the tower crane monitoring and correction method provided in this application.

[0023] Figure 5 This is a schematic diagram of the second process of the tower crane monitoring and correction method provided in this application.

[0024] Figure 6 This is a schematic diagram of the third process of the tower crane monitoring and correction method provided in this application.

[0025] Figure 7 This is a schematic diagram of the fourth process of the tower crane monitoring and correction method provided in this application.

[0026] Figure 8 This is a schematic diagram of the tower crane monitoring and correction device provided in this application.

[0027] Figure 9 A schematic diagram of the control device provided in this application.

[0028] Explanation of main component symbols 1000. Tower crane system; 100. Tower crane body; 101. Tower boom; 102. Hook; 200. Shooting device; 201. Mounting bracket; 202. Lens; 203. Housing; 204. Suspension bracket; 205. Mounting shaft; 300. Drive unit; 301. Hydraulic rod; 400. Control device; 401. Processor; 402. Memory; 500. Tower crane monitoring and correction device; 501. Data acquisition module; 502. Corner detection module; 503. Optical flow calculation module; 504. Control output module. Detailed Implementation

[0029] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0030] It should be noted that when an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be an element positioned in between. When an element is considered to be "set" on another element, it can be directly set on the other element or there may be an element positioned in between. In this application, unless otherwise expressly specified and limited, the terms "installed," "connected," "linked," "fixed," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two elements. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The terms “comprising” and “equivalent to”, and any variations thereof, in the specification, claims, and foregoing description of the drawings, are intended to cover non-exclusive inclusion.

[0032] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0033] With the development of artificial intelligence and the improvement of safety awareness, the use of AI technology to improve the safety of workers' working environment on construction sites is becoming increasingly common. In intelligent tower crane systems, tower cranes are typically equipped with a camera device, which is rotatably mounted on the tower crane's jib. The crane's hook is located below the camera device, which is used to capture the status of the hook and the working environment below. Users can observe the real-time status of the hook and the working environment through a display terminal.

[0034] The camera is typically mounted on the tower arm via a mounting bracket, and the camera and the mounting bracket are rotatably connected via a mounting shaft. A hydraulic rod is installed between the camera and the mounting bracket, with one end of the hydraulic rod rotatably connected to the camera and the other end rotatably connected to the mounting bracket.

[0035] As the tower jib moves relative to the main body of the tower crane, the camera mounted on it also moves. Due to its own weight, the camera moves relative to the mounting frame, allowing it to swing synchronously and remain aligned with the hook for shooting. During this process, the hydraulic rod acts as a damper, stretching and contracting to prevent instability caused by excessive relative movement or lens box swaying due to external wind speeds, ensuring a smooth and slow swing of the camera. Furthermore, the hydraulic rod can also be used as a drive power source; operators can manually control its stretching and contraction to swing the camera and adjust its shooting angle.

[0036] However, due to prolonged use and performance degradation, during the swinging process of the camera device, when the device's own weight or the extension / retraction of the hydraulic rod reaches a certain critical value, the hydraulic rod can no longer stably provide support. This causes a sudden, large-angle change in the camera device, deviating from the system's intended swing trajectory. Consequently, the hook may momentarily detach from the camera's field of view and disappear from the monitoring screen. At this point, the operator needs to manually adjust the camera's angle and retrieve the hook, impacting the user experience.

[0037] Furthermore, the image quality of the monitoring screen may become blurry the moment the hook moves out of the screen, and the current tracking method will fail to automatically track the hook after it moves out of the screen because there is no target. The auxiliary hardware detection solution will also bring additional costs.

[0038] Therefore, this application provides a tower crane monitoring and correction method, device, and tower crane system, which has the effect of automatically correcting the shooting angle and avoiding the loss of hook tracking.

[0039] Figure 1 A schematic diagram of the tower crane system provided in this application. Figure 2 This is a schematic diagram of the working state of the tower crane system provided in this application. Figure 3 This is a schematic diagram of the monitoring screen of the tower crane system provided in this application under normal conditions.

[0040] like Figures 1 to 3 As shown, this application first provides a tower crane monitoring and correction method, which can be applied to a tower crane system 1000. The tower crane system 1000 includes a tower crane body 100, a camera 200, and a drive unit 300. The tower crane body 100 has a tower arm 101, a hook 102, and a control unit 400. The hook 102 is connected to the tower arm 101. The camera 200 and the drive unit 300 are both located on the tower arm 101, with the hook 102 positioned below the camera 200. When the tower arm 101 moves relative to the tower crane body 100, the camera 200 swings synchronously to ensure that it is always aligned with the hook 102 for imaging. The images captured by the camera 200 can be displayed on the monitoring screen of a display terminal. Both the camera 200 and the drive unit 300 are electrically connected to the control unit 400, and the tower crane monitoring and correction method provided in this application can be executed by the control unit 400.

[0041] For example, the imaging device 200 includes a mounting frame 201, a lens 202, and a housing 203. The housing 203 is mounted to the tower arm 101 via the mounting frame 201. A mounting shaft 205 is provided on the upper part of the housing 203, parallel to the horizontal direction. The mounting shaft 205 is fixedly connected to the housing 203 and rotatably connected to the mounting frame 201. The lens 202 is fixed to the lower part of the housing 203, and the imaging end of the lens 202 is exposed at the bottom of the housing 203. The lens 202 can capture images of the status of the hook 102 and the working environment below it through the imaging end. The housing 203 can swing relative to the mounting frame 201 around the mounting shaft 205, and the lens 202 moves synchronously with the housing 203 to adjust the imaging angle of the lens 202. The installation position of the housing 203 can be set according to the position of the hook 102 relative to the shooting field of the lens 202. In the example of this application, the installation position of the housing 203 is configured such that the hook 102 is in the middle position of the shooting field of the lens 202, so that the hook 102 is kept in the central area of ​​the captured image.

[0042] For example, the drive device 300 is a hydraulic rod 301, and the mounting frame 201 is provided with a suspension frame 204. One end of the hydraulic rod 301 is rotatably connected to the mounting shaft 205, and the other end of the hydraulic rod 301 is rotatably connected to the suspension frame 204. The hydraulic rod 301 can function as a damper. When the tower boom 101 moves relative to the tower crane body 100, the housing 203 swings relative to the mounting frame 201 due to its own gravity, while the hydraulic rod 301 stretches / contracts to provide damping and buffering. The hydraulic rod 301 can also function as a drive power source. When it is necessary to adjust the shooting angle of the shooting device 200, the operator sends a control command to the drive device 300 through an interactive terminal to control the hydraulic rod 301 to actively stretch / contract, so that the housing 203 swings relative to the mounting frame 201, adjusting the lens 202 to a suitable shooting angle.

[0043] It should be understood that in actual operation scenarios, operators can control the movement of the boom 101 or the hook 102 to move the hook 102 to a designated position to grab / release goods, or to move the grabbed goods to a designated position. During this process, the imaging device 200 captures real-time images of the hook 102 and displays them on the monitoring screen. Operators can view the implementation status of the hook 102 and the operating environment through the monitoring screen.

[0044] In the example of this application, the display terminal can be a display screen inside the control cabin of the tower crane, or a remote communication smart terminal device. The smart terminal device can be a mobile phone, tablet computer, etc., and this application does not limit this. The interactive terminal can be an operating lever inside the control cabin of the tower crane. The interactive terminal has forward and reverse buttons. The operator can control the hydraulic lever 301 to extend by touching the forward button, and control the hydraulic lever 301 to retract by touching the reverse button. The duration of the touch determines the distance by which the hydraulic lever 301 extends / retracts.

[0045] Figure 4 This is a schematic diagram of the first process of the tower crane monitoring and correction method provided in this application.

[0046] Please refer to the following: Figure 4 In this embodiment, the tower crane monitoring and correction method includes the following steps.

[0047] S401: Acquire the first target image and the second target image.

[0048] The first target image and the second target image are both images captured by the imaging device 200, with the first target image being captured earlier than the second target image. The first target image and the second target image can be selected from the data stream of the imaging device 200 at a set interval.

[0049] For example, the first target image and the second target image are two adjacent frames. This minimizes changes in grayscale values, lighting, and movement of the same object in the first and second target images, reducing introduced errors and improving the accuracy of subsequent analysis. Furthermore, the tower crane operating environment is dynamically changing, requiring the system to detect and correct deviations in real time to prevent the hook 102 from being completely lost. The computational delay for processing adjacent frames in this application is minimal, allowing for the fastest generation of control signals, reducing system latency, and improving the effectiveness of auxiliary deviation correction.

[0050] S402: Based on the first target image, obtain the target corner point set.

[0051] The target corner point set includes multiple corner points, which are used to characterize the region corresponding to the hook 102 in the first target image. Corner points are points in the image that have significant gray-scale changes in a specified direction, and can represent the edges of real-world objects, including but not limited to: outline points or corner points of hook 102, rope, cargo, building materials, or background buildings.

[0052] S403: Based on the target corner point set and the second target image, obtain the optical flow change.

[0053] The optical flow change is used to reflect the motion change of corner points. In this embodiment, the optical flow change represents the motion vector of corner points in the image between the first target image and the second target image. It can accurately quantify the offset and direction of the imaging device 200 at the pixel motion level, providing a decision basis for subsequent automatic correction.

[0054] S404: Determine whether the shooting device 200 meets the preset correction conditions. If yes, proceed to step S405; otherwise, end.

[0055] Among them, the correction condition is used to determine whether the current shooting device 200 needs to perform correction.

[0056] In the example of this application, the correction conditions include a first condition and a second condition. Step S405 is executed only when both the first condition and the second condition are met simultaneously; otherwise, the process ends.

[0057] For example, the first condition is that the change in optical flow is greater than a preset activation threshold. The activation threshold can be set by testing the occurrence of hook 102 loss and the triggering of correction actions.

[0058] When the change in optical flow exceeds the activation threshold, it can be assumed that the cumulative offset of the imaging device 200 has reached a level that could easily cause the hook 102 to be lost, and the system needs to actively intervene to correct the shooting angle of the imaging device 200. Conversely, when the change in optical flow is less than or equal to the activation threshold, it can be assumed that the cumulative offset of the imaging device 200 has not reached the corresponding level, and the correction of the shooting angle of the imaging device 200 is not initiated temporarily.

[0059] For example, the second condition is: during the capture time of the first target image and the second target image, the driving device 300 does not drive the shooting device 200 to swing under the control of the interactive terminal. Specifically, when the operator actively stretches / contracts the hydraulic rod 301 via the interactive terminal to adjust the shooting angle of the lens 202, the correction of the shooting angle of the shooting device 200 is temporarily not initiated to avoid interfering with the operator's normal debugging operations.

[0060] Understandably, by using the correction condition as the trigger threshold for correction control, a balance can be achieved between the sensitivity and stability of the system, avoiding frequent activation of the correction action of the shooting device 200, which would cause the shooting device 200 to make constant fine adjustments or even vibrate, making the image more unstable and seriously affecting the user experience and system lifespan.

[0061] S405: Output deviation compensation information based on optical flow variation.

[0062] Specifically, in response to meeting preset correction conditions, deviation compensation information is output based on the change in optical flow. The deviation compensation information is used to control the drive device 300 to swing the shooting device 200 so that the hook 102 remains within the shooting field of view of the shooting device 200.

[0063] Understandably, during the operation of the hook 102, the first target image and the second target image are acquired in real time from the data stream of the shooting device 200. By calculating the change in optical flow at the corner points in the first target image and the second target image, deviation compensation information is generated when the change in optical flow meets the correction condition. The deviation compensation information is converted into a control signal to drive the hydraulic rod 301, and finally automatically adjusts the pitch angle of the shooting device 200.

[0064] In this way, automatic monitoring of the hook 102 can be achieved. When the shooting device 200 deviates from the swing trajectory due to a large angle change, the system triggers automatic correction to keep the hook 102 always stably in the center of the shooting field of the lens 202. This prevents the hook 102 from momentarily leaving the shooting field of the shooting device 200 and running out of the monitoring screen, thus improving the user experience. In addition, it can reduce the risk of blurry monitoring image quality caused by the hook 102 running out of the monitoring screen, ensuring that the system can maintain automatic tracking of the hook 102 and improving the stability of the hook 102 operation system.

[0065] Figure 5 This is a schematic diagram of the second process of the tower crane monitoring and correction method provided in this application.

[0066] Please refer to the following: Figure 5 In some embodiments, step S402 includes: S501: Perform corner detection on the first target image to obtain an initial corner set, and select multiple original corner clusters from the initial corner set.

[0067] In this process, the gradient values ​​Ix in the x direction and Iy in the y direction of the first target image are calculated, and the autocorrelation matrix M is constructed using formula (1). The eigenvalues ​​λ1 and λ2 of matrix M are calculated, and the pixels whose eigenvalues ​​λ1 and λ2 are both greater than a specified threshold are taken as corner points.

[0068] (1) Where w(x, y) is the Gaussian window function.

[0069] S502: Select the regional corner cluster from multiple original corner clusters.

[0070] This process involves a secondary selection of corner points based on the initial set. A density-based clustering algorithm is used to cluster the corner points, grouping spatially adjacent corner points into clusters of the same region. Each cluster represents a potential, independent object or target region with a regular geometric structure.

[0071] In this application, the regional corner point cluster includes a spatial regional corner point cluster that satisfies the spatial corner point condition and a background target corner point cluster that satisfies the hook corner point condition.

[0072] The spatial region corner point cluster is used to characterize the region corresponding to the activity space of the hook 102 in the first target image. In this application, under normal circumstances, the hook 102 is held in the central region of the captured image, that is, the activity space of the hook 102 is located in the central region of the first target image.

[0073] Background target corner point clusters are used to characterize the region corresponding to the contour of hook 102 in the first target image. In this application, the background target corner point clusters include multiple corner points distributed along the contour of hook 102.

[0074] S503: Combine the spatial region corner point cluster and the background target corner point cluster as the target corner point set.

[0075] The target corner point set includes spatial region corner point clusters and background target corner point clusters.

[0076] It is understandable that the background motion trend of the entire central area captured by the shooting device 200 can be perceived through the spatial area corner point cluster, and the motion of the hook 102 itself can be directly and accurately tracked through the background target corner point cluster. The spatial area corner point cluster and the background target corner point cluster are combined to obtain the target corner point set, which serves as the basis for inferring the offset of the shooting device 200.

[0077] Specifically, step S502 includes: S5021: Select original corner point clusters that meet the spatial corner point conditions from multiple original corner point clusters and use them as spatial region corner point clusters.

[0078] Among them, the spatial corner point condition includes: the candidate distance between the candidate region formed by the original corner point cluster and the specified region of the first target image is less than a preset distance threshold.

[0079] Specifically, the Euclidean distance between the geometric center of the region formed by each original corner point cluster and the physical center of the first target image is calculated as a candidate distance. Regions formed by original corner point clusters with candidate distances less than the distance threshold are selected as candidate regions. The candidate regions are determined as the central spatial region of the imaging device 200, thus obtaining the spatial region corner point cluster.

[0080] S5022: Select the original corner point clusters that meet the hook corner point conditions from multiple original corner point clusters and use them as background target corner point clusters.

[0081] The hook corner point conditions include at least one of the following: proportional filtering conditions, included angle filtering conditions, and graphic filtering conditions. That is, when the original corner point cluster satisfies at least one of the following filtering conditions, included angle filtering conditions, and graphic filtering conditions, the original corner point cluster is determined as the background target corner point cluster.

[0082] Specifically, the ratio filtering condition is that the length and width of the smallest bounding rectangle of the candidate region formed by the original corner point cluster are within a preset ratio range. The hook 102 is typically rectangular from the perspective of the shooting device 200. The ratio filtering condition can filter out irrelevant objects (such as bolts, distant pedestrians, buildings) that are too small or too large, or other irrelevant objects of other shapes (such as square bricks, round tires), by using the inherent geometric characteristic of the hook 102 as a narrow rectangle. This focuses the search on candidate targets of reasonable size, improving processing efficiency. For example, the ratio range can be set to a minimum bounding rectangle aspect ratio ≥ 10.

[0083] Specifically, the angle selection criterion is that the angle formed by the candidate regions created by the original corner point clusters is within a preset angle range. The two ends of the hook 102 are connected to the ends of the tower arm 101 via suspension ropes, and the two ropes form a certain angle from the perspective of the imaging device 200. The angle selection criterion utilizes the spatial distribution of corner points caused by the structure of the suspension ropes of the hook 102 to filter out irrelevant objects without nearby suspension ropes, and selects the candidate regions of the original corner point clusters with a specific angle distribution as the hook 102. For example, the angle range can be [60°, 120°].

[0084] Specifically, the graphic selection criteria are as follows: the graphic features of the candidate region formed by the original corner point clusters match the preset hook 102 template. The hook 102 is typically made of metal, and its surface may have specific anti-rust coatings (such as yellow or red paint) or wear marks, which form its unique appearance characteristics. By comparing the graphic features (such as color and texture features) of the candidate region with the features of the pre-stored hook 102 template, the original corner point clusters with a similarity higher than the preset value are identified as background target corner point clusters. The graphic selection criteria utilize the appearance characteristics of the hook 102 surface—material, coating, and visual texture—to improve the accuracy of recognition.

[0085] Understandably, this application filters background target corner point clusters through various hook corner point conditions to ensure that candidate targets for hook 102 can be found as much as possible under different conditions (such as partial occlusion, changes in light, and different angles). It establishes a multi-dimensional and complementary filtering strategy to ensure that the system can stably and accurately lock and track hook 102 in complex and ever-changing construction site environments.

[0086] It is worth noting that in other embodiments, the specific rules for the hook corner condition can be adjusted according to actual needs. For example, the hook corner condition can be at least two of the following: proportional screening condition, included angle screening condition, and graphic screening condition. This application does not impose any restrictions on this.

[0087] Figure 6 This is a schematic diagram of the third process of the tower crane monitoring and correction method provided in this application.

[0088] Please refer to the following: Figure 6 In some embodiments, the tower crane monitoring and correction method further includes: S601: Input the first target image into the preset blurred corner detection model to obtain the effective corner cluster, and execute step S503.

[0089] Among them, the blurred corner detection model is based on the distribution relationship between the corners of the blurred image and the corners of the clear image. The blurred corner detection model is used to predict and extract corners from the blurred image.

[0090] For example, the training method for the blurred corner detection model can be as follows: A dataset of clear images with abundant corner points is collected. This dataset includes clear images from various scenes (such as hook 102, rebar, indoor scenes, architectural landscapes, etc.). Then, simulating common blur types encountered in actual photography, various blur kernels are generated, including but not limited to the hook 102 swaying blur kernel, Gaussian blur kernel, and mixed blur kernel. Next, various blur kernels are used to perform convolution operations on the clear images to create various blur effects, resulting in blurred images and generating blurred-clear image pairs. Multiple corner points can be labeled on the clear images using a corner detection algorithm. Finally, a convolutional neural network is used to train the collected blurred-clear image pairs and the labeled corner points, training a blurred corner detection model capable of detecting blurred corner points.

[0091] The convolutional neural network includes a multi-scale feature fusion encoder and an SE (Squeeze-and-Excitation) attention module. Multi-scale feature fusion can capture both local details and global contextual information of an image, while the SE module allows the network to adaptively emphasize information-rich feature channels, suppress useless information, and improve feature representation capabilities.

[0092] Correspondingly, in step S503, the spatial region corner point cluster, the background target corner point cluster, and the effective corner point cluster are considered as the target corner point set. That is, the target corner point set includes the spatial region corner point cluster, the background target corner point cluster, and the effective corner point cluster.

[0093] Understandably, the blurred corner detection model provided in this application learns the mapping relationship between the corners of blurred images and the corners of clear images through data simulation and supervised learning. When the clarity of the first target image is poor, it can compensate and generate effective and reliable corners as effective corner clusters, providing key data compensation for downstream optical flow calculation tasks. By superimposing the compensation of effective features, the error caused by motion blur is reduced, and the robustness and system reliability are improved.

[0094] In some embodiments, step S601 is performed in response to the first target image satisfying a preset image blur condition.

[0095] Specifically, the tower crane monitoring and correction method also includes: determining whether the first target image meets the preset image blur condition; if yes, then step S601 is executed; if no, then step S503 is executed.

[0096] In some cases, when the imaging device 200 itself shakes or the operator actively adjusts the shooting angle of the imaging device 200, the image captured by the imaging device 200 may become blurry, resulting in insufficient accuracy in corner detection and recognition, and affecting the execution effect of subsequent correction actions. The image blur condition is used to determine whether the blur level of the first target image reaches a level that requires intervention and adjustment.

[0097] For example, the image blur condition includes: the blur degree of the first target image is greater than a preset blur threshold.

[0098] Specifically, a blur detection algorithm is used to detect the blur degree of the first target image. When the blur degree of the first target image exceeds a preset blur threshold, the first target image is determined to meet the image blur condition. The blur detection algorithm can be a blur detection method based on Laplacian variance, a blur detection method based on Fourier transform analysis, a blur detection method based on deep learning, etc., and this application does not impose any restrictions on it.

[0099] Figure 7 This is a schematic diagram of the fourth process of the tower crane monitoring and correction method provided in this application.

[0100] Please refer to the following: Figure 7 In some embodiments, step S403 includes: S701: Based on the grayscale image of the second target image, solve for the motion vector of each corner point in the target corner point set to obtain the optical flow of each corner point, so as to obtain the optical flow field corresponding to the spatial region corner point cluster, the background target corner point cluster and the effective corner point cluster.

[0101] Specifically, the optical flow of each corner point is calculated using the LK (Lucas-Kanade) sparse optical flow algorithm based on the spatial region corner point cluster, background target corner point cluster, effective corner point cluster, and grayscale image of the second target image.

[0102] Specifically, the LK sparse optical flow algorithm is based on two core assumptions: constant brightness and minute motion. Constant brightness means that the grayscale value of the same corner point remains unchanged in different frames (the first target image and the second target image). Minor motion means that the displacement of the corner point between different frames (the first target image and the second target image) is very small.

[0103] In this application, for each corner point, the LK sparse optical flow algorithm calculates the displacement vector (u, v) of the point from the first target image to the second target image by solving the gray-level gradient equation of the surrounding pixel blocks, where u is the horizontal displacement and v is the vertical displacement, and outputs the optical flow fields of the spatial region corner point cluster, the background target corner point cluster, and the effective corner point cluster.

[0104] S702: Based on the weight coefficients corresponding to the spatial region corner point cluster, background target corner point cluster and effective corner point cluster, as well as the influence coefficient of the effective corner point cluster, and combined with the preset change time, the optical flow fields corresponding to the spatial region corner point cluster, background target corner point cluster and effective corner point cluster are weighted and summed to obtain the optical flow change.

[0105] Wherein, the change time is the time difference between the shooting time of the first target image and the shooting time of the second target image, and in this application, it is the interval time between two adjacent frames.

[0106] In this embodiment, when the first target image does not meet the image blur condition, the influence coefficient is set to 0, and the effective corner point cluster is an empty set. When the first target image meets the image blur condition, the influence coefficient is set to 1, and the effective corner point cluster is a non-empty set.

[0107] Specifically, based on the optical flow fields corresponding to the spatial region corner point cluster, background target corner point cluster and effective corner point cluster, the optical flow fields within the change time are weighted and summed using formula (2) to obtain the total optical flow field, which is used as the optical flow change.

[0108] (2) Among them, F T denoted as , where is the optical flow variation. Fbg represents the spatial region corner point cluster. Wbg represents the weighting coefficient of the spatial region corner point cluster. Fhook represents the background target corner point cluster. Whook represents the weighting coefficient of the background target corner point cluster. Fblur represents the effective corner point cluster. Wblur represents the weighting coefficient of the effective corner point cluster. C represents the influence coefficient of the effective corner point cluster.

[0109] Understandably, this application fuses optical flow information from corner points in different regions, adding multimodal information based on the construction site operation background to obtain a more reliable global offset estimate, thereby obtaining a more credible optical flow variation. This application accumulates the optical flow field over a changing time period through temporal fusion, smooths the data within a time window, filters instantaneous noise, and enhances the effective signal.

[0110] By using spatial domain weighting, different weights are assigned to the optical flow fields of corner point clusters in different regions. For example, the weight coefficient of the background target corner point cluster can be configured to a higher weight to ensure that when the hook 102 is clearly visible, the system's decision-making is entirely dominated by the most direct and reliable corner point of the hook 102, thereby generating the most accurate correction action. The weight coefficients of the spatial region corner point cluster and the effective corner point cluster can be configured to a lower weight to provide important auxiliary and reference information, playing a role in verification and supplementation, thus achieving accurate, reliable, and adaptive offset sensing and control.

[0111] In some embodiments, the specific implementation of step S405 is as follows: by using formula (3), the average value of the optical flow change is calculated using the mapping operation function to obtain the offset vector, and then the projection size of the vector in the vertical direction is mapped to the size of the vertical offset through the mapping function fun to obtain the deviation compensation value. The deviation compensation value can be converted into a control signal to drive the hydraulic rod 301, and finally automatically adjust the pitch angle of the shooting device 200.

[0112] (3) Where O is the deviation compensation value, F T F is the change in optical flow. c Oc is the preset optical flow constant field, and Oc is the operation compensation value obtained by fitting data.

[0113] In the example of this application, the magnitude of the deviation compensation value O represents the specific value of the correction amount, the distance by which the hydraulic rod 301 is stretched / contracted. In the interactive terminal, the value of the deviation compensation value O can be mapped to the time when the forward / backward button is touched. The sign of the deviation compensation value O represents the direction of the correction movement, which drives the hydraulic rod 301 to stretch or contract. In the interactive terminal, the sign of the deviation compensation value O can be mapped to touching the forward button or touching the backward button.

[0114] In another embodiment, the tower crane monitoring and correction method further includes: S801: Set the value of the influence coefficient based on the blur of the first target image.

[0115] The blurriness of the first target image and the value of the influence coefficient can be directly proportional. That is, the greater the blurriness of the first target image, the higher the value of the influence coefficient. The influence coefficient can be in the range of [0,1].

[0116] Understandably, since the effective corner cluster is a set of corners predicted from a blurred image by the blurred corner detection model, its optical flow information has a certain degree of uncertainty. By dynamically adjusting the value of the influence coefficient, the contribution of the effective corner cluster to the calculation of optical flow changes can be adjusted under different degrees of blur. In this way, the effective corner cluster can be used cautiously as compensation information. While providing effective corners as a reference, it reduces the risk of unreliable information excessively interfering with system decision-making, enabling the system to automatically adapt to different operating conditions and cope with various emergencies.

[0117] Figure 8 This is a schematic diagram of the tower crane monitoring and correction device provided in this application.

[0118] like Figure 1 and Figure 8As shown, this application also provides a tower crane monitoring and correction device 500. The tower crane monitoring and correction device 500 is applied to a tower crane system 1000, which includes a tower crane body 100, a tower boom 101, a hook 102, a camera device 200, and a drive device 300. When the tower boom 101 moves relative to the tower crane body 100, the camera device 200 swings synchronously and aligns with the hook 102.

[0119] The tower crane monitoring and correction device 500 includes: The data acquisition module 501 is used to acquire a first target image and a second target image captured by the shooting device 200, wherein the first target image was captured earlier than the second target image.

[0120] The corner detection module 502 is used to obtain a target corner set based on the first target image, wherein the target corner set includes multiple corners, and the multiple corners are used to characterize the region corresponding to the hook 102 in the first target image.

[0121] The optical flow calculation module 503 is used to obtain the optical flow change based on the target corner point set and the second target image, wherein the optical flow change reflects the motion change of the corner points.

[0122] The control output module 504 is used to output deviation compensation information based on the change in optical flow in response to the satisfaction of preset correction conditions. The deviation compensation information is used to control the drive device 300 to drive the shooting device 200 to swing so that the hook 102 is kept within the shooting field of view of the shooting device 200.

[0123] It is understood that the module division described above is a logical functional division, and there may be other division methods in actual implementation. Furthermore, the functional modules in the various embodiments of this application can be integrated into the same processing unit, or each module can exist physically separately, or two or more modules can be integrated into the same unit. The integrated modules described above can be implemented in hardware or in a combination of hardware and software functional modules.

[0124] Figure 9 A schematic diagram of the control device provided in this application.

[0125] like Figure 1 and Figure 9 As shown, this application also provides a tower crane system 1000. The tower crane system 1000 includes a tower crane body 100, a camera 200, and a drive device 300. The tower crane body 100 has a tower boom 101, a hook 102, and a control device 400. When the tower boom 101 moves relative to the tower crane body 100, the camera 200 swings synchronously and aligns with the hook 102.

[0126] The control device 400 includes a processor 401 and a memory 402. The processor 401 is used to implement one or more instructions, and the memory 402 is used to store multiple instructions. The multiple instructions are suitable for being loaded by the processor 401 and executed by the above-mentioned tower crane monitoring and correction method.

[0127] The implementation principle of the tower crane monitoring and correction device 500 and the tower crane system 1000 provided in this application embodiment can be found in the relevant description in the above tower crane monitoring and correction method, and will not be repeated here.

[0128] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments described above should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application.

Claims

1. A method for monitoring and correcting deviations of a tower crane, characterized in that, Applied to a tower crane system, the tower crane system includes a tower crane body, a camera, and a drive device. The tower crane body has a tower arm and a hook. When the tower arm moves relative to the tower crane body, the camera swings synchronously and aligns with the hook. The tower crane monitoring and correction method includes: Acquire a first target image and a second target image captured by the imaging device, wherein the first target image was captured earlier than the second target image. Based on the first target image, a target corner point set is obtained, wherein the target corner point set includes multiple corner points, and the multiple corner points are used to characterize the area corresponding to the hook in the first target image; Based on the target corner point set and the second target image, the optical flow change is obtained, wherein the optical flow change is used to reflect the motion change of the corner points; In response to meeting the preset correction conditions, deviation compensation information is output based on the optical flow change, wherein the deviation compensation information is used to control the driving device to swing the shooting device so that the hook is kept within the shooting field of view of the shooting device.

2. The tower crane monitoring and correction method according to claim 1, characterized in that, The step of obtaining the target corner point set based on the first target image includes: Corner detection is performed on the first target image to obtain an initial corner set, and multiple original corner clusters are selected from the initial corner set; Regional corner point clusters are selected from multiple original corner point clusters, wherein the regional corner point clusters include spatial regional corner point clusters that satisfy spatial corner point conditions and background target corner point clusters that satisfy hook corner point conditions. The spatial regional corner point clusters are used to characterize the region corresponding to the activity space of the hook in the first target image, and the spatial regional corner point clusters are used to characterize the region corresponding to the outline of the hook in the first target image. The spatial region corner point cluster and the background target corner point cluster are used as the target corner point set.

3. The tower crane monitoring and correction method according to claim 2, characterized in that, The spatial corner point conditions include: The candidate distance between the candidate region formed by the original corner cluster and the designated region of the first target image is less than a preset distance threshold.

4. The tower crane monitoring and correction method according to claim 2, characterized in that, The hook corner point conditions include at least one of the following: the length and width of the smallest bounding rectangle of the candidate region formed by the original corner point cluster are within a preset ratio range; the angle of the included angle formed by the candidate region formed by the original corner point cluster is within a preset angle range; and the graphic features of the candidate region formed by the original corner point cluster match the preset hook template.

5. The tower crane monitoring and correction method according to claim 2, characterized in that, The method further includes: The first target image is input into a preset blurred corner detection model to obtain an effective corner cluster, wherein the blurred corner detection model is used to predict and extract corners from the blurred image; The spatial region corner point cluster, the background target corner point cluster, and the effective corner point cluster are used as the target corner point set.

6. The tower crane monitoring and correction method according to claim 5, characterized in that, The step of obtaining the optical flow change based on the target corner point set and the second target image includes: Based on the grayscale image of the second target image, the motion vector of each corner point in the target corner point set is solved to obtain the optical flow of each corner point, so as to obtain the optical flow field corresponding to the spatial region corner point cluster, the background target corner point cluster and the effective corner point cluster; Based on the weight coefficients corresponding to the spatial region corner point cluster, the background target corner point cluster, and the effective corner point cluster, as well as the influence coefficient of the effective corner point cluster, and combined with a preset change time, the optical flow fields corresponding to the spatial region corner point cluster, the background target corner point cluster, and the effective corner point cluster are weighted and summed to obtain the optical flow change.

7. The tower crane monitoring and correction method according to claim 6, characterized in that, The step of inputting the first target image into a preset blurred corner detection model is executed in response to the first target image satisfying a preset image blur condition; If the first target image does not meet the image blur condition, the value of the influence coefficient is set to 0.

8. The tower crane monitoring and correction method according to claim 7, characterized in that, The tower crane monitoring and correction method also includes: Blur detection is performed on the first target image to obtain the blur level of the first target image; The image blurring condition includes: the blurring degree of the first target image is greater than a preset blurring threshold.

9. The tower crane monitoring and correction method according to claim 8, characterized in that, The tower crane monitoring and correction method also includes: The value of the influence coefficient is set based on the blurriness of the first target image.

10. The tower crane monitoring and correction method according to claim 1, characterized in that, The correction condition includes: the change in optical flow is greater than a preset activation threshold.

11. The tower crane monitoring and correction method according to claim 10, characterized in that, The correction condition further includes: during the shooting time of the first target image and the second target image, the driving device does not drive the shooting device to swing under the control of the interactive terminal.

12. A tower crane monitoring and correction device, characterized in that, Applied to a tower crane system, the tower crane system includes a tower crane body, a camera, and a drive device. The tower crane body has a tower arm and a hook. When the tower arm moves relative to the tower crane body, the camera swings synchronously and aligns with the hook. The tower crane monitoring and correction device includes: The data acquisition module is used to acquire a first target image and a second target image captured by the shooting device, wherein the first target image was captured earlier than the second target image. A corner detection module is used to obtain a target corner set based on the first target image, wherein the target corner set includes multiple corners, and the multiple corners are used to characterize the region corresponding to the hook in the first target image; An optical flow calculation module is used to obtain the optical flow change based on the target corner point set and the second target image, wherein the optical flow change reflects the motion change of the corner points; The control output module is used to output deviation compensation information based on the optical flow change in response to the satisfaction of preset correction conditions. The deviation compensation information is used to control the driving device to swing the shooting device so that the hook is kept within the shooting field of view of the shooting device.

13. A tower crane system, characterized in that, It includes a tower crane body, a tower boom, a hook, a camera, a drive device, and a control device, wherein, when the tower boom moves relative to the tower crane body, the camera swings synchronously and aligns with the hook; The control device includes: A processor is used to implement one or more instructions; and, A memory for storing a plurality of instructions adapted to be loaded by a processor and executed as described in any one of claims 1 to 11.

Images