A crane hook position sensing and safety warning method and system

Abstract

The present invention relates to a crane hook position perception and safety warning method and system, the method comprising: collecting crane hook images, tracking the crane hook based on the real-time collected crane hook images and a pre-established crane hook template library; using SLAM technology to construct a crane operation scene map and locate the position of the camera in the map; calculating the distance between the hook and the camera based on the camera imaging principle and the hook tracking result; obtaining the position of the hook in the map based on the camera positioning result and the distance between the hook and the camera, and performing hook collision warning and hook swing monitoring based on the hook position. The present invention achieves accurate perception of the hook position through a monocular camera and hook tracking technology, avoiding the problem of a 2D camera lacking depth information. The hook is tracked based on a pre-established crane hook template library. The accuracy of hook position perception is improved by adaptively initializing the hook template features.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical safety monitoring, and in particular to a crane hook position sensing and safety early warning method and system. Background Art

[0002] Cranes, as large-scale engineering machinery, are widely used in construction, equipment installation, and consumer rescue operations, effectively improving work efficiency. However, crane hoisting is a high-altitude operation. For complex hoisting tasks, such as remote and blind cranes, operators often have difficulty accurately observing or perceiving the hook's position within the work environment due to limited vision. This can lead to incorrect operations and hook collisions, posing a serious safety hazard.

[0003] Currently, existing technologies typically use distance sensors or 2D cameras to monitor the position of the hook. However, distance sensor measurements only provide mechanical values ​​and fail to reflect the hook's swinging state due to inertia during operation or its positional relationship with the surrounding environment. To address this issue, researchers have proposed installing a 2D camera on the crane boom and using aerial images to monitor the hook's position within the operating scene. However, images captured by 2D cameras reflect the positional relationships of objects in two-dimensional space and lack three-dimensional depth information. This hinders operators from quickly executing effective and safe lifting operations. Summary of the Invention

[0004] In view of this, it is necessary to provide a crane hook position perception and safety warning method to solve the above-mentioned defects of the prior art.

[0005] To solve the above problems, in a first aspect, an embodiment of the present invention provides a crane hook position sensing and safety warning method, comprising:

[0006] S1, real-time collection of crane hook images, and crane hook tracking based on the real-time collected crane hook images and a pre-established crane hook template library;

[0007] S2 uses SLAM technology to build a map of the crane operation scene based on real-time collected images and locate the camera position in the map;

[0008] S3, calculates the distance between the hook and the camera based on the camera imaging principle and the hook tracking result;

[0009] S4, based on the camera positioning result and the distance between the hook and the camera, obtains the position of the hook in the map, and performs hook collision warning and hook swing monitoring based on the hook position.

[0010] Preferably, step S1 specifically includes:

[0011] S11, continuous images of the crane hook are captured using a monocular camera;

[0012] S12, constructing a crane hook template library containing a variety of hook images, and using the first several frames of images taken by the monocular camera to adaptively initialize the hook template features;

[0013] S13, tracking the crane hook in subsequent images based on the initialized template features;

[0014] S14, updating the online template queue according to the hook tracking result to eliminate the accumulated error.

[0015] Preferably, in step S12, the adaptive initialization of the hook template features using the first several frames of images taken by the monocular camera specifically includes:

[0016] The first frame image captured by the monocular camera is segmented into multiple regions. For each segmented region, it is matched with a hook template in the crane hook template library and a bounding box confidence score is calculated; the bounding box confidence score represents the similarity between the current region and the hook template.

[0017] Select the three hook templates with the highest bounding box confidence scores and calculate the average confidence of the three hook templates in several subsequent frames of images;

[0018] According to the calculated average confidence, the hook template with the highest average confidence score is selected to calculate the initial template feature T init , initial template feature T init Used to identify and track the hook in subsequent frames.

[0019] Preferably, in step S14, updating the online template queue according to the hook tracking result includes:

[0020] Calculate a confidence score, prediction box and template features for the hook template in the image at each moment;

[0021] Compare the difference between the historical aspect ratio and the current aspect ratio of the hook template in the prediction frame. If it is determined that the ratio of the historical aspect ratio to the current aspect ratio is less than a preset threshold, use an adaptive weight to aggregate the current template and the initial template to obtain an updated online template;

[0022] Using the updated online template and the initial template feature T init to track the position and predicted box of the crane hook in subsequent images.

[0023] Preferably, in step S3, the calculating of the distance between the hook and the camera based on the camera imaging principle and the hook tracking result specifically includes:

[0024] S31, based on the imaging principle of the monocular camera, the expressions for the pixel width x and pixel length y of the hook in the image are obtained:

[0025]

[0026]

[0027] Where, f x and f y The pixel width x and pixel length y represent the focal length parameters corresponding to the camera intrinsic parameter matrix; w and h represent the width and length of the hook in the real world; d is the distance between the camera and the hook;

[0028] S32, calculating the diagonal length L of the hook in the image based on the expressions of the hook's pixel width and pixel length and the crane hook tracking result obtained in step S1;

[0029]

[0030] Substituting the expressions of pixel width x and pixel length y into the above formula, we get the expression of the distance d between the hook and the camera:

[0031]

[0032] S33, since the hook is affected by factors such as swing during operation, the actual width w and actual length h corresponding to the pixel width x and pixel length y of the hook in the image fluctuate; therefore, the above relationship is converted into a power function form to obtain:

[0033] d=k×L α .

[0034] Where k and α are constants;

[0035] The crane hook is lowered from high to low at a constant speed, and the distance d between the hook and the camera and the diagonal length L of the hook in the image are recorded during the descent. By fitting these data points, the values ​​of k and α are obtained.

[0036] Preferably, in step S4, the hook collision warning process includes:

[0037] Get the distance s between each map point and the hook position in the crane operation scene map;

[0038] The map points whose distance s is less than the preset distance warning value are projected into the crane hook image and marked, and the results of the hook collision warning are visualized in the control room.

[0039] Preferably, in step S4, the hook swing monitoring process includes:

[0040] The hook's swing angle and swing distance are calculated based on the hook's position and trajectory in the map. The calculation formula is:

[0041]

[0042]

[0043]

[0044] Where, is the swing angle of the hook in the x-axis direction, is the swing angle of the hook in the y-axis direction, (x', y', z') is the position coordinate of the hook in three-dimensional space, O x and O y represents the coordinate point of the hook in a stationary state, and l represents the swing distance of the hook;

[0045] Visualize the results of hook swing monitoring to the control room;

[0046] The hook swing parameters are used as input signals of the anti-sway system, and the anti-sway parameters of the anti-sway system are adjusted according to the hook swing parameters to realize negative feedback control of the hook anti-sway system; wherein, the anti-sway parameters include the crane hydraulic pressure and the boom rotation speed.

[0047] In a second aspect, an embodiment of the present invention provides a crane hook position sensing and safety warning system, comprising:

[0048] Crane hook tracking module, used to collect crane hook images in real time, and track the crane hook based on the real-time collected crane hook images and a pre-established crane hook template library;

[0049] A map construction module is used to construct a map of the crane operation scene based on real-time collected images using SLAM technology and locate the camera position in the map;

[0050] The hook distance measurement module is used to calculate the distance between the hook and the camera based on the camera imaging principle and the hook tracking results;

[0051] The safety warning module is used to obtain the position of the hook in the map based on the camera positioning results and the distance between the hook and the camera, and to perform hook collision warning and hook swing monitoring based on the hook position.

[0052] In a third aspect, the present invention further provides an electronic device comprising a memory and a processor, wherein:

[0053] The memory is used to store programs;

[0054] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the crane hook position perception and safety warning method as described in the embodiment of the first aspect of the present invention.

[0055] In a fourth aspect, the present invention also provides a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the crane hook position perception and safety warning method as described in the embodiment of the first aspect of the present invention.

[0056] The crane hook position sensing and safety warning method and system provided by the present invention have the following beneficial effects compared with the prior art:

[0057] 1) This invention utilizes a monocular camera and hook tracking technology to achieve real-time, accurate perception of the hook's position, avoiding the lack of depth information associated with 2D cameras. Hook tracking is performed based on a pre-established library of crane hook templates. By adaptively initializing the hook template features, the accuracy of hook position perception is improved.

[0058] 2) The present invention performs hook collision warning and hook swing monitoring based on the position and trajectory of the hook in the map, and visualizes the warning results in the control room, thereby improving the operator's safety awareness and reaction speed.

[0059] 3) The present invention implements negative feedback control of the hook anti-sway system and adjusts the anti-sway parameters according to the hook swing parameters, such as the crane hydraulic pressure and boom rotation speed, to reduce the swing amplitude of the hook and improve the stability and safety of the crane. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A flow chart of the crane hook position sensing and safety warning method provided by the present invention;

[0061] Figure 2 A schematic diagram of the bounding box prediction of the hook template provided by the present invention in a crane hook image;

[0062] Figure 3 A flowchart of updating an online template queue provided by the present invention;

[0063] FIG4( a ) is a schematic diagram showing the spatial position relationship between the hook and the camera provided by the present invention;

[0064] FIG4( b ) is a schematic diagram of a power function provided by the present invention;

[0065] Figure 5 A schematic diagram of a hook collision warning label provided by an embodiment of the present invention;

[0066] Figure 6A schematic diagram of the swing angle and swing distance of the hook provided by the present invention;

[0067] Figure 7 The present invention provides a curve showing the change of the swing angle and swing distance of the hook at different times;

[0068] Figure 8 This is a structural block diagram of the crane hook position sensing and safety warning system provided by the present invention;

[0069] Figure 9 This is a structural block diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0070] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0071] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0072] Currently, existing technologies typically use distance sensors or 2D cameras to monitor the position of the hook. However, distance sensor measurements only provide mechanical values ​​and fail to reflect the hook's swinging state due to inertia during operation or its positional relationship with the surrounding environment. To address this issue, researchers have proposed installing a 2D camera on the crane boom and using aerial images to monitor the hook's position within the operating scene. However, images captured by 2D cameras reflect the positional relationships of objects in two-dimensional space and lack three-dimensional depth information. This hinders operators from quickly executing effective and safe lifting operations.

[0073] In light of this, the present invention provides a crane hook position sensing and safety warning method. Using a monocular camera and hook tracking technology, this method achieves real-time, accurate sensing of the hook's position, avoiding the lack of depth information associated with 2D cameras. Based on the hook's position and trajectory within a map, it provides hook collision warnings and hook swing monitoring. The warning results are visualized in the control room, improving the operator's safety awareness and response speed. This method will be described and illustrated below through multiple examples.

[0074] Figure 1 The flowchart of the crane hook position sensing and safety warning method provided by the present invention. Figure 1As shown, the crane hook position perception and safety warning method includes steps S1 to S4, wherein:

[0075] Step S1: real-time acquisition of crane hook images, and crane hook tracking based on the real-time acquired crane hook images and a pre-established crane hook template library.

[0076] In a preferred embodiment of the present invention, step S1 specifically includes the following steps S11 to S14:

[0077] S11, continuous images of the crane hook are captured using a monocular camera.

[0078] In actual operation, a monocular camera is first mounted on the crane boom, and then used to capture continuous images of the crane hook. In this embodiment, the images captured by the monocular camera are used for crane hook tracking and simultaneous positioning and mapping.

[0079] S12, constructs a crane hook template library containing various hook images, and uses the first several frames of images taken by the monocular camera to adaptively initialize the hook template features.

[0080] Specifically, a template library containing various hook images is pre-built for different operation scenarios. This template library is the basis for subsequent hook recognition and tracking.

[0081] Next, the first frame image captured by the monocular camera is divided into multiple regions. Figure 2 Schematic diagram of the bounding box prediction of the hook template provided by the present invention in the crane hook image, refer to Figure 2 In this example, the first frame of the monocular camera image is segmented into five regions. Each segmented region is matched against a hook template in the crane hook template library and a bounding box confidence score is calculated. The bounding box confidence score (Bounding Box Prediction Score) represents the similarity between the current region and the hook template. This score is typically a value between 0 and 1, with higher values ​​indicating greater similarity.

[0082] Next, the three hook templates with the highest bounding box confidence scores are selected, and the average confidence of each of these three hook templates in the subsequent 2-5 frames is calculated. In this embodiment, confidence, which is output by the tracking model, represents the template's ability to stably track the hook in subsequent frames. The average confidence is the average of the confidence scores across multiple image frames and is used to assess the stability of the hook template's performance across consecutive frames.

[0083] Furthermore, based on the calculated average confidence, the hook template with the highest average confidence score is selected to calculate the initial template feature T init , initial template feature T init For identifying and tracking the hook in subsequent frames, the initial template feature T init The expression is:

[0084]

[0085]

[0086] Where C i,j is the confidence score of the i-th hook template with the highest average confidence score in the j-th frame image, F i,j is the template feature of the i-th hook template in the j-th frame image, P i is the total confidence score of the i-th hook template in the 2-5 frame image.

[0087] This embodiment selects the hook template with the highest average confidence score to calculate the initial template features for subsequent tracking, ensuring that the initial template features can more accurately match the actual photographed hook, thereby improving the accuracy of recognition and tracking.

[0088] S13, tracking the crane hook in subsequent images based on the initialized template features.

[0089] Specifically, based on the template features initialized in step S12, the system begins to track the hook in subsequent images. By comparing the features in the current image with the template features, the system can determine the position of the hook.

[0090] S14, updating the online template queue according to the hook tracking result to eliminate the accumulated error.

[0091] It is understandable that during hook tracking, various factors (such as lighting changes, hook swing, camera shake, etc.) may cause errors in the tracking results. If these errors are not corrected promptly, they will gradually accumulate as tracking progresses, leading to tracking failure or reduced accuracy. To address this issue, the embodiments of the present invention adopt an improved online update strategy, which updates the online template queue based on the results of hook position tracking in the current frame combined with the initial template features.

[0092] In this embodiment, the "online template queue" is a data structure used to store and manage hook templates. During hook tracking, the system continuously updates the templates in this queue to adapt to changes in the hook under different scenarios. This embodiment uses an improved online update strategy to update the hook templates online, eliminating accumulated errors and maintaining tracking accuracy and stability.

[0093] Figure 3 For the update online template queue flow chart provided by the present invention, refer to Figure 3 ,First, calculate a confidence score C for the hook template in the image at each moment t , prediction box B t and template trait T t .

[0094] Then, by comparing the historical aspect ratio R of the hook template in the prediction box t and the current aspect ratio R t , to determine whether the hook has changed significantly. h and the current aspect ratio R t If the ratio of is less than the preset threshold, an adaptive weight β is used to aggregate the current template feature T t and the initial template feature T init , get the updated online template feature T online .

[0095] β=C init / (C t +C init )

[0096] T online =(1-β)T t +βT init

[0097] Where β is the initial template feature T init and the current template feature T t The confidence score (C init ,C t ) determines the adaptive weight.

[0098] Finally, use the updated online template feature T online and the initial template feature T init To track the position and prediction box of the crane hook in subsequent images, the online template can be dynamically adjusted to eliminate the accumulated error and ensure the accuracy and real-time performance of hook tracking.

[0099] This embodiment of the present invention utilizes a monocular camera and hook tracking technology to achieve real-time, accurate perception of the hook's position, avoiding the lack of depth information associated with 2D cameras. Hook tracking is performed based on a pre-established library of crane hook templates. By adaptively initializing the hook template features, the accuracy of hook position perception is improved.

[0100] Step S2: SLAM technology is used to construct a map of the crane operation scene based on the real-time collected images, and the position of the camera is located in the map.

[0101] Specifically, SLAM (Simultaneous Localization and Mapping) is a technology that enables simultaneous localization and mapping in unknown environments. It utilizes environmental information acquired by sensors (such as cameras and lidar) and processes it through algorithms to enable autonomous navigation and positioning of robots. In this example, a monocular camera captures real-time images of a crane operating scene, and SLAM technology is used to construct a 3D map of the crane operating scene.

[0102] The map-building process involves extracting features from the input crane scene image, such as corners, edges, and textures. These feature points form the basis for map construction and camera positioning. By matching feature points with those in the previous image or map, the system determines the relative position between the current image and the map. Based on these matched feature points, a 3D reconstruction algorithm is used to gradually construct a 3D map of the crane scene. This map contains the location information of all objects and obstacles in the scene.

[0103] While building the map, the system also locates the current camera's position on the map in real time. Camera positioning can be achieved by comparing the current image with feature points in the map.

[0104] In this embodiment, by adopting SLAM technology, a 3D map of the crane operation scene is constructed based on real-time collected images, which can obtain comprehensive environmental information of the crane hook, and then realize the position perception of the camera and hook in the map, providing accurate data support for safety warning and anti-sway control.

[0105] Step S3: Calculate the distance between the hook and the camera based on the camera imaging principle and the hook tracking result.

[0106] In a preferred embodiment of the present invention, step S3 may include the following steps S31 to S33;

[0107] S31, based on the imaging principle of the monocular camera, the expressions for the pixel width x and pixel length y of the hook in the image are obtained:

[0108]

[0109]

[0110] Where, f x and f y where x represents the focal length of the hook in the camera's intrinsic parameter matrix, while y represents the width and length of the hook in the real world. d represents the distance between the camera and the hook. Figure 4(a) shows the spatial relationship between the hook and the camera.

[0111] S32, calculating the diagonal length L of the hook in the image based on the expressions of the hook's pixel width and pixel length and the crane hook tracking result obtained in step S1;

[0112]

[0113] Substituting the expressions of pixel width x and pixel length y into the above formula, we get the expression of the distance d between the hook and the camera:

[0114]

[0115] S33, since the hook is affected by factors such as swing during operation, the width x and length y of the hook in the image fluctuate; therefore, the above relationship is converted into a power function form to obtain:

[0116] d=k×L α .

[0117] Wherein, k and α are constants. FIG4(b) is a schematic diagram of the power function provided by the present invention. The hook size in FIG4(b) refers to the diagonal length L of the hook.

[0118] In this example, to obtain accurate hook height, the constants k and α in the power function above need to be calibrated. Specifically, the crane hook is lowered from a high position to a low position at a constant speed. During the descent, the distance d between the hook and the camera and the diagonal length L of the hook in the image are recorded. The values ​​of k and α are obtained by fitting these data points.

[0119] Step S4: Obtain the position of the hook in the map based on the camera positioning result and the distance between the hook and the camera, and perform hook collision warning and hook swing monitoring based on the hook position.

[0120] Specifically, after obtaining the camera's position on the map and the distance between the hook and the camera, the hook's position coordinates (x', y', z') on the map can be calculated through spatial geometric transformation. This process requires the use of the camera's intrinsic parameter matrix (including parameters such as focal length and optical center) and the transformation relationship between the camera and the map (including the rotation matrix R and translation vector T). The calculation formula is:

[0121]

[0122] Where u, v, 1 T is the homogeneous coordinate of the center pixel of the hook in the tracking result, R is the rotation matrix, T is the translation vector, and K is the intrinsic parameter matrix of the camera.

[0123] Furthermore, hook collision warning and hook swing monitoring are performed according to the hook position. In a preferred embodiment of the present invention, the hook collision warning process includes:

[0124] First, the distance s between each map point in the crane operation scene map and the hook position is obtained. This step can obtain the distance between the hook position coordinates and obstacles or potential collision points in the map.

[0125] The calculated distance s is then compared with a preset distance warning value. If s is less than the distance warning value, a collision risk is considered present. Map points where the distance s is less than the preset distance warning value are projected onto the crane hook image and highlighted (e.g., as red dots). The hook collision warning results are visualized in the control room, allowing operators in the control room to intuitively identify potential collision risk areas. Figure 5 A schematic diagram of the hook collision warning marking provided in an embodiment of the present invention.

[0126] In a preferred embodiment of the present invention, the hook swing monitoring process includes:

[0127] The hook's swing angle and swing distance are calculated based on the hook's position and trajectory in the map. In this embodiment, the hook's spatial coordinates at different times can be obtained based on the hook's position and motion trajectory in the 3D map; the hook's swing angle and swing distance are calculated using the hook's spatial coordinates at different times. Figure 6 This is a schematic diagram of the swing angle and swing distance of the hook provided by the present invention.

[0128] The calculation formula for the swing angle and swing distance of the crane hook is:

[0129]

[0130]

[0131]

[0132] Where, is the swing angle of the hook in the x-axis direction, is the swing angle of the hook in the y-axis direction, (x', y', z') is the position coordinate of the hook in three-dimensional space, O x and O y represents the coordinate point of the hook in a stationary state, and l represents the swing distance of the hook.

[0133] After calculating the swing parameters of the crane hook, the swing angle and swing distance of the hook are displayed graphically on the display screen in the control room, so that the operator can intuitively understand the swing of the hook. Figure 7The present invention provides a curve showing the change of the hook's swing angle and swing distance at different times. Researchers can analyze the hook's swing state based on the hook's swing parameters.

[0134] In a preferred embodiment of the present invention, after the hook swing monitoring is completed, negative feedback control can be performed on the hook anti-swing system according to the hook swing parameters obtained by monitoring.

[0135] Specifically, the hook's swing parameters can be used as input signals for the anti-sway system, and the anti-sway parameters of the anti-sway system can be adjusted according to the hook's swing parameters to implement negative feedback control of the hook's anti-sway system; wherein the anti-sway parameters include the crane's hydraulic pressure and the boom's rotation speed. Here, negative feedback control serves as a control strategy for the anti-sway system, and adjusts the system's input (such as the crane's hydraulic pressure, boom's rotation speed, etc.) according to the system's output (the hook's swing parameters) to achieve the purpose of reducing errors or maintaining stability. In this embodiment, by adjusting the anti-sway parameters, the anti-sway system can correct the hook's swing in real time, thereby reducing the hook's swing amplitude and improving the stability and safety of the crane.

[0136] Figure 8 The structural diagram of the crane hook position sensing and safety warning system provided by the present invention is shown in FIG. Figure 8 The crane hook position sensing and safety warning system 800 includes:

[0137] The crane hook tracking module 801 is used to collect crane hook images in real time and track the crane hook based on the real-time collected crane hook images and a pre-established crane hook template library;

[0138] A map construction module 802 is used to construct a map of the crane operation scene based on the real-time collected images using SLAM technology and locate the position of the camera in the map;

[0139] The hook distance measurement module 803 is used to calculate the distance between the hook and the camera based on the camera imaging principle and the hook tracking result;

[0140] The safety warning module 804 is used to obtain the position of the hook in the map based on the camera positioning result and the distance between the hook and the camera, and perform hook collision warning and hook swing monitoring based on the hook position.

[0141] The crane hook position sensing and safety warning system provided by the present invention executes the crane hook position sensing and safety warning methods provided by the aforementioned embodiments through the above-mentioned modules. The crane hook position sensing and safety warning methods have been described in detail in the above-mentioned embodiments, and will not be repeated here in this embodiment.

[0142] The crane hook position sensing and safety warning method and system provided by the present invention have the following beneficial effects compared with the prior art:

[0143] 1) This invention utilizes a monocular camera and hook tracking technology to achieve real-time, accurate perception of the hook's position, avoiding the lack of depth information associated with 2D cameras. Hook tracking is performed based on a pre-established library of crane hook templates. By adaptively initializing the hook template features, the accuracy of hook position perception is improved.

[0144] 2) The present invention performs hook collision warning and hook swing monitoring based on the position and trajectory of the hook in the map, and visualizes the warning results in the control room, thereby improving the operator's safety awareness and reaction speed.

[0145] 3) The present invention implements negative feedback control of the hook anti-sway system and adjusts the anti-sway parameters according to the hook swing parameters, such as the crane hydraulic pressure and boom rotation speed, to reduce the swing amplitude of the hook and improve the stability and safety of the crane.

[0146] Figure 9 The structural block diagram of the electronic device provided by the present invention is as follows: Figure 9 As shown, the present invention further provides an electronic device, wherein the electronic device 900 can be a computing device such as a mobile terminal, a desktop computer, a notebook computer, a PDA, or a server. The electronic device 900 includes a processor 901 and a memory 902, wherein the memory 902 stores a crane hook position sensing and safety warning program 909.

[0147] In some embodiments, the memory 902 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 902 may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Furthermore, the memory 902 may also include both an internal storage unit of the computer device and an external storage device. The memory 902 is used to store application software and various types of data installed on the computer device, such as program codes for installing the computer device. The memory 902 may also be used to temporarily store data that has been output or is to be output. In one embodiment, when the crane hook position perception and safety warning program 909 is executed by the processor 901, the following steps are implemented:

[0148] S1, real-time collection of crane hook images, and crane hook tracking based on the real-time collected crane hook images and a pre-established crane hook template library;

[0149] S2 uses SLAM technology to build a map of the crane operation scene based on real-time collected images and locate the camera position in the map;

[0150] S3, calculates the distance between the hook and the camera based on the camera imaging principle and the hook tracking result;

[0151] S4, based on the camera positioning result and the distance between the hook and the camera, obtains the position of the hook in the map, and performs hook collision warning and hook swing monitoring based on the hook position.

[0152] In some embodiments, the processor 901 can be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program codes or process data stored in the memory 902, such as executing crane hook position sensing and safety warning programs.

[0153] This embodiment further provides a computer-readable storage medium storing a crane hook position sensing and safety warning program. When the crane hook position sensing and safety warning program is executed by a processor, the following steps are implemented:

[0154] S1, real-time collection of crane hook images, and crane hook tracking based on the real-time collected crane hook images and a pre-established crane hook template library;

[0155] S2 uses SLAM technology to build a map of the crane operation scene based on real-time collected images and locate the camera position in the map;

[0156] S3, calculates the distance between the hook and the camera based on the camera imaging principle and the hook tracking result;

[0157] S4, based on the camera positioning result and the distance between the hook and the camera, obtains the position of the hook in the map, and performs hook collision warning and hook swing monitoring based on the hook position.

[0158] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A crane hook position sensing and safety warning method, characterized in that: include: S1, real-time collection of crane hook images, and crane hook tracking based on the real-time collected crane hook images and a pre-established crane hook template library; S2 uses SLAM technology to build a map of the crane operation scene based on real-time collected images and locate the camera position in the map; S3, calculates the distance between the hook and the camera based on the camera imaging principle and the hook tracking result; S4, based on the camera positioning result and the distance between the hook and the camera, the position of the hook in the map is obtained, and hook collision warning and hook swing monitoring are performed according to the hook position; Step S1 specifically includes: S11, continuous images of the crane hook are captured using a monocular camera; S12, constructing a crane hook template library containing a variety of hook images, and using the first several frames of images taken by the monocular camera to adaptively initialize the hook template features; S13, tracking the crane hook in subsequent images based on the initialized template features; S14, updating the online template queue according to the hook tracking result to eliminate the accumulated error; In step S12, the adaptive initialization of the hook template features using the first several frames of images taken by the monocular camera specifically includes: The first frame image captured by the monocular camera is segmented into multiple regions. For each segmented region, it is matched with a hook template in the crane hook template library and a bounding box confidence score is calculated; wherein the bounding box confidence score represents the similarity between the current region and the hook template; Select the three hook templates with the highest bounding box confidence scores and calculate the average confidence of the three hook templates in several subsequent frames of images; According to the calculated average confidence, the hook template with the highest average confidence score is selected to calculate the initial template feature T init , initial template feature T init Used to identify and track the hook in subsequent frames; Initial template feature T init The expression is: Where C i,j is the confidence score of the i-th hook template with the highest average confidence score in the j-th frame image, F i,j is the template feature of the i-th hook template in the j-th frame image, P i is the total confidence score of the i-th hook template in the 2-5 frame image.

2. The crane hook position sensing and safety warning method according to claim 1 is characterized in that: In step S14, updating the online template queue according to the hook tracking result includes: Calculate a confidence score, prediction box and template features for the hook template in the image at each moment; Compare the difference between the historical aspect ratio and the current aspect ratio of the hook template in the prediction frame. If it is determined that the ratio of the historical aspect ratio to the current aspect ratio is less than a preset threshold, use an adaptive weight to aggregate the current template and the initial template to obtain an updated online template; Using the updated online template and the initial template feature T init to track the position and predicted box of the crane hook in subsequent images.

3. The crane hook position sensing and safety warning method according to claim 1 is characterized in that: In step S3, the distance between the hook and the camera is calculated based on the camera imaging principle and the hook tracking result, which specifically includes: S31, based on the imaging principle of the monocular camera, the expressions for the pixel width x and pixel length y of the hook in the image are obtained: Where, f x and f y The pixel width x and pixel length y represent the focal length parameters corresponding to the camera intrinsic parameter matrix; w and h represent the width and length of the hook in the real world; d is the distance between the camera and the hook; S32, calculating the diagonal length L of the hook in the image based on the expressions of the hook's pixel width and pixel length and the crane hook tracking result obtained in step S1; Substituting the expressions of pixel width x and pixel length y into the above formula, we get the expression of the distance d between the hook and the camera: S33: Since the hook is affected by the swinging factor during operation, the actual width w and length h corresponding to the pixel width x and pixel length y of the hook in the image fluctuate. Therefore, the expression of the distance d between the hook and the camera is converted into a power function form, and the result is: d=k×L α Where k and α are constants; The crane hook is lowered from high to low at a constant speed, and the distance d between the hook and the camera and the diagonal length L of the hook in the image are recorded during the descent. By fitting these data points, the values ​​of k and α are obtained.

4. The crane hook position sensing and safety warning method according to claim 1, characterized in that: In step S4, the hook collision warning process includes: Get the distance s between each map point and the hook position in the crane operation scene map; The map points whose distance s is less than the preset distance warning value are projected into the crane hook image and marked, and the results of the hook collision warning are visualized in the control room.

5. The crane hook position sensing and safety warning method according to claim 1, characterized in that: In step S4, the hook swing monitoring process includes: The hook's swing angle and swing distance are calculated based on the hook's position and trajectory in the map. The calculation formula is: Where, is the swing angle of the hook in the x-axis direction, is the swing angle of the hook in the y-axis direction, (x', y', z') is the position coordinate of the hook in three-dimensional space, O x and O y represents the coordinate point of the hook in a stationary state, and l represents the swing distance of the hook; Visualize the results of hook swing monitoring to the control room; The hook swing parameters are used as input signals of the anti-sway system, and the anti-sway parameters of the anti-sway system are adjusted according to the hook swing parameters to realize negative feedback control of the hook anti-sway system; wherein, the anti-sway parameters include the crane hydraulic pressure and the boom rotation speed.

6. A crane hook position sensing and safety warning system, characterized in that: include: Crane hook tracking module, used to collect crane hook images in real time, and track the crane hook based on the real-time collected crane hook images and a pre-established crane hook template library; Specifically include: Use a monocular camera to capture continuous images of the crane hook; Build a crane hook template library containing various hook images, and use the first few frames of images taken by a monocular camera to adaptively initialize the hook template features; Track the crane hook in subsequent images based on the initialized template features; Update the online template queue based on the hook tracking results to eliminate the accumulated errors; The adaptive initialization hook template features of the first several frames of images taken by the monocular camera specifically include: The first frame image captured by the monocular camera is segmented into multiple regions. For each segmented region, it is matched with a hook template in the crane hook template library and a bounding box confidence score is calculated; wherein the bounding box confidence score represents the similarity between the current region and the hook template; Select the three hook templates with the highest bounding box confidence scores and calculate the average confidence of the three hook templates in several subsequent frames of images; According to the calculated average confidence, the hook template with the highest average confidence score is selected to calculate the initial template feature T init , initial template feature T init Used to identify and track the hook in subsequent frames; Initial template feature T init The expression is: Where C i,j is the confidence score of the i-th hook template with the highest average confidence score in the j-th frame image, F i,j is the template feature of the i-th hook template in the j-th frame image, P i is the total confidence score of the i-th hook template in the 2-5 frame image; A map construction module is used to construct a map of the crane operation scene based on real-time collected images using SLAM technology and locate the camera position in the map; The hook distance measurement module is used to calculate the distance between the hook and the camera based on the camera imaging principle and the hook tracking results; The safety warning module is used to obtain the position of the hook in the map based on the camera positioning results and the distance between the hook and the camera, and to perform hook collision warning and hook swing monitoring based on the hook position.

7. An electronic device, It is characterized by: comprising a memory and a processor, wherein, The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the crane hook position perception and safety warning method as described in any one of claims 1 to 5 above.

8. A computer-readable storage medium, characterized in that Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps of the crane hook position perception and safety warning method as described in any one of claims 1 to 5 above.

Images