Crane hook position sensing and safety early warning method and system

Abstract

The invention relates to a crane hook position sensing and safety early warning method and system, and the method comprises the steps: collecting an image of a crane hook, and carrying out the tracking of the crane hook based on the real-time collected image of the crane hook and a pre-established crane hook template library; constructing a crane operation scene map by adopting an SLAM (Simultaneous Localization and Mapping) technology, and positioning the position of a camera in the map; calculating the distance between the lifting hook and the camera according to a camera imaging principle and a lifting hook tracking result; and obtaining the position of the lifting hook in the map according to the camera positioning result and the distance between the lifting hook and the camera, and carrying out lifting hook collision early warning and lifting hook swinging monitoring according to the lifting hook position. Through the monocular camera and the lifting hook tracking technology, the position of the lifting hook is accurately sensed, and the problem that a 2D camera lacks depth information is solved. And tracking the lifting hook based on a pre-established crane lifting hook template library. And the accuracy of sensing the position of the lifting hook is improved by self-adaptively initializing the characteristics of the lifting hook template.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical safety monitoring, and in particular to a crane hook position sensing and safety early warning method and system. Background Art

[0002] As a large-scale engineering machinery, cranes are widely used in construction, equipment installation and consumer rescue, which can effectively improve work efficiency. However, crane hoisting is an aerial operation. For some complex hoisting tasks, such as mountain cranes and blind cranes, operators are limited by their field of vision and usually find it difficult to accurately observe or perceive the position of the hook in the working environment, which leads to incorrect operations and hook collisions, posing serious safety hazards.

[0003] At present, existing technologies usually use distance sensors or 2D cameras to monitor the position of the hook, but the measurement results of the distance sensor can only provide mechanical values, and cannot reflect the swing state of the hook due to the inertia of the movement during the operation and the positional relationship between the hook and the surrounding environment. To address this problem, researchers proposed installing a 2D camera on the crane boom and monitoring the position of the hook in the operation scene from images taken from high altitudes. However, the images taken by the 2D camera reflect the positional relationship of objects in two-dimensional space and lack three-dimensional depth information. As a result, the operator cannot quickly perform effective and safe lifting operations. Summary of the invention

[0004] In view of this, it is necessary to provide a crane hook position perception and safety warning method to solve the above-mentioned defects of the prior art.

[0005] In order to solve the above problems, in a first aspect, an embodiment of the present invention provides a crane hook position sensing and safety warning method, comprising:

[0006] S1, real-time collection of crane hook images, and crane hook tracking based on the real-time collected crane hook images and a pre-established crane hook template library;

[0007] S2, uses SLAM technology to build a map of the crane operation scene based on real-time collected images and locates the camera position in the map;

[0008] S3, calculating the distance between the hook and the camera according to the camera imaging principle and the hook tracking result;

[0009] S4, according to the camera positioning result and the distance between the hook and the camera, the position of the hook in the map is obtained, and the hook collision warning and hook swing monitoring are performed according to the hook position.

[0010] Preferably, step S1 specifically includes:

[0011] S11, continuous images of the crane hook are captured using a monocular camera;

[0012] S12, constructing a crane hook template library containing a variety of hook images, and using the first several frames of images taken by the monocular camera to adaptively initialize the hook template features;

[0013] S13, tracking the crane hook in the subsequent image according to the initialized template features;

[0014] S14, updating the online template queue according to the hook tracking result to eliminate the accumulated error.

[0015] Preferably, in step S12, the adaptive initialization of the hook template features using the first several frames of images taken by the monocular camera specifically includes:

[0016] The first frame image taken by the monocular camera is segmented into multiple regions. For each segmented region, it is matched with the hook template in the crane hook template library, and a bounding box confidence score is calculated; wherein the bounding box confidence score represents the similarity between the current region and the hook template.

[0017] Select the three hook templates with the highest bounding box confidence scores, and calculate the average confidence of the three hook templates in several subsequent frame images;

[0018] According to the calculated average confidence, the hook template with the highest average confidence score is selected to calculate the initial template feature T init , initial template feature T init Used to identify and track the hook in subsequent frames.

[0019] Preferably, in step S14, updating the online template queue according to the hook tracking result includes:

[0020] Calculate a confidence score, prediction box and template features for the hook template in the image at each moment;

[0021] Compare the difference between the historical aspect ratio and the current aspect ratio of the hook template in the prediction frame. If it is determined that the ratio of the historical aspect ratio to the current aspect ratio is less than a preset threshold, use an adaptive weight to aggregate the current template and the initial template to obtain an updated online template.

[0022] Using the updated online template and the initial template feature T init to track the location and predicted box of the crane hook in subsequent images.

[0023] Preferably, in step S3, the calculating the distance between the hook and the camera according to the camera imaging principle and the hook tracking result specifically includes:

[0024] S31, according to the imaging principle of the monocular camera, the expressions of the pixel width x and pixel length y of the hook in the image are obtained:

[0025]

[0026]

[0027] In the formula, f x and f y They represent the focal length parameters corresponding to the pixel width x and pixel length y in the camera intrinsic parameter matrix; w and h represent the width and length of the hook in the real world; d is the distance between the camera and the hook;

[0028] S32, calculating the diagonal length L of the hook in the image according to the expressions of the pixel width and pixel length of the hook and the crane hook tracking result obtained in step S1;

[0029]

[0030] Substituting the expressions of pixel width x and pixel length y into the above formula, we get the expression of the distance d between the hook and the camera:

[0031]

[0032] S33, since the hook is affected by factors such as swing during operation, the real width w and real length h corresponding to the pixel width x and pixel length y of the hook in the image fluctuate; therefore, the above relationship is converted into a power function form to obtain:

[0033] d=k×L α .

[0034] Where k and α are constants;

[0035] The crane hook is lowered from high to low at a constant speed, and the distance d between the hook and the camera and the diagonal length L of the hook in the image are recorded during the descent. By fitting these data points, the values ​​of k and α are obtained.

[0036] Preferably, in step S4, the hook collision warning process includes:

[0037] Get the distance s between each map point and the hook position in the crane operation scene map;

[0038] The map points whose distance s is less than the preset distance warning value are projected into the crane hook image and marked, and the results of the hook collision warning are visualized in the control room.

[0039] Preferably, in step S4, the hook swing monitoring process includes:

[0040] The swing angle and swing distance of the hook are calculated according to the hook position and trajectory in the map. The calculation formula is:

[0041]

[0042]

[0043]

[0044] In the formula, is the swing angle of the hook in the x-axis direction, is the swing angle of the hook in the y-axis direction, (x', y', z') are the position coordinates of the hook in three-dimensional space, O x and O y represents the coordinate point of the hook in a stationary state, and l represents the swing distance of the hook;

[0045] Visualize the results of hook swing monitoring to the control room;

[0046] The swing parameters of the hook are used as input signals of the anti-sway system, and the anti-sway parameters of the anti-sway system are adjusted according to the swing parameters of the hook to realize negative feedback control of the anti-sway system of the hook; wherein the anti-sway parameters include the crane hydraulic pressure and the boom rotation speed.

[0047] In a second aspect, an embodiment of the present invention provides a crane hook position sensing and safety warning system, comprising:

[0048] A crane hook tracking module is used to collect crane hook images in real time and to track the crane hook based on the real-time collected crane hook images and a pre-established crane hook template library;

[0049] A map construction module, which is used to construct a map of the crane operation scene based on the real-time collected images using SLAM technology and locate the position of the camera in the map;

[0050] The hook distance measurement module is used to calculate the distance between the hook and the camera based on the camera imaging principle and the hook tracking result;

[0051] The safety warning module is used to obtain the position of the hook in the map based on the camera positioning result and the distance between the hook and the camera, and to perform hook collision warning and hook swing monitoring based on the hook position.

[0052] In a third aspect, the present invention further provides an electronic device, including a memory and a processor, wherein:

[0053] The memory is used to store programs;

[0054] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the crane hook position perception and safety warning method as described in the embodiment of the first aspect of the present invention.

[0055] In a fourth aspect, the present invention further provides a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the crane hook position perception and safety warning method as described in the embodiment of the first aspect of the present invention.

[0056] Compared with the prior art, the crane hook position sensing and safety warning method and system provided by the present invention have the following beneficial effects:

[0057] 1) The present invention realizes real-time and accurate perception of the hook position through a monocular camera and hook tracking technology, avoiding the problem of lack of depth information of 2D cameras. The hook is tracked based on a pre-established crane hook template library. The accuracy of hook position perception is improved by adaptively initializing the hook template features.

[0058] 2) The present invention performs hook collision warning and hook swing monitoring according to the position and trajectory of the hook in the map, and visualizes the warning results in the control room, thereby improving the operator's safety awareness and reaction speed.

[0059] 3) The present invention implements negative feedback control of the hook anti-sway system and adjusts the anti-sway parameters according to the hook's swing parameters, such as the crane hydraulic pressure and boom rotation speed, to reduce the swing amplitude of the hook, thereby improving the stability and safety of the crane. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A flow chart of the crane hook position sensing and safety warning method provided by the present invention;

[0061] Figure 2 A schematic diagram of the bounding box prediction of the hook template provided by the present invention in the crane hook image;

[0062] Figure 3 A flow chart of updating the online template queue provided by the present invention;

[0063] FIG4( a ) is a schematic diagram of the spatial position relationship between the hook and the camera provided by the present invention;

[0064] FIG4( b ) is a schematic diagram of a power function provided by the present invention;

[0065] Figure 5 A schematic diagram of a hook collision warning label provided in an embodiment of the present invention;

[0066] Figure 6A schematic diagram of the swing angle and swing distance of the hook provided by the present invention;

[0067] Figure 7 The present invention provides a curve showing the change of the swing angle and the swing distance of the hook at different times;

[0068] Figure 8 A structural block diagram of the crane hook position sensing and safety warning system provided by the present invention;

[0069] Fig. 9 This is a structural block diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0070] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0071] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0072] At present, existing technologies usually use distance sensors or 2D cameras to monitor the position of the hook, but the measurement results of the distance sensor can only provide mechanical values, and cannot reflect the swing state of the hook due to the inertia of the movement during the operation and the positional relationship between the hook and the surrounding environment. To address this problem, researchers proposed installing a 2D camera on the crane boom and monitoring the position of the hook in the operation scene from images taken from high altitudes. However, the images taken by the 2D camera reflect the positional relationship of objects in two-dimensional space and lack three-dimensional depth information. As a result, the operator cannot quickly perform effective and safe lifting operations.

[0073] In view of this, the present invention provides a crane hook position perception and safety warning method, which realizes real-time and accurate perception of the hook position through a monocular camera and hook tracking technology, avoiding the problem of lack of depth information of a 2D camera. According to the position and trajectory of the hook in the map, hook collision warning and hook swing monitoring are performed. The warning results are visualized in the control room, which improves the safety awareness and reaction speed of the operator. The following will be explained and introduced through multiple embodiments.

[0074] Figure 1 The flowchart of the crane hook position sensing and safety warning method provided by the present invention. Figure 1As shown, the crane hook position sensing and safety warning method includes steps S1 to S4, wherein:

[0075] Step S1, real-time acquisition of crane hook images, and crane hook tracking based on the real-time acquired crane hook images and a pre-established crane hook template library.

[0076] In a preferred embodiment of the present invention, step S1 specifically includes the following steps S11 to S14:

[0077] S11, continuous images of the crane hook are captured using a monocular camera.

[0078] In actual operation, the monocular camera is first installed on the crane boom, and then the monocular camera is used to collect continuous images of the crane hook. In this embodiment, the images collected by the monocular camera are used for crane hook tracking and synchronous positioning and map construction.

[0079] S12, constructing a crane hook template library containing various hook images, and using the first several frames of images taken by the monocular camera to adaptively initialize the hook template features.

[0080] Specifically, first, a template library containing various hook images is pre-built for different operation scenarios. This template library is the basis for subsequent recognition and tracking of hooks.

[0081] Next, the first frame image taken by the monocular camera is divided into multiple regions. Figure 2 This is a schematic diagram of the bounding box prediction of the hook template provided by the present invention in the crane hook image, referring to Figure 2 In this embodiment, the first frame image captured by the monocular camera is divided into five regions. For each divided region, it is matched with the hook template in the crane hook template library, and the bounding box confidence score is calculated; wherein the bounding box confidence score (Bounding Box PredictionScore) represents the similarity between the current region and the hook template. This score is usually a value between 0 and 1, and the higher the value, the higher the similarity.

[0082] Then, the three hook templates with the highest bounding box confidence scores are selected, and the average confidence of the three hook templates in the subsequent 2-5 frames of images is calculated respectively. In this embodiment, the confidence can represent the ability of the template to stably track the hook in the subsequent frames, which is obtained through the tracking model output. The average confidence is the average value of the confidence in multiple image frames, which is used to evaluate the performance stability of the hook template in consecutive frames.

[0083] Furthermore, according to the calculated average confidence, the hook template with the highest average confidence score is selected to calculate the initial template feature T init , initial template feature T init For identifying and tracking the hook in subsequent frames, the initial template feature T init The expression is:

[0084]

[0085]

[0086] In the formula, C i,j is the confidence score of the i-th hook template with the highest average confidence score in the j-th frame image, F i,j is the template feature of the i-th hook template in the j-th frame image, P i is the total confidence score of the i-th hook template in frames 2-5.

[0087] This embodiment selects the hook template with the highest average confidence score to calculate the initial template features for subsequent tracking, ensuring that the initial template features can more accurately match the actual photographed hook, thereby improving the accuracy of recognition and tracking.

[0088] S13, tracking the crane hook in subsequent images according to the initialized template features.

[0089] Specifically, according to the template features initialized in step S12, the system starts to track the hook in subsequent images. By comparing the features in the current image with the template features, the system can determine the position of the hook.

[0090] S14, updating the online template queue according to the hook tracking result to eliminate the accumulated error.

[0091] It is understandable that in the process of hook tracking, due to the influence of various factors (such as illumination changes, hook swing, camera shake, etc.), the tracking results may produce errors. If these errors are not corrected in time, they will gradually accumulate as the tracking proceeds, resulting in tracking failure or reduced accuracy. In order to solve this problem, the embodiment of the present invention adopts an improved online update strategy, that is, updating the online template queue according to the results of hook position tracking in the current frame combined with the initial template features.

[0092] In this embodiment, the "online template queue" is a data structure for storing and managing hook templates. During the hook tracking process, the system will continuously update the templates in this queue to adapt to the changes of the hook in different scenarios. This embodiment uses an improved online update strategy to update the hook template online to eliminate cumulative errors and maintain the accuracy and stability of tracking.

[0093] Figure 3 For the update online template queue flow chart provided by the present invention, refer to Figure 3 First, a confidence score C is calculated for the hook template in the image at each moment t , prediction box B t and the template trait T t .

[0094] Then, by comparing the historical aspect ratio R of the hook template in the prediction box t and the current aspect ratio R t , to determine whether the hook has changed significantly. h and the current aspect ratio R t If the ratio of is less than the preset threshold, an adaptive weight β is used to aggregate the current template feature T t and the initial template feature T init , get the updated online template feature T online .

[0095] β=C init / (C t +C init )

[0096] T online =(1-β)T t +βT init

[0097] In the formula, β is the initial template feature T init and the current template feature T t The confidence score (C init ,C t ) determines the adaptive weight.

[0098] Finally, use the updated online template feature T online and the initial template feature T init To track the position and prediction box of the crane hook in subsequent images, the online template can be dynamically adjusted to eliminate accumulated errors and ensure the accuracy and real-time performance of hook tracking.

[0099] The embodiment of the present invention realizes real-time and accurate perception of the hook position through a monocular camera and hook tracking technology, avoiding the problem of a 2D camera lacking depth information. The hook is tracked based on a pre-established crane hook template library. The accuracy of hook position perception is improved by adaptively initializing the hook template features.

[0100] Step S2, using SLAM technology to build a map of the crane operation scene based on the real-time collected images, and locate the position of the camera in the map.

[0101] Specifically, SLAM (Simultaneous Localization and Mapping) is a technology that can simultaneously perform positioning and map construction in an unknown environment. It uses environmental information obtained by sensors (such as cameras, laser radars, etc.) and processes them through algorithms to achieve autonomous navigation and positioning of the robot. In this embodiment, a monocular camera is used to collect images of the crane operation scene in real time, and SLAM technology is used to build a 3D map of the crane operation scene.

[0102] The map construction process may include: extracting features from the input crane operation scene image, such as corners, edges, textures, etc. These feature points are the basis for building maps and positioning cameras. By matching with the feature points in the previous frame image or map, the system can determine the relative position relationship between the current image and the map. Based on the matched feature points, the 3D map of the crane operation scene is gradually constructed using a 3D reconstruction algorithm. This map contains the location information of all objects and obstacles in the operation scene.

[0103] While building the map, the system will also locate the current camera position in the map in real time. Camera positioning can be achieved by comparing the current image with the feature points in the map.

[0104] In this embodiment, by adopting SLAM technology, a 3D map of the crane operation scene is constructed based on real-time collected images, so that comprehensive environmental information of the crane hook can be obtained, and then the position perception of the camera and the hook in the map can be realized, providing accurate data support for safety warning and anti-sway control.

[0105] Step S3, calculating the distance between the hook and the camera according to the camera imaging principle and the hook tracking result.

[0106] In a preferred embodiment of the present invention, step S3 may include the following steps S31 to S33;

[0107] S31, according to the imaging principle of the monocular camera, the expressions of the pixel width x and pixel length y of the hook in the image are obtained:

[0108]

[0109]

[0110] In the formula, f x and f y represents the focal length parameters corresponding to the pixel width x and pixel length y in the camera intrinsic parameter matrix; w and h represent the width and length of the hook in the real world; d is the distance between the camera and the hook. Figure 4(a) is a schematic diagram of the spatial position relationship between the hook and the camera provided by the present invention.

[0111] S32, calculating the diagonal length L of the hook in the image according to the expressions of the pixel width and pixel length of the hook and the crane hook tracking result obtained in step S1;

[0112]

[0113] Substituting the expressions of pixel width x and pixel length y into the above formula, we get the expression of the distance d between the hook and the camera:

[0114]

[0115] S33, since the hook is affected by factors such as swing during operation, the width x and length y of the hook in the image fluctuate; therefore, the above relationship is converted into a power function form to obtain:

[0116] d=k×L α .

[0117] In the formula, k and α are constants. Fig. 4(b) is a schematic diagram of the power function provided by the present invention. The hook size in Fig. 4(b) refers to the diagonal length L of the hook.

[0118] In this embodiment, in order to obtain an accurate hook height, the constants k and α in the above power function need to be calibrated. Specifically, the crane hook is lowered from high to low at a constant speed, and the distance d between the hook and the camera and the diagonal length L of the hook in the image are recorded during the descent process. By fitting these data points, the values ​​of k and α are obtained.

[0119] Step S4, obtaining the position of the hook in the map according to the camera positioning result and the distance between the hook and the camera, and performing hook collision warning and hook swing monitoring according to the hook position.

[0120] Specifically, after obtaining the position of the camera in the map and the distance between the hook and the camera, the position coordinates (x′, y′, z′) of the hook in the map can be calculated through spatial geometric transformation. This process requires the use of the camera's intrinsic parameter matrix (including parameters such as focal length and optical center) and the transformation relationship between the camera and the map (including rotation matrix R and translation vector T). The calculation formula is:

[0121]

[0122] Where u, v, 1 T is the homogeneous coordinate of the center pixel of the hook in the tracking result, R is the rotation matrix, T is the translation vector, and K is the intrinsic parameter matrix of the camera.

[0123] Further, hook collision warning and hook swing monitoring are performed according to the hook position. In a preferred embodiment of the present invention, the hook collision warning process includes:

[0124] First, the distance s between each map point in the crane operation scene map and the hook position is obtained. This step can obtain the distance between the hook position coordinates and obstacles or potential collision points in the map.

[0125] Then, the calculated distance s is compared with the preset distance warning value. If s is less than the distance warning value, it is considered that there is a collision risk. The map points where the distance s is less than the preset distance warning value are projected into the crane hook image and marked in a striking way (such as red dot marking), and the results of the hook collision warning are visualized in the control room, so that the operator in the control room can intuitively see the potential collision risk area. Figure 5 A schematic diagram of a hook collision warning label provided in an embodiment of the present invention.

[0126] In a preferred embodiment of the present invention, the hook swing monitoring process includes:

[0127] The swing angle and swing distance of the hook are calculated according to the position and trajectory of the hook in the map. In this embodiment, the spatial coordinates of the hook at different times can be obtained according to the position and movement trajectory of the hook in the 3D map; the swing angle and swing distance of the hook are calculated using the spatial coordinates of the hook at different times. Figure 6 A schematic diagram of the swing angle and swing distance of the hook provided by the present invention.

[0128] The calculation formula for the swing angle and swing distance of the crane hook is:

[0129]

[0130]

[0131]

[0132] In the formula, is the swing angle of the hook in the x-axis direction, is the swing angle of the hook in the y-axis direction, (x', y', z') are the position coordinates of the hook in three-dimensional space, O x and O y represents the coordinate point of the hook in a static state, and l represents the swing distance of the hook.

[0133] After calculating the swing parameters of the crane hook, the swing angle and swing distance of the hook are displayed graphically on the display screen in the control room, so that the operator can intuitively understand the swing of the hook. Figure 7The present invention provides a variation curve of the swing angle and the swing distance of the hook at different times, and researchers can analyze the swing state of the hook according to the swing parameters of the hook.

[0134] In a preferred embodiment of the present invention, after the hook swing monitoring is completed, negative feedback control can be performed on the hook anti-swing system according to the hook swing parameters obtained by monitoring.

[0135] Specifically, the swing parameters of the hook can be used as the input signal of the anti-sway system, and the anti-sway parameters of the anti-sway system can be adjusted according to the swing parameters of the hook to implement negative feedback control of the anti-sway system of the hook; wherein the anti-sway parameters include the hydraulic pressure of the crane and the rotation speed of the boom. Here, negative feedback control is used as a control strategy of the anti-sway system, and the input of the system (such as the hydraulic pressure of the crane, the rotation speed of the boom, etc.) is adjusted according to the output of the system (the swing parameters of the hook) to achieve the purpose of reducing errors or maintaining stability. In this embodiment, by adjusting the anti-sway parameters, the anti-sway system can correct the swing of the hook in real time, thereby reducing the swing amplitude of the hook and improving the stability and safety of the crane.

[0136] Figure 8 The structural block diagram of the crane hook position sensing and safety warning system provided by the present invention is shown in FIG. Figure 8 , Crane hook position sensing and safety warning system 800 includes:

[0137] A crane hook tracking module 801 is used to collect crane hook images in real time, and to track the crane hook based on the real-time collected crane hook images and a pre-established crane hook template library;

[0138] A map construction module 802 is used to construct a map of the crane operation scene based on the real-time collected images using SLAM technology, and locate the position of the camera in the map;

[0139] The hook distance measurement module 803 is used to calculate the distance between the hook and the camera according to the camera imaging principle and the hook tracking result;

[0140] The safety warning module 804 is used to obtain the position of the hook in the map according to the camera positioning result and the distance between the hook and the camera, and to perform hook collision warning and hook swing monitoring according to the hook position.

[0141] The crane hook position sensing and safety warning system provided by the present invention executes the crane hook position sensing and safety warning methods provided by the aforementioned embodiments through the above-mentioned modules. The crane hook position sensing and safety warning methods have been described in detail in the above-mentioned embodiments, and will not be repeated in this embodiment.

[0142] Compared with the prior art, the crane hook position sensing and safety warning method and system provided by the present invention have the following beneficial effects:

[0143] 1) The present invention realizes real-time and accurate perception of the hook position through a monocular camera and hook tracking technology, avoiding the problem of lack of depth information of 2D cameras. The hook is tracked based on a pre-established crane hook template library. The accuracy of hook position perception is improved by adaptively initializing the hook template features.

[0144] 2) The present invention performs hook collision warning and hook swing monitoring according to the position and trajectory of the hook in the map, and visualizes the warning results in the control room, thereby improving the operator's safety awareness and reaction speed.

[0145] 3) The present invention implements negative feedback control of the hook anti-sway system and adjusts the anti-sway parameters according to the hook's swing parameters, such as the crane hydraulic pressure and boom rotation speed, to reduce the swing amplitude of the hook, thereby improving the stability and safety of the crane.

[0146] Fig. 9 The structural block diagram of the electronic device provided by the present invention is as follows: Fig. 9 As shown, the present invention also provides an electronic device, and the electronic device 900 can be a computing device such as a mobile terminal, a desktop computer, a notebook, a palm computer, and a server. The electronic device 900 includes a processor 901 and a memory 902, wherein the memory 902 stores a crane hook position sensing and safety warning program 909.

[0147] In some embodiments, the memory 902 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 902 may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 902 may also include both an internal storage unit of the computer device and an external storage device. The memory 902 is used to store application software and various types of data installed on the computer device, such as program codes for installing the computer device. The memory 902 may also be used to temporarily store data that has been output or is to be output. In one embodiment, when the crane hook position perception and safety warning program 909 is executed by the processor 901, the following steps are implemented:

[0148] S1, real-time collection of crane hook images, and crane hook tracking based on the real-time collected crane hook images and a pre-established crane hook template library;

[0149] S2, uses SLAM technology to build a map of the crane operation scene based on real-time collected images and locates the camera position in the map;

[0150] S3, calculating the distance between the hook and the camera according to the camera imaging principle and the hook tracking result;

[0151] S4, according to the camera positioning result and the distance between the hook and the camera, the position of the hook in the map is obtained, and the hook collision warning and hook swing monitoring are performed according to the hook position.

[0152] In some embodiments, the processor 901 can be a central processing unit (CPU), a microprocessor or other data processing chip, which is used to run the program code or process data stored in the memory 902, such as executing crane hook position perception and safety warning programs.

[0153] This embodiment also provides a computer-readable storage medium, on which a crane hook position sensing and safety warning program is stored. When the crane hook position sensing and safety warning program is executed by a processor, the following steps are implemented:

[0154] S1, real-time collection of crane hook images, and crane hook tracking based on the real-time collected crane hook images and a pre-established crane hook template library;

[0155] S2, uses SLAM technology to build a map of the crane operation scene based on real-time collected images and locates the camera position in the map;

[0156] S3, calculating the distance between the hook and the camera according to the camera imaging principle and the hook tracking result;

[0157] S4, according to the camera positioning result and the distance between the hook and the camera, the position of the hook in the map is obtained, and the hook collision warning and hook swing monitoring are performed according to the hook position.

[0158] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A crane hook position sensing and safety warning method, characterized in that: include: S1, real-time collection of crane hook images, and crane hook tracking based on the real-time collected crane hook images and a pre-established crane hook template library; S2, uses SLAM technology to build a map of the crane operation scene based on real-time collected images and locates the camera position in the map; S3, calculating the distance between the hook and the camera according to the camera imaging principle and the hook tracking result; S4, according to the camera positioning result and the distance between the hook and the camera, the position of the hook in the map is obtained, and the hook collision warning and hook swing monitoring are performed according to the hook position.

2. The crane hook position sensing and safety warning method according to claim 1 is characterized in that: Step S1 specifically includes: S11, continuous images of the crane hook are captured using a monocular camera; S12, constructing a crane hook template library containing a variety of hook images, and using the first several frames of images taken by the monocular camera to adaptively initialize the hook template features; S13, tracking the crane hook in the subsequent image according to the initialized template features; S14, updating the online template queue according to the hook tracking result to eliminate the accumulated error.

3. The crane hook position sensing and safety warning method according to claim 2 is characterized in that: In step S12, the first several frames of images taken by the monocular camera are used to adaptively initialize the hook template features, specifically including: The first frame image taken by the monocular camera is segmented into multiple regions. For each segmented region, it is matched with the hook template in the crane hook template library, and the confidence score of the bounding box is calculated; wherein the bounding box confidence score represents the similarity between the current region and the hook template. Select the three hook templates with the highest bounding box confidence scores, and calculate the average confidence of the three hook templates in several subsequent frame images; According to the calculated average confidence, the hook template with the highest average confidence score is selected to calculate the initial template feature T init , initial template feature T init Used to identify and track the hook in subsequent frames.

4. The crane hook position sensing and safety warning method according to claim 2 is characterized in that: In step S14, updating the online template queue according to the hook tracking result includes: Calculate a confidence score, prediction box and template features for the hook template in the image at each moment; Compare the difference between the historical aspect ratio and the current aspect ratio of the hook template in the prediction frame. If it is determined that the ratio of the historical aspect ratio to the current aspect ratio is less than a preset threshold, use an adaptive weight to aggregate the current template and the initial template to obtain an updated online template. Using the updated online template and the initial template feature T init to track the location and predicted box of the crane hook in subsequent images.

5. The crane hook position sensing and safety warning method according to claim 1 is characterized in that: In step S3, the distance between the hook and the camera is calculated according to the camera imaging principle and the hook tracking result, which specifically includes: S31, according to the imaging principle of the monocular camera, the expressions of the pixel width x and pixel length y of the hook in the image are obtained: In the formula, f x and f y They represent the focal length parameters corresponding to the pixel width x and pixel length y in the camera intrinsic parameter matrix; w and h represent the width and length of the hook in the real world; d is the distance from the camera to the hook; S32, calculating the diagonal length L of the hook in the image according to the expressions of the pixel width and pixel length of the hook and the crane hook tracking result obtained in step S1; Substituting the expressions of pixel width x and pixel length y into the above formula, we get the expression of the distance d between the hook and the camera: S33, since the hook is affected by factors such as swing during operation, the real width w and real length h corresponding to the pixel width x and pixel length y of the hook in the image fluctuate; therefore, the above relationship is converted into a power function form to obtain: d=k×L α 。 Where k and α are constants; The crane hook is lowered from high to low at a constant speed, and the distance d between the hook and the camera and the diagonal length L of the hook in the image are recorded during the descent. By fitting these data points, the values ​​of k and α are obtained.

6. The crane hook position sensing and safety warning method according to claim 1 is characterized in that: In step S4, the hook collision warning process includes: Get the distance s between each map point and the hook position in the crane operation scene map; The map points whose distance s is less than the preset distance warning value are projected into the crane hook image and marked, and the results of the hook collision warning are visualized in the control room.

7. The crane hook position sensing and safety warning method according to claim 1 is characterized in that: In step S4, the hook swing monitoring process includes: The swing angle and swing distance of the hook are calculated according to the hook position and trajectory in the map. The calculation formula is: In the formula, is the swing angle of the hook in the x-axis direction, is the swing angle of the hook in the y-axis direction, (x', y', z') are the position coordinates of the hook in three-dimensional space, O x and O y represents the coordinate point of the hook in a stationary state, and l represents the swing distance of the hook; Visualize the results of hook swing monitoring to the control room; The swing parameters of the hook are used as input signals of the anti-sway system, and the anti-sway parameters of the anti-sway system are adjusted according to the swing parameters of the hook to realize negative feedback control of the anti-sway system of the hook; wherein the anti-sway parameters include the crane hydraulic pressure and the boom rotation speed.

8. A crane hook position sensing and safety warning system, characterized in that: include: A crane hook tracking module is used to collect crane hook images in real time and to track the crane hook based on the real-time collected crane hook images and a pre-established crane hook template library; A map construction module, which is used to construct a map of the crane operation scene based on the real-time collected images using SLAM technology and locate the position of the camera in the map; The hook distance measurement module is used to calculate the distance between the hook and the camera based on the camera imaging principle and the hook tracking result; The safety warning module is used to obtain the position of the hook in the map based on the camera positioning result and the distance between the hook and the camera, and to perform hook collision warning and hook swing monitoring based on the hook position.

9. An electronic device, It is characterized in that comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the crane hook position perception and safety warning method as described in any one of claims 1 to 7 above.

10. A computer-readable storage medium, characterized in that: Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the crane hook position perception and safety warning method as described in any one of claims 1 to 7 above.

Images