Intelligent monitoring and early warning method and system for crane and storage medium

Abstract

The invention relates to the technical field of intelligent monitoring of cranes, and discloses an intelligent monitoring and early warning method and system for a crane and a storage medium. The method comprises the steps of periodically collecting monitoring data of a crane, generating an operation area image, setting a lifting hook and a sling on the operation area image in a blurring mode, generating an area marking graph and displaying the area marking graph on a display screen, recognizing ground objects on the operation area image, and if the ground objects are human bodies, setting a protection area. Calculating the horizontal distance between the human body and the projection point of the lifting hook on the reference plane, sending out first early warning information when the human body is located in the protection area, calculating the horizontal distance and the vertical distance between the object and the load if the ground object is the object, and sending out second early warning information when the horizontal distance is smaller than or equal to a first preset value and the vertical distance is smaller than or equal to a second preset value; according to the method and the device, the transparency and the controllability of crane operation are improved, and the monitoring accuracy and the intelligent degree are improved.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent monitoring of cranes, and in particular to an intelligent monitoring and early warning method, system and storage medium for cranes. Background Art

[0002] With the rapid development of industrial automation and intelligence, cranes, as important lifting and transportation equipment, play a vital role in the fields of construction, ports, logistics, etc. However, there are many safety hazards in the operation of cranes, such as operating errors, equipment failures, environmental factors, etc., which may lead to serious safety accidents. In order to improve the safety and reliability of cranes and reduce the accident rate, intelligent monitoring and early warning systems came into being.

[0003] Similar prior art includes a Chinese patent application with publication number CN114314355A, which discloses a crane safety monitoring method, device, system and remote monitoring center. When it is detected that the distance between the obstacle and the crane is less than the distance threshold, a deceleration instruction is sent to the crane controller to control the crane to decelerate; when the obstacle is detected to be accelerating toward the crane and the sampling detection number is greater than the preset detection number threshold, a steering instruction is sent to the crane controller to control the crane to turn in the direction away from the obstacle. There is also a Chinese patent application with publication number CN102092640A, which discloses a crane safety monitoring device, method and a crane including the device, which detects the surrounding space of the crane and generates corresponding distance information according to the detected set target; reads the current operating parameter information of the crane, compares the distance information with the current operating parameter information to determine whether the target is within the operating range of the crane, and generates a corresponding output signal according to the comparison result; and performs a predetermined operation in response to the output signal.

[0004] The above-mentioned prior art only considers the distance between the target object and the crane when monitoring, does not distinguish the target object, and cannot effectively prompt the driver to adjust the operation action. Therefore, it is an urgent problem to provide an intelligent monitoring and early warning method, system and storage medium for cranes to improve the transparency and controllability of operations and improve the accuracy and intelligence of monitoring. Summary of the invention

[0005] The present application provides an intelligent monitoring and early warning method, system and storage medium for a crane, which are used to improve the operating efficiency and control accuracy of the crane.

[0006] In a first aspect, the present application provides an intelligent monitoring and early warning method for a crane, the intelligent monitoring and early warning method for a crane comprising: Step 1: Collect monitoring data of the crane according to a preset period, wherein the monitoring data includes image data; Step 2: generating a first operation area image based on the image data; Step 3: setting the hook and sling of the crane in a blurred manner on the first working area image to generate a second working area image, then generating a region annotation map of the working area based on the monitoring data and the second working area image, and displaying the region annotation map on the crane display screen, wherein the region annotation image includes a first relative distance from the lower surface of the load to the reference plane; Step 4, identifying the ground object on the image of the first working area, if the ground object is a human body, proceed to step 5, if the ground object is an object, proceed to step 6; Step 5: Set a protection area based on the monitoring data, and calculate the first horizontal distance between the projection point of the human body and the hook on the reference plane. Based on the first horizontal distance, determine whether the human body is within the protection area. If so, issue a first warning message. Step 6: Calculate the second horizontal distance between the object and the projection point, and the vertical distance between the load and the object. When the second horizontal distance is less than or equal to the first preset value and the vertical distance is less than or equal to the second preset value, issue a second warning message.

[0007] In combination with the first aspect, in a first implementation of the first aspect of the present application, in step 2, a method for generating the first working area image is: Step 21: extract any image data, input any image data into a preset model, and obtain recognition results of all elements in any image data; Step 22: based on the recognition result, extract the coordinates of the pixel points corresponding to the hook and the sling, and define them as the first coordinate points; based on the first coordinate points, intercept a side image that does not contain the hook and the sling from any image data, and define it as the first image; Step 23: After traversing all the image data, generate a first working area image based on all the first images.

[0008] In combination with the first aspect, in a second implementation of the first aspect of the present application, the monitoring data includes first three-dimensional point cloud data and geometric correction parameters, and in step 3, generating an area annotation map of the working area based on the monitoring data and the second working area image includes: Step 31, adjusting the coordinates of the first three-dimensional point cloud based on the geometric correction parameters to generate a second three-dimensional point cloud, and then calculating a second relative distance from a reference plane in the operation area to the three-dimensional data acquisition device based on the second three-dimensional point cloud; Step 32: based on the first relative distance and the second three-dimensional point cloud, obtain the first relative distance, the third relative distance from the upper surface of the load to the reference plane, and the fourth relative distance from the upper surface of the object to the reference plane, and mark the first relative distance, the third relative distance, and the fourth relative distance on the second working area image correspondingly; Step 33: Acquire device parameters of the image acquisition device, and calculate the data acquisition range of the image acquisition device based on the first relative distance and the device parameters; Step 34, obtain the length of the crane boom, take the fulcrum for the left and right movement of the crane boom as the first origin, generate a first trajectory line for the left and right movement of the crane boom based on the length of the crane boom, and draw the first trajectory line within the data acquisition range on the second working area image. At the same time, obtain the vertical projection line from the hook to the reference plane based on the projection point, and generate a second trajectory line for the forward and backward movement of the crane boom based on the first origin and the vertical projection line, and draw the second trajectory line within the data acquisition range on the second working area image.

[0009] In combination with the first aspect, in a third implementation of the first aspect of the present application, the second relative distance is calculated as follows: Step 311: setting a plurality of longitudinal first grids based on the area of ​​the working area, and dividing the point cloud corresponding to any first grid into any first grid; Step 312: extract any first grid, calculate the first vertical distance between each scanning point in any first grid and the three-dimensional data acquisition device, and after traversing all scanning points, extract the scanning point corresponding to the maximum value of the first vertical distance and define it as the first reference point; Step 313, calculating the vertical distance difference between any scanning point and the first reference point, and when the vertical distance difference is less than or equal to a third preset value, defining any scanning point as the second reference point; Step 314: after traversing all scanning points, generate a first reference point group based on the first reference point and all second reference points, and use the average of the first vertical distances of all reference points in the first reference point group as the reference distance of any first grid; Step 315: After traversing all first grids, the minimum reference distance is used as the second relative distance.

[0010] In combination with the first aspect, in a fourth implementation of the first aspect of the present application, step 32 includes: Step 321: setting a plurality of horizontal second grids based on the second relative distance and the preset interval, and dividing the point cloud corresponding to any second grid into any second grid; Step 322: extract any second grid, extract any two scanning points, calculate the point distance between the any two scanning points, and determine whether the point distance is less than or equal to a fourth preset value. If not, reacquire the two scanning points and repeat the determination of step 322. If yes, define the any two scanning points as a second reference point group, and then proceed to step 323. Step 323, calculating the geometric center of the second reference point group, determining whether there is a first scanning point in any second grid whose point distance to the geometric center is less than or equal to a fourth preset value, if so, generating a new second reference point group based on the second reference point group and the first scanning point, and then returning to step 323, if not, determining whether point group division is completed for all scanning points in any second grid, if so, proceeding to step 324, if not, returning to step 322; Step 324: cluster all the scan points in the second reference point group respectively to obtain multiple point sets, extract any point set, calculate the point-to-surface distance between any scan point in any point set and the reference plane, and take the average of all the point-to-surface distances as the fifth relative distance of any point set; Step 325, after traversing all the second grids, extract any two adjacent point sets belonging to different second grids, calculate the inter-set distance of the two point sets based on the fifth relative distance, and determine whether the inter-set distance is less than or equal to the fifth preset value. If so, proceed to step 326, if not, return to step 325; Step 326: determine whether the two point sets overlap in the vertical direction. If so, determine that the two point sets belong to the same object. If not, return to step 325. Step 327: After traversing all point sets, divide all point sets into different objects, assign an identifier to each object, and identify the load and ground objects based on the division results; Step 328, extract the first identifier corresponding to the load, and extract all point sets corresponding to the first identifier, define them as the first set, take the maximum value of the fifth relative distance corresponding to the first set as the second relative distance, and the minimum value of the fifth relative distance as the first relative distance, and at the same time extract the second identifier corresponding to any ground feature, and extract all point sets corresponding to the second identifier, define them as the second set, and take the maximum value of the fifth relative distance corresponding to the second set as the fourth relative distance.

[0011] In combination with the first aspect, in a fifth implementation of the first aspect of the present application, a calibration device is provided on the hook fixing device, the first calibration device and the hook fixing point are on the same horizontal line, the second calibration device is fixed outside the first calibration device and on the same horizontal line as the first calibration device, the third calibration device is fixed below the first calibration device, and a data acquisition device is provided on the crane. In step 34, the method for obtaining the crane boom length is: Step 341: set a three-dimensional coordinate system with the data acquisition device as the second origin, and obtain first relative coordinates, second relative coordinates, and third relative coordinates of the first calibration device, the second calibration device, and the third calibration device in the three-dimensional coordinate system respectively; Step 342: Generate a correction matrix based on the first relative coordinate, the second relative coordinate, and the third relative coordinate; Step 343, extracting the fourth relative coordinate of the hook fixing point relative to the first calibration device, and generating a first vector based on the fourth relative coordinate; Step 344: define the product of the correction matrix and the first vector as a second vector, and obtain the fifth relative coordinate of the hook fixing point in the three-dimensional coordinate system based on the second vector; Step 345: Obtain the sixth relative coordinate of the first origin relative to the second origin, and calculate the crane boom length based on the fifth relative coordinate and the sixth relative coordinate.

[0012] In combination with the first aspect, in the sixth implementation method of the first aspect of the present application, the monitoring data includes load characteristic parameters and environmental parameters. In step 5, the method for setting the protection area is: based on the first relative distance, environmental parameters and load characteristic parameters, the protection range corresponding to the load is calculated, and then the protection area is set with the projection point as the center of the circle.

[0013] In combination with the first aspect, in a seventh implementation of the first aspect of the present application, a method for calculating the horizontal distance is: The pixel coordinates of any ground feature closest to the load are defined as the second coordinate point, and the pixel distance between the second coordinate point and the projection point is obtained; The relative distance between any object and the reference plane and the device parameters of the image acquisition device are obtained, the pixel size of one pixel is obtained based on the device parameters, the first relative distance and the fourth relative distance of any object, and the horizontal distance is calculated based on the pixel spacing and the pixel size.

[0014] In a second aspect, the present application provides an intelligent monitoring and early warning system for a crane, the intelligent monitoring and early warning system for a crane comprising: a data acquisition module, an image generation module, an image annotation module, a ground object judgment module, a first early warning module and a second early warning module; A data acquisition module, used for collecting monitoring data of the crane according to a preset period, wherein the monitoring data includes image data; An image generation module, used to generate a first operation area image according to the image data; An image annotation module is used to set the hook and the sling of the crane in a blurred manner on the first working area image to generate a second working area image, then generate a region annotation map of the working area based on the monitoring data and the second working area image, and display the region annotation map on the crane display screen, wherein the region annotation image includes a first relative distance from the lower surface of the load to the reference plane; A ground object judgment module is used to identify the ground object on the first operation area image. If the ground object is a human body, the first warning module is entered; if the ground object is an object, the second warning module is entered; A first warning module is used to set a protection area according to the monitoring data, and calculate a first horizontal distance between the projection point of the human body and the hook on the reference plane, and determine whether the human body is within the protection area based on the first horizontal distance. If so, a first warning message is issued; The second warning module is used to calculate the second horizontal distance between the object and the projection point, and the vertical distance between the load and the object, and issue a second warning message when the second horizontal distance is less than or equal to the first preset value and the vertical distance is less than or equal to the second preset value.

[0015] A third aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned intelligent monitoring and early warning method for a crane.

[0016] Compared with the prior art, the beneficial effects of the technical solution of the present application are at least as follows: 1. Automatically process images taken from multiple angles to generate images of the working area, avoiding blind spots in shooting, improving the availability of images of the working area, enabling the monitoring system to cover the working area more comprehensively, and enhancing the comprehensiveness and effectiveness of monitoring.

[0017] 2. Through image processing technology and three-dimensional point cloud data, it can accurately identify and locate the load and objects in the working area, calculate the relative distance between the load, objects and the reference plane, as well as the horizontal distance between the objects and the load and other key parameters, providing accurate data support for early warning judgment, improving the accuracy and reliability of early warning judgment, and reducing the risk of false alarms and missed alarms.

[0018] 3. The area marking map of the operation area is displayed in real time on the display screen, so that the operator can intuitively understand the real-time situation and warning information of the operation area, which is convenient for effective management and decision-making, and improves the transparency and controllability of the operation. 4. Timely identify human bodies and objects in the working area, accurately determine their relative position relationship with the load, and immediately issue a warning message once it is found that a person enters the protection area or an object is too close to the load, thereby effectively avoiding accidents and improving work safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0020] Figure 1 This is a schematic diagram of an embodiment of an intelligent monitoring and early warning method for a crane in an embodiment of the present application; Figure 2 It is a schematic diagram of an embodiment of an intelligent monitoring and early warning system for a crane in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The embodiments of the present application provide an intelligent monitoring and early warning method, system and storage medium for cranes. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 , an embodiment of the intelligent monitoring and early warning method for a crane in the embodiment of the present application includes: Step 1: Collect monitoring data of the crane according to a preset period, wherein the monitoring data includes image data.

[0023] The preset period is set according to the experience of those skilled in the art or according to the actual application scenario, and the embodiments of the present application are not limited to this.

[0024] Specifically, the monitoring data includes at least environmental parameters, image data, and three-dimensional point cloud data.

[0025] Step 2: Generate a first working area image based on the image data.

[0026] In a specific embodiment, in step 2, the method for generating the first working area image is: Step 21: extract any image data, input any image data into a preset model, and obtain recognition results of all elements in any image data.

[0027] Step 22: based on the recognition result, extract the coordinates of the pixel points corresponding to the hook and the sling, and define them as the first coordinate points; based on the first coordinate points, intercept a side image that does not contain the hook and the sling from any image data, and define it as the first image.

[0028] Step 23: After traversing all the image data, generate a first working area image based on all the first images.

[0029] Specifically, there are multiple image data, each of which is captured by a different image acquisition device. The image acquisition device is set on the hook fixing device at the front end of the boom. Since the image acquisition device is located above the load, the captured image data will have blind spots due to the obstruction of the sling and the like. In order to obtain the image of the working area in all directions, multiple image acquisition devices are set, each of which captures the working area from a different angle, and the working area image is generated based on the images captured from multiple angles, avoiding blind spots and improving the comprehensiveness of the working area image.

[0030] The preset model is a pre-trained neural network model that recognizes elements on the image data (loads, hooks, slings, people, objects, etc.) and identifies the pixels belonging to any element.

[0031] Based on the recognition result, a first image excluding the hook and the sling is intercepted from the image data. Preferably, the shape of the first image can be determined according to the number of image data. Exemplarily, when there are two image data, the shape of the first image is a rectangle, the side excluding the sling pixel is defined as the first side, the pixel coordinates of the hook closest to the first side are extracted, and the image area from the above pixel coordinates to the edge of the first side is taken as the first image, and then the two first images are merged to generate the first working area image.

[0032] By intercepting and merging multiple image data, it is possible to obtain images of the working area with no or fewer obstructed parts, thereby ensuring the accuracy and completeness of the working area images. It is further possible to more accurately identify loads and ground objects from the working area images, and more accurately calculate the distance between the loads and ground objects, thereby enhancing the comprehensiveness and effectiveness of the monitoring images.

[0033] Step 3: Blurred settings of the hook and sling of the crane are set on the first working area image to generate a second working area image, and then an area annotation map of the working area is generated based on the monitoring data and the second working area image, and the area annotation map is displayed on the crane display screen, wherein the area annotation image includes a first relative distance from the lower surface of the load to the reference plane.

[0034] The crane hook and sling are set to be blurred on the first work area image, so that the operator can intuitively understand the position of the hook and sling in the work area, enhancing the visualization of the work area information. The regional annotation map of the work area is generated based on the monitoring data and the second work area image, which can accurately obtain the data parameters of the work area, facilitate the operator to carry out work planning and safety monitoring, and provide an accurate basis for subsequent early warning judgment, which helps to improve the accuracy and reliability of the early warning system and effectively prevent accidents.

[0035] In a specific embodiment, the monitoring data includes first three-dimensional point cloud data and geometric correction parameters. In step 3, generating an area annotation map of the working area based on the monitoring data and the second working area image includes: Step 31, adjusting the coordinates of the first three-dimensional point cloud based on the geometric correction parameters to generate a second three-dimensional point cloud, and then calculating a second relative distance from a reference plane in a working area of ​​the second three-dimensional point cloud to a three-dimensional data acquisition device.

[0036] Step 32: Based on the first relative distance and the second three-dimensional point cloud, obtain the first relative distance, the third relative distance between the upper surface of the load and the reference plane, and the fourth relative distance between the upper surface of the object and the reference plane, and mark the first relative distance, the third relative distance and the fourth relative distance on the second working area image accordingly.

[0037] Step 33: Obtain device parameters of the image acquisition device, and calculate the data acquisition range of the image acquisition device based on the first relative distance and the device parameters.

[0038] Step 34, obtain the length of the crane boom, take the fulcrum for the left and right movement of the crane boom as the first origin, generate a first trajectory line for the left and right movement of the crane boom based on the length of the crane boom, and draw the first trajectory line within the data acquisition range on the second working area image. At the same time, obtain the vertical projection line from the hook to the reference plane based on the projection point, and generate a second trajectory line for the forward and backward movement of the crane boom based on the first origin and the vertical projection line, and draw the second trajectory line within the data acquisition range on the second working area image.

[0039] Specifically, the horizontal viewing angle, vertical viewing angle, acceleration, etc. when the image acquisition device and the three-dimensional data acquisition device acquire data are obtained through the measuring device. The coordinates of the first three-dimensional point cloud are adjusted based on the geometric correction parameters to eliminate errors caused by the scanning angle, etc., to improve the precision and accuracy of the three-dimensional point cloud data, and to provide a reliable data basis for subsequent operation area analysis and early warning judgment.

[0040] Specifically, the reference plane is the plane where the load is placed, which can be the ground or a plane such as a roof.

[0041] The data acquisition range of the image acquisition device is the area that the operator can directly observe. By marking the relative distances between the lower surface of the load and the reference plane, the upper surface of the load and the reference plane, and the upper surface of the ground object and the reference plane within the data acquisition range of the image acquisition device, and marking the first trajectory line representing the left and right movement of the crane and the second trajectory line representing the forward and backward movement, the operator can intuitively understand the movement range of the crane and the potential path of the load during the rotation process. This intuitive visual information enhances the visualization effect of the early warning system, helps the operator to better plan and execute lifting operations, thereby avoiding collisions with surrounding obstacles and ensuring the safety of the operation.

[0042] In a specific embodiment, the second relative distance is calculated as follows: Step 311: Set a plurality of longitudinal first grids based on the area of ​​the working area, and divide the point cloud corresponding to any first grid into any first grid.

[0043] Step 312, extract any first grid, calculate the first vertical distance between each scanning point in any first grid and the three-dimensional data acquisition device, and after traversing all scanning points, extract the scanning point corresponding to the maximum value of the first vertical distance and define it as the first reference point.

[0044] Step 313: Calculate the vertical distance difference between any scanning point and the first reference point. When the vertical distance difference is less than or equal to a third preset value, define any scanning point as the second reference point.

[0045] Step 314: After traversing all scanning points, a first reference point group is generated based on the first reference point and all second reference points, and an average value of the first vertical distances of all reference points in the first reference point group is used as a reference distance of any first grid.

[0046] Step 315: After traversing all first grids, the minimum reference distance is used as the second relative distance.

[0047] The third preset value is set according to the experience of those skilled in the art or according to the actual application scenario, and the embodiment of the present application is not limited to this.

[0048] Specifically, the three-dimensional point cloud data is vertically divided into a plurality of grid point clouds of the same shape and the same upper surface area. The first reference point corresponding to the maximum value of the first vertical distance is the scanning point closest to the ground in the first grid, and the scanning points whose vertical distance difference with the first reference point is less than or equal to the third preset value are considered to be scanning points located on the same horizontal plane, that is, the scanning points in the first reference point group are scanning points on the bottom horizontal plane of the first grid, and the average value of the first vertical distance between the first reference point and the second reference point in the first reference point group is used as the reference distance of the bottom horizontal plane of the first grid.

[0049] By setting multiple longitudinal first grids and performing detailed analysis on the point cloud in each grid, the point cloud data can be divided and processed more finely by calculating the vertical distance between the scanning point and the three-dimensional data acquisition device, extracting reference points, etc., thereby improving the accuracy of calculating the second relative distance, which helps to more accurately reflect the height changes and terrain characteristics in the operating area. In the calculation process, by extracting the first reference point group and generating the second relative distance based on the first reference point group, the influence of the error of a single scanning point on the overall calculation result can be effectively reduced, the robustness of the data is enhanced, and the calculated relative distance is more stable and reliable.

[0050] The technical solution of the present application can adapt to terrain conditions of varying degrees of complexity. Through grid processing and multi-point reference, it can better handle operating areas with large or irregular terrain undulations, provide technical support for the application of early warning systems in complex environments, and improve the applicability and flexibility of the system.

[0051] In a specific embodiment, step 32 includes: Step 321: setting a plurality of horizontal second grids based on the second relative distance and the preset interval, and dividing the point cloud corresponding to any second grid into any second grid.

[0052] Step 322, extract any second grid, and extract any two scanning points, calculate the point distance between any two scanning points, and determine whether the point distance is less than or equal to the fourth preset value. If not, re-acquire the two scanning points and repeat the determination of step 322. If so, define any two scanning points as the second reference point group, and then enter step 323.

[0053] Step 323, calculate the geometric center of the second reference point group, and determine whether there is a first scanning point in any second grid whose point distance to the geometric center is less than or equal to a fourth preset value. If so, generate a new second reference point group based on the second reference point group and the first scanning point, and then return to step 323. If not, determine whether the point group division is completed for all scanning points in any second grid. If so, proceed to step 324. If not, return to step 322.

[0054] Step 324: cluster all the scan points in the second reference point group respectively to obtain multiple point sets, extract any point set, calculate the point-to-surface distance between any scan point in any point set and the reference plane, and take the average of all point-to-surface distances as the fifth relative distance of any point set.

[0055] Step 325, after traversing all the second grids, extract any two adjacent point sets belonging to different second grids, calculate the inter-set distance of the two point sets based on the fifth relative distance, and determine whether the inter-set distance is less than or equal to the fifth preset value. If so, proceed to step 326, if not, return to step 325.

[0056] Step 326 , determine whether the two point sets overlap in the vertical direction. If so, determine that the two point sets belong to the same object. If not, return to step 325 .

[0057] Step 327: After traversing all point sets, all point sets are divided into different objects, and an identifier is assigned to each object. At the same time, the load and ground objects are identified based on the division results.

[0058] Step 328, extract the first identifier corresponding to the load, and extract all point sets corresponding to the first identifier, define them as the first set, take the maximum value of the fifth relative distance corresponding to the first set as the second relative distance, and the minimum value of the fifth relative distance as the first relative distance, and at the same time extract the second identifier corresponding to any ground feature, and extract all point sets corresponding to the second identifier, define them as the second set, and take the maximum value of the fifth relative distance corresponding to the second set as the fourth relative distance.

[0059] The preset interval, the fourth preset value, and the fifth preset value are set according to the experience of those skilled in the art or according to actual application scenarios, and the embodiments of the present application are not limited to this.

[0060] Specifically, the three-dimensional point cloud data is layered according to preset intervals and divided into a plurality of horizontal grid point clouds.

[0061] Specifically, there may be multiple horizontal planes (i.e., horizontal planes belonging to different objects) in any grid point cloud. Any two scanning points are extracted according to spatial proximity. When the point distance between any two scanning points is less than the fourth preset value, the two scanning points are considered to be located on the same plane. Subsequently, any two scanning points are defined as a second reference point group, and scanning points belonging to the same plane are searched again based on the geometric center of the second reference point group. A single scanning point may not be able to accurately represent the true position of the plane due to measurement errors or noise. Finding scanning points belonging to the same plane through the geometric center can effectively reduce random errors, more stably reflect the center position of the plane, and more accurately classify neighboring points into the correct plane, especially when the point distribution is uneven or there are multiple planes intersecting, thereby improving the accuracy and robustness of plane recognition. By estimating the scanning points belonging to the same plane through the above method, the upper and lower surfaces of the load and the upper surface of the object can be quickly identified, reducing the amount of calculation and improving the calculation efficiency.

[0062] When the distance between two adjacent point sets belonging to different second grids is less than or equal to the fifth preset value and the two point sets overlap in the vertical direction, it means that the two point sets belong to the same object. After traversing all point sets, all point sets belonging to the same object are identified. When the object is not connected to the reference plane, the object is a load. When the object is connected to the reference plane, the object is a ground object. Further, the relative distance between the load and the ground object and the reference plane is calculated based on the fifth relative distance of the point set corresponding to each object. Through the above method, the relative distance between the object and the reference plane can be counted without using the normal vector of the three-dimensional point cloud, which reduces the amount of calculation and improves the calculation efficiency.

[0063] By setting multiple horizontal second grids and performing detailed clustering and analysis on the point cloud in each grid, different objects in the working area can be more accurately identified and divided, which helps to distinguish between loads and ground objects. At the same time, by calculating the geometric center and point-to-surface distance of the point set, as well as the distance between the point sets, the relative position relationship between the object and the reference plane can be accurately determined. This technical solution can adapt to working environments with objects of different shapes and sizes and different load characteristics. Through grid processing and cluster analysis, it can better handle working areas with large or irregular terrain, improve the adaptability, flexibility and robustness of the system, and ensure stable operation even in complex environments.

[0064] In a specific embodiment, a calibration device is arranged on the hook fixing device, the first calibration device is on the same horizontal line as the hook fixing point, the second calibration device is fixed outside the first calibration device and on the same horizontal line as the first calibration device, the third calibration device is fixed below the first calibration device, and a data acquisition device is arranged on the crane. In step 34, the method for obtaining the crane boom length is: Step 341: set a three-dimensional coordinate system with the data acquisition device as the second origin, and respectively obtain the first relative coordinates, the second relative coordinates and the third relative coordinates of the first calibration device, the second calibration device and the third calibration device in the three-dimensional coordinate system.

[0065] Step 342: Generate a correction matrix based on the first relative coordinate, the second relative coordinate, and the third relative coordinate.

[0066] Step 343: extract the fourth relative coordinate of the hook fixing point relative to the first calibration device, and generate a first vector based on the fourth relative coordinate.

[0067] Step 344: define the product of the correction matrix and the first vector as a second vector, and obtain the fifth relative coordinate of the hook fixing point in the three-dimensional coordinate system based on the second vector.

[0068] Step 345: Obtain the sixth relative coordinate of the first origin relative to the second origin, and calculate the crane boom length based on the fifth relative coordinate and the sixth relative coordinate.

[0069] Specifically, the data acquisition device is arranged above the cockpit, and the calibration device can be observed without obstruction. The first calibration device, the hook fixing point, and the second calibration device are located on the same horizontal line parallel to the horizontal plane. The third calibration device and the first calibration device are located on the same vertical line pointing downward, and the horizontal line and the vertical line intersect vertically. The data acquisition device has the function of measuring distance and direction angles (pitch angle, roll angle, rotation angle), and obtains the first relative coordinates, second relative coordinates, and third relative coordinates of the first calibration device, the second calibration device, and the third calibration device in the three-dimensional coordinate system based on the measurement results. Preferably, the relative coordinates are the relative coordinates of the center point of the calibration device.

[0070] The first vector, the second vector and the third vector of the first calibration device, the second calibration device and the third calibration device relative to the second origin are obtained based on the first relative coordinate, the second relative coordinate and the third relative coordinate respectively, and the first vector is calculated based on the first vector and the second vector. The calculation formula is: , the first vector is calculated based on the first vector and the second vector, and the calculation formula is: , where VE1 is the first vector, VE2 is the second vector, V1 is the first vector, V2 is the second vector, and V3 is the third vector. For the modular operation, the first vector and the second vector are then subjected to a vector product operation to generate a third vector. The correction matrix is ​​generated based on the first vector, the second vector and the third vector, and the column vector of the correction matrix is ​​composed of the first vector, the second vector and the third vector.

[0071] The hook fixing point is located inside the hook fixing device, directly above the hook, and the relative coordinates of the hook fixing point in the three-dimensional coordinate system cannot be directly obtained through the measurement of the data acquisition device. The fourth relative coordinate of the hook fixing point relative to the first calibration device is a fixed value pre-set according to the design data, and the fourth relative coordinate is converted into a fifth relative coordinate in the three-dimensional coordinate system through the correction matrix.

[0072] Specifically, the position of the data acquisition device is fixed, and the fulcrum for the left and right movement of the crane arm is fixed. The sixth relative coordinate of the first origin relative to the second origin can be obtained according to the design data, and the length of the crane arm and the horizontal distance between the hook fixed point and the left and right movement fulcrum of the crane arm can be further calculated. The horizontal distance is used as the radius of the left and right movement range of the crane arm, that is, the radius of the first trajectory line.

[0073] According to the technical solution of the present application, positions that cannot be reached by direct measurement can be indirectly measured, and the position of the hook fixing point can be accurately determined when the boom is bent by factors such as load, deadweight, wind, etc., which can overcome the limitations of direct line of sight or direct measurement, and achieve accurate monitoring of the crane boom length and boom movement radius, providing reliable data support for the safe use and operation of the crane.

[0074] Step 4: Identify the ground feature on the image of the first working area. If the ground feature is a human body, proceed to step 5; if the ground feature is an object, proceed to step 6.

[0075] Preferably, the ground objects within the preset range of the projection point are identified, and there is no need to identify and judge all the ground objects on the working area image, thereby reducing the amount of computational data.

[0076] By identifying the ground objects on the image of the first operating area, two different ground object types, human body and objects, are distinguished, and the early warning strategy is flexibly adjusted according to different ground object types to improve the accuracy and effectiveness of the early warning system.

[0077] Step 5: Set a protection area based on the monitoring data, and calculate the first horizontal distance between the projection point of the human body and the hook on the reference plane. Based on the first horizontal distance, determine whether the human body is within the protection area. If so, issue a first warning message.

[0078] During the load lifting process, due to the influence of the load itself or environmental factors, if the load falls, there will be different sizes of impact range. Based on the monitoring data, a protection area is set and real-time monitoring is carried out to determine whether the human body is within the protection area, and the potential dangerous distance between the human body and the hook is discovered in time. Before the dangerous situation occurs, an early warning is issued in time to remind the operator to take safety measures, thereby effectively avoiding the occurrence of collision accidents, enhancing the reliability and effectiveness of monitoring, and reducing the possibility of false alarms and missed alarms.

[0079] Preferably, the protection area is drawn on the area annotation map, which helps the operator to issue reminders to users approaching the protection area in advance based on observation.

[0080] Preferably, when the first warning information is issued, a safety locking signal or a deceleration signal is sent to the controller of the crane to control the crane to lock or decelerate.

[0081] In a specific embodiment, the monitoring data includes load characteristic parameters and environmental parameters. In step 5, the method for setting the protection area is: based on the first relative distance, environmental parameters and load characteristic parameters, the protection range corresponding to the load is calculated, and then the protection area is set with the projection point as the center of the circle.

[0082] Specifically, the load characteristic parameters include but are not limited to volume data and weight data, and the environmental parameters include but are not limited to wind speed and wind direction.

[0083] The higher the height of the load and the heavier the weight, the larger the scattering area or dust range when it falls, and a larger protection area needs to be set up; the greater the wind speed, the larger the scattering area or dust range when it falls, and a larger protection area needs to be set up; and the scattering area or dust direction when the load falls will change with the wind direction, so the wind direction should be considered when setting the protection area.

[0084] Flexible adjustment of the protection area based on monitoring data can more accurately determine whether the human body is in a dangerous area, issue early warning information in a timely manner, and remind operators to take corresponding safety measures, thereby effectively preventing potential safety accidents, improving the accuracy and effectiveness of early warnings, and the safety of the entire operation process.

[0085] In a specific embodiment, the horizontal distance is calculated as follows: The pixel coordinates of any ground feature closest to the load are defined as the second coordinate point, and the pixel distance between the second coordinate point and the projection point is obtained; The relative distance between any object and the reference plane and the device parameters of the image acquisition device are obtained, the pixel size of one pixel is obtained based on the device parameters, the first relative distance and the fourth relative distance of any object, and the horizontal distance is calculated based on the pixel spacing and the pixel size.

[0086] The image of the working area is formed by merging the image data taken from a bird's-eye view. The pixel size of any coordinate point in the image is related to the parameters of the image acquisition device, the height of the image acquisition device from the reference plane, and the height of the object relative to the reference plane. By defining the coordinates of the pixel point closest to the load as the second coordinate point, and combining the pixel spacing, the relative distance between the object and the reference plane, and the device parameters of the image acquisition device, the minimum horizontal distance between the object and the projection point can be accurately calculated, and the relative relationship between the human body and the protection area, as well as the potential collision risk between the object and the load, can be accurately evaluated, thereby improving the reliability of the early warning system and reducing the possibility of false alarms and missed alarms.

[0087] Step 6: Calculate the second horizontal distance between the object and the projection point, and the vertical distance between the load and the object. When the second horizontal distance is less than or equal to the first preset value and the vertical distance is less than or equal to the second preset value, issue a second warning message.

[0088] The first preset value and the second preset value are set according to the experience of those skilled in the art or according to actual application scenarios, and the embodiments of the present application do not limit this.

[0089] Specifically, a fourth relative distance of the object relative to the reference plane and a first relative distance from the lower surface of the load to the reference plane are obtained, and a difference between the first relative distance and the fourth relative distance is used as a vertical distance between the load and the object.

[0090] By calculating the horizontal distance between the object and the load, as well as the vertical distance between the load and the object, the relative position relationship between the object and the load and the potential risk of collision can be accurately assessed, helping operators to better plan and adjust the work process, avoid work interruptions and equipment damage caused by collisions, and improve work efficiency.

[0091] The above describes the intelligent monitoring and early warning method for cranes in the embodiment of the present application. The following describes the intelligent monitoring and early warning system for cranes in the embodiment of the present application. Figure 2 In one embodiment of the present application, an intelligent monitoring and early warning system for cranes includes: a data acquisition module 101, an image generation module 102, an image annotation module 103, a ground feature judgment module 104, a first early warning module 105 and a second early warning module 106.

[0092] The data acquisition module 101 is used to collect monitoring data of the crane according to a preset period, wherein the monitoring data includes image data.

[0093] The image generation module 102 is used to generate a first operation area image according to the image data.

[0094] The image annotation module 103 is used to blur the hook and sling of the crane on the first working area image to generate a second working area image, and then generate an area annotation map of the working area based on the monitoring data and the second working area image, and display the area annotation map on the crane display screen 200, wherein the area annotation image includes a first relative distance from the lower surface of the load to the reference plane.

[0095] The ground feature judgment module 104 is used to identify the ground features on the first operation area image. If the ground feature is a human body, the first warning module is entered; if the ground feature is an object, the second warning module is entered.

[0096] The first warning module 105 is used to set a protection area according to the monitoring data, and calculate the first horizontal distance between the projection point of the human body and the hook on the reference plane, and judge whether the human body is located in the protection area based on the first horizontal distance. If so, a first warning message is issued.

[0097] The second warning module 106 is used to calculate the second horizontal distance between the object and the projection point, and the vertical distance between the load and the object, and issue a second warning message when the second horizontal distance is less than or equal to the first preset value and the vertical distance is less than or equal to the second preset value.

[0098] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the intelligent monitoring and early warning method for a crane.

[0099] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0100] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0101] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An intelligent monitoring and early warning method for a crane, characterized in that: The intelligent monitoring and early warning method for a crane comprises: Step 1: Collect monitoring data of the crane according to a preset period, wherein the monitoring data includes image data; Step 2: generating a first operation area image based on the image data; Step 3: setting the hook and sling of the crane in a blurred manner on the first working area image to generate a second working area image, then generating a region annotation map of the working area based on the monitoring data and the second working area image, and displaying the region annotation map on the crane display screen, wherein the region annotation image includes a first relative distance from the lower surface of the load to the reference plane; Step 4, identifying the ground feature on the first working area image, if the ground feature is a human body, proceed to step 5, if the ground feature is an object, proceed to step 6; Step 5: setting a protection area based on the monitoring data, calculating a first horizontal distance between the human body and the projection point of the hook on the reference plane, judging whether the human body is within the protection area based on the first horizontal distance, and if so, issuing a first warning message; Step 6: Calculate a second horizontal distance between the object and the projection point, and a vertical distance between the load and the object, and issue a second warning message when the second horizontal distance is less than or equal to a first preset value and the vertical distance is less than or equal to a second preset value.

2. The intelligent monitoring and early warning method for cranes according to claim 1 is characterized in that: In step 2, the method for generating the first working area image is: Step 21: extract any image data, input any of the image data into a preset model, and obtain recognition results of all elements in any of the image data; Step 22: based on the recognition result, extract the coordinates of the pixel points corresponding to the hook and the sling, and define them as first coordinate points; based on the first coordinate points, intercept a side image that does not contain the hook and the sling from any of the image data, and define it as the first image; Step 23: After traversing all the image data, generate the first working area image based on all the first images.

3. The intelligent monitoring and early warning method for cranes according to claim 1 is characterized in that: The monitoring data includes first three-dimensional point cloud data and geometric correction parameters. In step 3, generating an area annotation map of the working area based on the monitoring data and the second working area image includes: Step 31, adjusting the coordinates of the first three-dimensional point cloud based on the geometric correction parameters to generate a second three-dimensional point cloud, and then computing a second relative distance between the reference plane in the working area and the three-dimensional data acquisition device based on the second three-dimensional point cloud; Step 32: based on the first relative distance and the second three-dimensional point cloud, obtain the first relative distance, the third relative distance between the upper surface of the load and the reference plane, and the fourth relative distance between the upper surface of the object and the reference plane, and mark the first relative distance, the third relative distance and the fourth relative distance on the second working area image correspondingly; Step 33: Acquire device parameters of an image acquisition device, and calculate a data acquisition range of the image acquisition device based on the first relative distance and the device parameters; Step 34, obtain the length of the crane boom, take the fulcrum for the left and right movement of the crane boom as the first origin, generate a first trajectory line for the left and right movement of the crane boom based on the length of the crane boom, draw the first trajectory line within the data acquisition range on the second work area image, and at the same time, obtain the vertical projection line of the hook to the reference plane based on the projection point, and generate a second trajectory line for the forward and backward movement of the crane boom based on the first origin and the vertical projection line, and draw the second trajectory line within the data acquisition range on the second work area image.

4. The intelligent monitoring and early warning method for cranes according to claim 3 is characterized in that: The calculation method of the second relative distance is: Step 311: setting a plurality of longitudinal first grids based on the area of ​​the operation area, and dividing a point cloud corresponding to any first grid into any first grid; Step 312: extract any of the first grids, calculate a first vertical distance between each scanning point in any of the first grids and the three-dimensional data acquisition device, and after traversing all scanning points, extract a scanning point corresponding to the maximum value of the first vertical distance and define it as a first reference point; Step 313, calculating a vertical distance difference between any scanning point and the first reference point, and when the vertical distance difference is less than or equal to a third preset value, defining any scanning point as a second reference point; Step 314: after traversing all the scanning points, generate a first reference point group based on the first reference point and all the second reference points, and use the average of the first vertical distances of all the reference points in the first reference point group as the reference distance of any of the first grids; Step 315: After traversing all first grids, the minimum reference distance is used as the second relative distance.

5. The intelligent monitoring and early warning method for cranes according to claim 3 is characterized in that: The step 32 comprises: Step 321: setting a plurality of second horizontal grids based on the second relative distance and a preset interval, and dividing a point cloud corresponding to any second grid into any second grid; Step 322: extract any second grid, and extract any two scanning points, calculate the point distance between any two scanning points, and determine whether the point distance is less than or equal to a fourth preset value. If not, reacquire two scanning points and repeat the determination of step 322. If yes, define any two scanning points as a second reference point group, and then proceed to step 323. Step 323, calculating the geometric center of the second reference point group, determining whether there is a first scanning point in any of the second grids whose point distance to the geometric center is less than or equal to the fourth preset value, if so, generating a new second reference point group based on the second reference point group and the first scanning point, and then returning to step 323, if not, determining whether point group division is completed for all scanning points in any of the second grids, if so, proceeding to step 324, if not, returning to step 322; Step 324: clustering all the scan points in the second reference point group respectively to obtain multiple point sets, extracting any point set, calculating the point-to-surface distance between any scan point in any of the point sets and the reference plane, and taking the average of all the point-to-surface distances as the fifth relative distance of any of the point sets; Step 325, after traversing all the second grids, extract any two adjacent point sets belonging to different second grids, calculate the inter-set distance of the two point sets based on the fifth relative distance, and determine whether the inter-set distance is less than or equal to the fifth preset value, if so, proceed to step 326, if not, return to step 325; Step 326: determine whether the two point sets overlap in the vertical direction. If so, determine that the two point sets belong to the same object. If not, return to step 325. Step 327: After traversing all point sets, divide all the point sets into different objects, assign an identifier to each object, and identify the load and ground objects based on the division results; Step 328, extract the first identifier corresponding to the load, and extract all point sets corresponding to the first identifier, define them as the first set, and use the maximum value of the fifth relative distance corresponding to the first set as the second relative distance, and the minimum value of the fifth relative distance as the first relative distance. At the same time, extract the second identifier corresponding to any ground feature, and extract all point sets corresponding to the second identifier, define them as the second set, and use the maximum value of the fifth relative distance corresponding to the second set as the fourth relative distance.

6. The intelligent monitoring and early warning method for cranes according to claim 3 is characterized in that: A calibration device is arranged on the hook fixing device, the first calibration device is on the same horizontal line as the hook fixing point, the second calibration device is fixed outside the first calibration device and on the same horizontal line as the first calibration device, the third calibration device is fixed below the first calibration device, and a data acquisition device is arranged on the crane. In the step 34, the method for obtaining the boom length of the crane is: Step 341: set a three-dimensional coordinate system with the data acquisition device as a second origin, and respectively obtain first relative coordinates, second relative coordinates, and third relative coordinates of the first calibration device, the second calibration device, and the third calibration device in the three-dimensional coordinate system; Step 342: Generate a correction matrix based on the first relative coordinate, the second relative coordinate and the third relative coordinate; Step 343: extract the fourth relative coordinate of the hook fixing point relative to the first calibration device, and generate a first vector based on the fourth relative coordinate; Step 344: define the product of the correction matrix and the first vector as a second vector, and obtain the fifth relative coordinate of the hook fixing point in the three-dimensional coordinate system based on the second vector; Step 345: Obtain a sixth relative coordinate of the first origin relative to the second origin, and calculate the crane boom length based on the fifth relative coordinate and the sixth relative coordinate.

7. The intelligent monitoring and early warning method for cranes according to claim 1 is characterized in that: The monitoring data includes load characteristic parameters and environmental parameters. In step 5, the method for setting the protection area is: based on the first relative distance, the environmental parameters and the load characteristic parameters, the protection range corresponding to the load is calculated, and then the protection area is set with the projection point as the center of the circle.

8. The intelligent monitoring and early warning method for cranes according to claim 3 is characterized in that: The calculation method of horizontal distance is: The pixel point coordinates of any ground feature closest to the load are defined as a second coordinate point, and the pixel distance between the second coordinate point and the projection point is obtained; Obtain the relative distance between any of the above-mentioned objects and the reference plane, and the device parameters of the image acquisition device, obtain the pixel size of one pixel based on the device parameters, the first relative distance and the fourth relative distance of any of the above-mentioned objects, and calculate the horizontal distance based on the pixel spacing and the pixel size.

9. An intelligent monitoring and early warning system for cranes, characterized in that: The intelligent monitoring and early warning system for cranes comprises: a data acquisition module, an image generation module, an image annotation module, a ground object judgment module, a first early warning module and a second early warning module; The data acquisition module is used to collect monitoring data of the crane according to a preset period, wherein the monitoring data includes image data; The image generation module is used to generate a first operation area image according to the image data; The image annotation module is used to set the hook and the sling of the crane in a blurred manner on the first working area image to generate a second working area image, then generate a region annotation map of the working area based on the monitoring data and the second working area image, and display the region annotation map on the crane display screen, wherein the region annotation image includes a first relative distance from the lower surface of the load to the reference plane; The ground object judgment module is used to identify the ground object on the first operation area image, and if the ground object is a human body, enter the first early warning module; if the ground object is an object, enter the second early warning module; The first warning module is used to set a protection area according to the monitoring data, and calculate a first horizontal distance between the human body and the projection point of the hook on the reference plane, and determine whether the human body is located in the protection area based on the first horizontal distance, and if so, issue a first warning message; The second warning module is used to calculate the second horizontal distance between the object and the projection point, and the vertical distance between the load and the object, and issue a second warning message when the second horizontal distance is less than or equal to the first preset value and the vertical distance is less than or equal to the second preset value.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the intelligent monitoring and early warning method for a crane as described in any one of claims 1-8 is implemented.

Images