Intelligent inspection method and system for gantry cranes based on BIM and drones

Abstract

The embodiment of the present application provides a method and system for intelligent inspection of gantry cranes based on BIM and drones. The method includes: determining the gantry crane BIM model of the gantry crane and the oblique photography model of the construction site environment where the gantry crane is located; performing three-dimensional space alignment, model voxelization processing and axis-aligned bounding box calculation based on the BIM model and the oblique photography model to determine the inspection area; performing collision detection on the inspection area through the inspection object to determine the inspection viewpoint and line of sight direction; performing path planning based on the inspection viewpoint and the obstacle distribution information corresponding to the inspection viewpoint to obtain the optimal inspection path; generating an inspection task based on the inspection object, line of sight direction and the optimal inspection path, so that the drone collects gantry crane data based on the inspection task, and generates gantry crane inspection results based on the gantry crane data returned by the drone. The above scheme can improve the inspection efficiency and inspection results of gantry cranes.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent detection of construction equipment, and specifically to an intelligent inspection method and system for gantry cranes based on BIM and drones. Background Art

[0002] At traditional construction sites, gantry cranes, as part of the larger equipment, require special attention to prevent potential safety hazards. Currently, inspections rely primarily on manual labor, which presents challenges such as high-altitude work risks, low inspection frequency, and delayed inspection data. Large gantry cranes, with their complex structures and wide operating range, make manual inspections difficult to cover key areas and inefficient. This makes comprehensive monitoring of gantry cranes difficult, and the accuracy of inspection results is low.

[0003] Related technologies combine BIM (Building Information Modeling) technology with drones to enable inspections of gantry cranes. For example, they can monitor the structure and operating status of gantry cranes, improving the comprehensiveness and accuracy of monitoring compared to manual methods while also reducing the risks and costs of manual inspections. However, these inspections of gantry cranes are based solely on image recognition and comparison, without optimizing the drone's inspection space or routing. This can lead to inaccurate data and / or repeated inspections due to obstacles encountered during flight, reducing the efficiency and accuracy of gantry crane inspections. Summary of the Invention

[0004] The embodiments of the present application hope to provide a BIM- and drone-based intelligent inspection method and system for gantry cranes, which can improve the inspection efficiency and inspection results of gantry cranes.

[0005] The technical solution of this application is achieved as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for intelligent inspection of gantry cranes based on BIM and drones, the method comprising:

[0007] Determining a gantry crane BIM model of a gantry crane and an oblique photography model of the construction site environment in which the gantry crane is located; wherein the gantry crane BIM model is refined down to individual components of the gantry crane;

[0008] Based on the gantry crane BIM model and the oblique photography model, three-dimensional space alignment, model voxelization, and axis-aligned bounding box calculation are performed to determine an initial inspection space; and a Boolean difference operation is performed based on the initial inspection space and the oblique photography model to determine an inspection area;

[0009] Performing collision detection on the inspection area through a predetermined inspection object to determine an initial inspection viewpoint; and determining the inspection viewpoint and line of sight direction by calculating the pitch angle, azimuth angle, and line of sight clustering of the initial inspection viewpoint; wherein the inspection object is at least one component of the gantry crane;

[0010] Based on the inspection viewpoint and the obstacle distribution information corresponding to the inspection viewpoint, path planning is performed using the A* algorithm to obtain an initial inspection path; and the node access sequence of the initial inspection path is optimized using the ant colony algorithm to obtain an optimal inspection path;

[0011] Based on the inspection object, the line of sight direction and the optimal inspection path, an inspection task is generated so that the drone flies according to the line of sight direction and the optimal inspection path in the inspection task and collects gantry crane data, and a gantry crane inspection result is generated based on the gantry crane data returned by the drone.

[0012] In the above solution, the three-dimensional space alignment, model voxelization and axis-aligned bounding box calculation based on the gantry crane BIM model and the oblique photography model to determine the initial inspection space include:

[0013] Obtaining three-dimensional coordinate information of multiple reference points in the gantry crane BIM model;

[0014] Based on the three-dimensional coordinate information, calculating a conversion matrix between the coordinates of the gantry crane BIM model and the coordinates of the oblique photography model;

[0015] Performing three-dimensional spatial alignment on the gantry crane BIM model and the oblique photography model using the conversion matrix to obtain the gantry crane BIM model aligned with the oblique photography model;

[0016] Based on the aligned gantry crane BIM model, obtaining the three-dimensional spatial coordinate information of the gantry crane, calculating the minimum and maximum values of the gantry crane in the three axes of X, Y, and Z, and determining the axis-aligned bounding box of the gantry crane;

[0017] Based on the axis-aligned bounding box corresponding to the gantry crane, expansion processing is performed to determine the preliminary inspection space.

[0018] In the above solution, the expansion process is performed based on the axis-aligned bounding box corresponding to the gantry crane to determine the preliminary inspection space, including:

[0019] Performing a first dilation process based on the axis-aligned bounding box corresponding to the gantry crane to obtain a first spatial bounding box; wherein the first dilation process is used to increase the safety distance;

[0020] Performing a second dilation process on the first spatial bounding box to obtain a second spatial bounding box; wherein the spatial range enlarged by the second dilation process is within a preset imaging area of the drone camera;

[0021] The preliminary inspection space is determined based on the axis-aligned bounding box corresponding to the gantry crane and the second space bounding box.

[0022] In the above solution, performing a Boolean difference operation based on the initial inspection space and the oblique photography model to determine the inspection area includes:

[0023] Performing voxelization and spatial expansion on the oblique photography model to determine a voxelized model; wherein the voxelized model includes a plurality of voxel grids;

[0024] performing a Boolean difference operation on the three-dimensional voxel grid of the preliminary inspection space and the multiple voxel grids of the voxelized model to determine an obstacle area;

[0025] The obstacle area is removed from the preliminary inspection space to determine the inspection area.

[0026] In the above scheme, the initial inspection viewpoint is determined by performing collision detection between the determined inspection object and the inspection area; and the inspection viewpoint and line of sight direction are determined by calculating the pitch angle, azimuth angle and line of sight clustering of the initial inspection viewpoint, including:

[0027] Determining the inspection object of the drone;

[0028] Based on the inspection object, emitting rays into the inspection space, and performing ray tracing analysis and collision detection on the inspection object to determine blocked rays;

[0029] Eliminating the blocked rays from the emitted rays, performing collision analysis on the remaining rays and the voxel grid of the inspection space, and determining the initial inspection viewpoint; wherein the initial inspection viewpoint is the viewpoint corresponding to the same voxel unit hit by rays emitted from different sources;

[0030] Calculating the pitch angle and azimuth angle of the initial inspection viewpoint, and determining the initial inspection viewpoint that meets the rotation range of the UAV gimbal as the inspection viewpoint;

[0031] Perform sight line clustering and direction merging on the inspection viewpoints to determine the sight line direction.

[0032] In the above scheme, the path planning is performed using the A* algorithm based on the inspection viewpoint and the obstacle distribution information corresponding to the inspection viewpoint to obtain an initial inspection path; and the node access sequence of the initial inspection path is optimized using the ant colony algorithm to obtain the optimal inspection path, including:

[0033] Adding an obstacle avoidance turning point for flight based on obstacle distribution information corresponding to the patrol viewpoint and the inspection viewpoint;

[0034] Determine the path nodes based on the patrol viewpoint and the obstacle avoidance turning point; and calculate the shortest path and distance matrix between each node in the path nodes by the A* algorithm to determine the initial patrol path;

[0035] The node access sequence of the initial inspection path is optimized by using the ant colony algorithm and the distance matrix to determine the optimal inspection path.

[0036] In the above solution, generating the gantry crane inspection result based on the gantry crane data returned by the drone includes:

[0037] Acquiring image data from the gantry crane data;

[0038] Performing denoising, cropping, and color balancing preprocessing on the component image in the image data to obtain a preprocessed image; wherein the component image includes at least one of a weld image, a bolt image, and a pin image;

[0039] The preprocessed image is detected by a gantry crane detection model to obtain the gantry crane inspection result; wherein the gantry crane detection model is pre-trained by a YOLOv8 model based on sample weld images, sample bolt images, sample pin shaft images and their corresponding annotated data;

[0040] After detecting the pre-processed image using the gantry crane detection model to obtain the gantry crane inspection result, the method further includes:

[0041] Matching and associating the gantry crane inspection results with corresponding components in the gantry crane inspection result BIM model to ensure that each test result data in the gantry crane inspection results accurately corresponds to the specific corresponding component in the gantry crane inspection result BIM model;

[0042] The severity of different inspection result data is visually presented on the specific corresponding components in the BIM model of the gantry crane inspection results based on a preset color scheme to intuitively display the health status of each component.

[0043] In a second aspect, an embodiment of the present application provides a BIM-based and UAV-based intelligent inspection system for gantry cranes, the BIM-based and UAV-based intelligent inspection system for gantry cranes comprising: a model building module, an inspection area generation module, an inspection viewpoint and sightline generation module, an inspection path generation module, and a flight inspection module, wherein:

[0044] The model building module is used to determine a gantry crane BIM model of the gantry crane and an oblique photography model of the construction site environment in which the gantry crane is located; wherein the gantry crane BIM model is refined to a single component of the gantry crane;

[0045] The inspection area generation module is configured to perform three-dimensional space alignment, model voxelization, and axis-aligned bounding box calculation based on the gantry crane BIM model and the oblique photography model to determine an initial inspection space; and perform a Boolean difference operation based on the initial inspection space and the oblique photography model to determine an inspection space; wherein the inspection space is the inspection area of the drone;

[0046] The inspection viewpoint and sight line generation module is configured to perform collision detection on the inspection area using a predetermined inspection object to determine an initial inspection viewpoint; and to determine the inspection viewpoint and sight line direction by calculating the pitch angle, azimuth angle, and sight line clustering of the initial inspection viewpoint; wherein the inspection object is at least one component of the gantry crane;

[0047] The inspection path generation module is used to perform path planning using the A* algorithm based on the inspection viewpoint and the obstacle distribution information corresponding to the inspection viewpoint to obtain an initial inspection path; and to optimize the node access sequence of the initial inspection path using the ant colony algorithm to obtain an optimal inspection path;

[0048] The flight inspection module is used to generate an inspection task based on the inspection object, the line of sight direction and the optimal inspection path, so that the drone flies according to the line of sight direction and the optimal inspection path in the inspection task and collects gantry crane data, and generates gantry crane inspection results based on the gantry crane data returned by the drone.

[0049] In a third aspect, an embodiment of the present application provides a gantry crane intelligent inspection device based on BIM and drones, comprising: a processor and a memory; wherein,

[0050] The memory is used to store computer programs;

[0051] The processor is configured to call and run the computer program from the memory to execute the method according to the first aspect.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the method described in the first aspect.

[0053] An embodiment of the present application provides a method and system for intelligent inspection of gantry cranes based on BIM and UAVs, the method comprising: determining a gantry crane BIM model of the gantry crane and an oblique photography model of the construction site environment in which the gantry crane is located; wherein the gantry crane BIM model is refined to a single component of the gantry crane; based on the gantry crane BIM model and the oblique photography model, performing three-dimensional space alignment, model voxelization processing and axis-aligned bounding box calculation to determine an initial inspection space; and performing a Boolean difference operation based on the initial inspection space and the oblique photography model to determine an inspection area; performing collision detection on the inspection area through a predetermined inspection object to determine an initial inspection viewpoint; and Calculate the pitch angle and azimuth angle of the initial inspection viewpoint and the line of sight clustering to determine the inspection viewpoint and line of sight direction; wherein the inspection object is at least one component of the gantry crane; based on the inspection viewpoint and the obstacle distribution information corresponding to the inspection viewpoint, perform path planning using the A* algorithm to obtain an initial inspection path; and optimize the node access sequence of the initial inspection path using the ant colony algorithm to obtain an optimal inspection path; generate an inspection task based on the inspection object, the line of sight direction, and the optimal inspection path, so that the drone flies according to the line of sight direction and the optimal inspection path in the inspection task and collects gantry crane data, and generate gantry crane inspection results based on the gantry crane data returned by the drone. In the above scheme, the gantry crane BIM model is combined with the oblique photography model of the construction site environment where the gantry crane is located, and three-dimensional space alignment, model voxelization, axis-aligned bounding box calculation, and Boolean difference operations are performed to determine the inspection space. Determining the inspection space is conducive to narrowing the drone's search range, thereby improving the efficiency of gantry crane inspections. By performing collision detection and line-of-sight clustering on the inspection space and inspection objects, and determining the inspection viewpoint and line-of-sight direction, the inspection viewpoint and line-of-sight direction can be set more precisely, and the inspection viewpoint determination is more accurate, making the data collected by the drone more accurate, thereby improving the accuracy of the inspection results. Using the A* algorithm for path planning and the ant colony algorithm for path optimization can avoid the decrease in inspection efficiency caused by repeated flights or long-path inspections, thereby improving the inspection efficiency of gantry cranes. This solution can simultaneously improve the inspection efficiency of gantry cranes and the accuracy of gantry crane inspection results, thereby facilitating the timely detection of potential quality issues in gantry cranes and preventing gantry crane safety accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings herein are incorporated into and constitute a part of this specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, serve to illustrate the technical solutions of the present application. Obviously, the drawings described below are merely some embodiments of the present application. Those skilled in the art can, without inventive effort, derive other drawings from these drawings.

[0055] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0056] Figure 1 An optional flowchart of a method for intelligent inspection of gantry cranes based on BIM and drones provided in an embodiment of the present application;

[0057] Figure 2 A schematic diagram of an inspection object for a gantry crane intelligent inspection method based on BIM and drones provided in an embodiment of the present application;

[0058] Figure 3 A schematic diagram of a framework of a gantry crane intelligent inspection system based on BIM and drones provided in an embodiment of the present application;

[0059] Figure 4 A voxelized schematic diagram of an optional oblique photography model for a BIM- and drone-based intelligent inspection method for gantry cranes provided in an embodiment of the present application;

[0060] Figure 5 A schematic diagram of a three-dimensional voxel grid of an optional inspection area for a BIM- and drone-based intelligent inspection method for gantry cranes provided in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of an optional viewpoint preliminary selection logic for a gantry crane intelligent inspection method based on BIM and drones provided in an embodiment of the present application;

[0062] Figure 7 A schematic diagram of an optional inspection planning result of a gantry crane intelligent inspection method based on BIM and drones provided in an embodiment of the present application;

[0063] Figure 8 A schematic diagram of the structure of a gantry crane intelligent inspection system based on BIM and drones provided in an embodiment of the present application;

[0064] Figure 9A structural schematic diagram of an intelligent inspection device for gantry cranes based on BIM and drones provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0067] In the following description, references to “some embodiments,” “this embodiment,” “embodiments of the present application,” and examples, etc., describe a subset of all possible embodiments. However, it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

[0068] If similar descriptions of "first / second" appear in the application documents, the following explanation is added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0069] The solution of the embodiment of the present application aims to realize intelligent inspection of gantry cranes at construction sites by combining BIM, oblique photography modeling and drone technology. Oblique photography modeling technology can efficiently construct three-dimensional real-scene models, and has the technical advantages of fast data acquisition and high modeling accuracy. The BIM model contains the complete design geometry and structural information of the crane, and can digitally express the component form and functional characteristics of the gantry crane. Drones have the unique advantages of strong high-altitude maneuverability, flexible observation from multiple perspectives, and contact-free detection in high-risk areas. Based on the fusion of BIM and oblique photography models, combined with the drone intelligent inspection system, high-precision automated inspection of gantry cranes is achieved, solving the problems of low efficiency and large safety hazards of traditional manual inspections.

[0070] Based on this, the embodiment of the present application provides a gantry crane intelligent inspection method based on BIM and drones. Figure 1 An optional flow chart of an intelligent inspection method for gantry cranes based on BIM and drones provided in the embodiment of the present application will be combined with Figure 1The steps shown are explained.

[0071] S101. Determine the gantry crane BIM model of the gantry crane and the oblique photography model of the construction site environment where the gantry crane is located; wherein the gantry crane BIM model is refined to the individual components of the gantry crane.

[0072] In some embodiments of the present application, BIM (Building Information Modeling) technology is the most advanced modeling method currently used in the construction industry, achieving the highest degree of digitalization. This method enables information integration and sharing throughout the building lifecycle, improving collaborative efficiency at various stages, including design, construction, and maintenance. BIM models contain rich design geometry and semantic information, enabling digital representation of the physical and functional characteristics of target objects. The rapid development of Unmanned Aerial Vehicle (UAV) technology has provided construction sites with an efficient means of data collection, enabling the acquisition of large-scale, high-precision data in a short period of time, providing strong support for intelligent engineering management.

[0073] In some embodiments of this application, BIM technology can provide a detailed 3D model and structural parameters of the gantry crane, laying the data foundation for intelligent inspections. The oblique photography model uses drone aerial photography to generate high-precision, real-world 3D data of the construction site, accurately reflecting environmental information such as the distribution of surrounding obstacles and terrain changes. By integrating the BIM model with the oblique photography model, the system can intelligently generate 3D drone inspection routes, effectively mitigating potential risks in the operating environment.

[0074] In some embodiments of the present application, during the inspection process, the drone is equipped with a high-resolution optical camera and dedicated sensors.

[0075] In some embodiments of the present application, the intelligent inspection method for gantry cranes based on BIM and drones is applicable to the intelligent inspection system for gantry cranes based on BIM and drones.

[0076] For example, BIM-based inspection object data management first requires the creation of a detailed Revit model of a gantry crane, containing complete geometric information (component dimensions and spatial location) and non-geometric information (component unique identification codes and material properties). This level of detail extends down to individual components, such as welds, bolts, and pins. The Dynamo visual programming tool is used to achieve dual data linkage for automatic component ID extraction and geometric feature analysis, building a structured component information database. The research specifically focuses on data collection at three key connection locations: welds, bolted connections, and pins, establishing a precise and reliable target feature library for subsequent autonomous drone inspections.

[0077] S102. Based on the gantry crane BIM model and the oblique photography model, perform three-dimensional space alignment, model voxelization processing, and calculation of the axis-aligned bounding box to determine the initial inspection space; and perform Boolean difference operation based on the initial inspection space and the oblique photography model to determine the inspection area.

[0078] In some embodiments of the present application, the inspection area, also referred to as the inspection space, is the area where the drone collects image data of the inspection object of the gantry crane.

[0079] In some embodiments of the present application, the three-dimensional coordinate information and physical location of multiple reference points in the gantry crane BIM model are obtained. The conversion matrix between the coordinates of the gantry crane BIM model and the coordinates of the oblique photography model is calculated, and the gantry crane BIM model and the oblique photography model are aligned in three dimensions to achieve spatial registration. The oblique photography model is voxelized and spatially expanded to determine the voxelized model. The three-dimensional spatial coordinate information of the gantry crane is obtained, the minimum and maximum values of the gantry crane in the X, Y, and Z axes are calculated, and the axis-aligned bounding box of the gantry crane is determined. Based on the axis-aligned bounding box corresponding to the gantry crane, expansion processing is performed to determine the initial inspection space.

[0080] In some embodiments of the present application, a Boolean difference operation is performed based on the preliminary inspection space and the oblique photography model to determine the obstacle area. The obstacle area is removed from the preliminary inspection space to determine the inspection area.

[0081] S103. Perform collision detection on the inspection area through a predetermined inspection object to determine an initial inspection viewpoint; and determine the inspection viewpoint and line of sight direction by calculating the pitch angle, azimuth angle and line of sight clustering of the initial inspection viewpoint; wherein the inspection object is at least one component of the gantry crane.

[0082] In some embodiments of the present application, at least one component of a gantry crane is determined as an inspection object; an initial inspection viewpoint is determined by performing collision detection between the inspection object and the inspection area; and the inspection viewpoint and line of sight direction are determined by calculating the pitch angle, azimuth angle, and line of sight clustering of the initial inspection viewpoint.

[0083] In some embodiments of the present application, the inspection object is at least one component of the gantry crane.

[0084] For example, the inspection object may be at least one of a weld, a bolt connection, and a pin. Figure 2 As shown below:

[0085] 1) Weld

[0086] As a permanent connection for the primary load-bearing structure, the quality of welds directly determines the overall strength of the metal structure. The BIM-based weld location information extraction process uses a two-stage precision positioning method: first, plane extraction is performed on both ends of the rod, followed by edge extraction to precisely locate the weld.

[0087] 2) Bolts

[0088] As a critical connecting component, its tightening state directly affects the safety and stability of the overall structure. The BIM-based bolt location information extraction process uses a geometric feature quantification method: first, an axis-aligned bounding box (AABB) model of the bolt is established to determine the detection area. The upper endpoint (the center of the top surface of the bolt head) and the lower endpoint (the reference plane at the end of the thread) are then precisely located to precisely locate the bolt position.

[0089] 3) Pin

[0090] As a core component of articulated joints, the pin's geometric accuracy is crucial to the mechanism's kinematic performance. The BIM-based pin information extraction process adheres to the principle of geometric feature quantification: First, a cylinder fitting algorithm is used to reconstruct the pin's basic geometric features. Morphological analysis is then used to determine the key parameters of the pin's end (larger area) and open end (smaller area), precisely determining the pin's position.

[0091] S104 , based on the inspection viewpoint and the obstacle distribution information corresponding to the inspection viewpoint, perform path planning using the A* algorithm to obtain an initial inspection path; and optimize the node access sequence of the initial inspection path using the ant colony algorithm to obtain an optimal inspection path.

[0092] In some embodiments of the present application, obstacle avoidance turning points are added to the flight process based on obstacle distribution information corresponding to the patrol viewpoint and the inspection viewpoint. Path nodes are determined based on the patrol viewpoint and the obstacle avoidance turning points. The A* algorithm is then used to calculate the shortest paths and distance matrix between each node in the path to determine the initial inspection path. The ant colony algorithm and the distance matrix are used to optimize the node visit sequence of the initial inspection path to determine the optimal inspection path. Inspection tasks are generated based on the inspection object and the optimal inspection path.

[0093] S105. Generate an inspection task based on the inspection object, sight direction and optimal inspection path, so that the UAV flies according to the sight direction and optimal inspection path in the inspection task and collects gantry crane data, and generates a gantry crane inspection result based on the gantry crane data returned by the UAV.

[0094] In some embodiments of the present application, an inspection task is generated based on the inspection object, line of sight direction, and optimal inspection path. The inspection task is transmitted to a drone, which then flies according to the line of sight and optimal inspection path specified in the inspection task and collects gantry crane data. Image data from the gantry crane data is acquired; component images in the image data are preprocessed by denoising, cropping, and color balancing to obtain a preprocessed image; and the preprocessed image is inspected using a gantry crane inspection model to obtain a gantry crane inspection result.

[0095] In some embodiments of the present application, the component image includes at least one of a weld image, a bolt image, and a pin image.

[0096] For example, the rapid development of drone technology has introduced efficient data collection methods for construction sites, which can obtain real-time data over a wide range in a shorter time, providing strong support for intelligent construction management. Drone technology can quickly obtain high-altitude image information, and has the unique advantages of flexibility and high efficiency. In the embodiment of the present application, BIM, oblique photography model and drone technology are combined to perform intelligent inspections of gantry cranes on construction sites. After determining the above-mentioned optimal inspection path, an inspection task is generated based on the inspection object and the optimal inspection path. The inspection task can be transmitted to the drone, and the drone flies along the optimal inspection path and collects gantry crane data. For example, it stops briefly at each path node to collect gantry crane data such as image data, and then generates gantry crane inspection results based on the gantry crane data returned by the drone. For example, the gantry crane inspection results are obtained based on status information such as rust or cracks of the components identified by the image.

[0097] For example, Figure 3 The framework of the intelligent inspection system for gantry cranes based on BIM and drones is shown, which includes the following three main parts: (1) inspection object data management and application based on BIM; (2) inspection task generation based on BIM, oblique photography model and drones; (3) inspection defect detection based on computer vision and BIM result visualization.

[0098] It can be understood that the inspection space is determined by combining the gantry crane BIM model with an oblique photographic model of the gantry crane's construction site environment, performing 3D spatial alignment, voxelization, axis-aligned bounding box calculation, and Boolean difference operations. This helps narrow the drone's search range, thereby improving gantry crane inspection efficiency. Collision detection and line-of-sight clustering between the inspection space and the inspection object, as well as determining the inspection viewpoint and line-of-sight direction, allow for more precise viewpoint setting and determination, resulting in more accurate drone data collection and improved inspection results. Path planning using the A* algorithm and path optimization using the ant colony algorithm can avoid the reduction in inspection efficiency caused by repeated or long flight paths, thereby improving gantry crane inspection efficiency. This solution can simultaneously improve gantry crane inspection efficiency and the accuracy of inspection results, facilitating timely detection of potential gantry crane quality issues and preventing gantry crane safety accidents.

[0099] In some embodiments of the present application, the three-dimensional space alignment, model voxelization, and axis-aligned bounding box calculation based on the gantry crane BIM model and the oblique photography model in S102 to determine the initial inspection space can be achieved through S201-S205 as follows:

[0100] S201. Obtain three-dimensional coordinate information of multiple reference points in the gantry crane BIM model.

[0101] In some embodiments of the present application, the three-dimensional coordinate information is the coordinate information corresponding to the physical locations in reality corresponding to the selected multiple reference points. The multiple reference points can be three reference points, four reference points, or six reference points, which are not specifically limited in the embodiments of the present application.

[0102] For example, let's use the example of four datum points. These are four key datum points (e.g., the four corners of the base) in the BIM model of a gantry crane. The 3D coordinate information for these four key datum points is extracted from the BIM coordinate system. In real physical space, the corresponding physical locations in the BIM model are found.

[0103] S202. Based on the three-dimensional coordinate information, calculate the conversion matrix between the coordinates of the gantry crane BIM model and the coordinates of the oblique photography model.

[0104] In some embodiments of the present application, after obtaining the three-dimensional coordinate information of multiple reference points in the gantry crane BIM model, the conversion matrix between the coordinates of the gantry crane BIM model and the coordinates of the oblique photography model can be calculated through the three-dimensional coordinate information of the gantry crane BIM model and the three-dimensional coordinate information of the oblique photography model.

[0105] S203. Perform three-dimensional spatial alignment on the gantry crane BIM model and the oblique photography model through a conversion matrix to obtain the gantry crane BIM model aligned with the oblique photography model.

[0106] For example, the construction of the inspection space is the basis of mission planning. To improve computing efficiency and ensure the safety of drone flight, the inspection range is limited to a certain spatial range. The BIM-oblique photography model spatial registration includes the following:

[0107] Spatial registration of the BIM-oblique photography model is a fundamental step in drone inspection mission planning. Its core purpose is to precisely align the 3D design model with the physical coordinates in the real environment to support high-precision flight path calculation and spatial obstacle avoidance analysis. The specific implementation process is as follows: First, four key reference points (such as the four corners of the base) are selected in the BIM model of the gantry crane, and their 3D coordinate information is extracted from the BIM coordinate system. Next, the physical locations corresponding to the BIM model are found in the actual physical space, and the geographic coordinates of these points are obtained using high-precision measurement equipment (such as RTK). Because the oblique photography model is reconstructed based on multi-angle aerial photography data, its spatial coordinates directly correspond to real-world geographic coordinates. Therefore, once the physical locations of these reference points are obtained, they can be used directly as a reference for alignment.

[0108] After data collection is complete, the BIM model's coordinate system is mapped into real space by calculating the transformation matrix between the BIM model coordinates and the oblique photography model coordinates. This transformation is solved using the least squares method to minimize spatial errors and ensure that the spatial deviation of the four reference points is minimized. Finally, the model is aligned through matrix transformation, aligning the BIM model and the oblique photography model in three-dimensional space. This precise alignment not only ensures the spatial consistency of the model but also provides a reliable geometric foundation for subsequent voxel processing, path obstacle avoidance, and inspection viewpoint selection, thereby improving the accuracy and safety of the overall inspection task.

[0109] S204: Acquire the three-dimensional spatial coordinate information of the gantry crane, calculate the minimum and maximum values of the gantry crane in the three axes of X, Y, and Z, and determine the axis-aligned bounding box of the gantry crane.

[0110] For example, the axis-aligned bounding box of the gantry crane, that is, the AABB calculation of the gantry crane, is calculated.

[0111] While the oblique photography model is being voxelized, the 3D spatial representation of the gantry crane requires further calculation of its AABB. The specific steps are as follows: First, after spatial registration between the BIM model and the oblique photography model, the gantry crane's 3D coordinate information is obtained. The minimum and maximum values of the gantry crane along the X, Y, and Z axes are calculated to determine the six boundary faces of the AABB (Axis-Aligned Bounding Box). This enveloping volume effectively outlines the gantry crane's 3D footprint and serves as a key reference for inspection route planning.

[0112] It should be noted that AABB stands for Axis-Aligned Bounding Box. This term is often used to describe a simple geometric structure in 3D space. Its specific characteristics are as follows:

[0113] ‌Structural Characteristics‌: An AABB is a hexahedron, with each face parallel to an axis (X, Y, Z) in a three-dimensional coordinate system. Its dimensions are determined by the maximum and minimum values of the object along the three coordinate axes, forming six bounding faces.

[0114] Application Scenarios: In crane design, AABBs can be used to simplify collision detection or spatial planning, such as defining a crane's operating range, track layout, or safe distances between components. Due to the axis-aligned nature of AABBs, computational complexity is low, making them suitable for quickly determining spatial relationships between objects.

[0115] S205. Perform expansion processing based on the axis-aligned bounding box corresponding to the gantry crane to determine the preliminary inspection space.

[0116] In some embodiments of the present application, a first expansion process is performed based on the axis-aligned bounding box corresponding to the gantry crane to obtain a first spatial bounding box; wherein, the first expansion process is used to increase the safety distance; based on the first spatial bounding box, a second expansion process is performed to obtain a second spatial bounding box; wherein, the spatial range increased by the second expansion process is within the preset imaging area of the drone camera; based on the axis-aligned bounding box corresponding to the gantry crane and the second spatial bounding box, a preliminary inspection space is determined.

[0117] It should be noted that the first dilation process is used to increase the safety distance, allowing the drone to collect data at a distance greater than the first preset distance and less than the second preset distance, where the second preset distance is greater than the first preset distance; both the first preset distance and the second preset distance are the distances between the drone and the surface of the structure. The spatial range increased by the second dilation process falls within the preset imaging area of the drone's camera, which is the area that the drone's camera can capture.

[0118] For example, the six AABB boundary surfaces of a gantry crane might be used to define a simplified 3D model of its mechanical structure or operating area, facilitating simulation analysis and safety assessment during the design phase. However, a single AABB is insufficient to completely mitigate potential risks during drone flight, such as windage, flight jitter, and error accumulation. Therefore, after initially calculating the AABB of a gantry crane, a "double expansion" process is required to improve spatial safety:

[0119] First expansion treatment: safety distance expansion

[0120] Using the gantry crane's AABB as a base, we inflate it outward by a safe distance to ensure the drone doesn't fly too close to the structure during inspections. This reduces the risk of interference from the crane's metal structure and avoids potential collisions caused by close paths.

[0121] Second expansion process: visual range

[0122] Taking the visual range of drone inspections into account, a second spatial dilation is performed. Typically, the optimal distance for drone inspections needs to be within a certain range to ensure clear and detailed images. Therefore, the range of this second dilation is strictly controlled within the effective imaging area of the drone camera to avoid loss of detail due to excessive distance.

[0123] After these two expansion processes, an independent volume area is formed between the outer and inner boundaries of the AABB. This area is the initial inspection space of the drone.

[0124] In some embodiments of the present application, the step of performing a Boolean difference operation based on the initial inspection space and the oblique photography model to determine the inspection space in S102 includes:

[0125] S206 , performing voxelization processing and spatial expansion on the oblique photography model to determine a voxelized model; wherein the voxelized model includes a plurality of voxel grids.

[0126] Exemplarily, performing voxelization and spatial expansion on the oblique photography model to determine the voxelized model, i.e., model voxelization, includes: after completing the spatial registration of the BIM and the oblique photography model, performing voxelization on the oblique photography model (e.g., Figure 4 (as shown in the figure), enabling more efficient path planning and collision detection. The oblique photography model fully represents the real-world environment surrounding the inspection object in space. Voxelization is crucial for drones to accurately perceive the spatial location of these obstacles and avoid collisions with buildings during flight.

[0127] Voxelization is a discretization method that divides a 3D model into regular, equal-volume cubic units. In practice, the oblique photography model is first spatially partitioned to generate a standard 3D voxel grid. These voxel units efficiently record the occupancy state of the space, i.e., whether the voxel is filled with buildings or vacant. Because the voxelized model is a regular grid, the algorithm can perform rapid queries and calculations directly based on voxel indices during path planning and obstacle avoidance, significantly reducing the overhead of complex geometric operations.

[0128] To further enhance flight safety, the voxel grid undergoes a moderate outward spatial expansion. Specifically, after voxelization is complete, each obstacle voxel is expanded outward by a certain safety radius. This radius simulates the actual flight protection distance of the drone, ensuring that sufficient buffer space is reserved during path planning to avoid collision risks caused by air disturbances or navigation errors. Finally, the expanded voxelized model is used in subsequent Boolean operations and spatial culling to further optimize the safety and feasibility of the inspection path.

[0129] S207: Based on the preliminary inspection space and the oblique photography model, a Boolean difference operation is performed to determine the obstacle area.

[0130] S208: Eliminate obstacle areas from the preliminary inspection space to determine the inspection area.

[0131] Exemplarily, a Boolean difference operation is performed based on the initial inspection space and the oblique photography model to determine the inspection area, that is, a three-dimensional voxel grid of the inspection space based on the Boolean operation is generated. After the AABB calculation and double expansion processing of the gantry crane are completed, the preliminary inspection space obtained represents the theoretical safe flight area of the UAV. In order to further eliminate the actual obstacle area, it is necessary to perform a Boolean difference operation on the voxel grid of the preliminary inspection space and the oblique photography model. Specifically, the voxel grid points in the preliminary inspection space and the oblique photography model are compared one by one, and all areas that intersect with the model will be removed from the preliminary inspection space. This Boolean operation can accurately exclude buildings, structural parts and equipment around the gantry crane to ensure that the UAV can fly unobstructed on the inspection path. After the operation is completed, an optimized three-dimensional voxel grid is obtained. This area is the final safe inspection area for the UAV. The UAV will perform path planning and data collection tasks in this area to ensure that there is no collision risk during the inspection process. The three-dimensional voxel grid of the inspection area is as follows: Figure 5 As shown, the three-dimensional voxel grid of the inspection area includes a top-down perspective and an internal perspective.

[0132] It can be understood that the gantry crane BIM model is combined with an oblique photography model of the gantry crane's construction site environment. Three-dimensional spatial alignment, model voxelization, and axis-aligned bounding box calculation are performed to determine a preliminary inspection space. A Boolean difference operation is performed based on the preliminary inspection space and the oblique photography model to determine the obstacle area. Obstacle areas are removed from the preliminary inspection space to determine the inspection space. Determining the inspection space helps narrow the drone's search range, thereby improving gantry crane inspection efficiency.

[0133] In some embodiments of the present application, S103 may be implemented through S301-S305 as follows:

[0134] S301: Determine the inspection object of the drone.

[0135] S302: Based on the inspection object, rays are emitted to the inspection space, and ray tracing analysis and collision detection are performed on the inspection object to determine blocked rays.

[0136] S303: Eliminate blocked rays from the emitted rays, perform collision analysis on the remaining rays and the voxel grid of the inspection space, and determine an initial inspection viewpoint; wherein the initial inspection viewpoint is the viewpoint corresponding to the same voxel unit when rays emitted from different sources all hit the same voxel unit.

[0137] S304: Calculate the pitch angle and azimuth angle of the initial inspection viewpoint, and determine the initial inspection viewpoint that meets the rotation range of the drone gimbal as the inspection viewpoint.

[0138] S305: perform sight line clustering and direction merging on the inspection viewpoints to determine the sight line direction.

[0139] For example, after completing the construction of the inspection space, it is necessary to further determine the inspection viewpoint and sight direction of the drone. First, by performing ray tracing analysis on the inspection object, rays are emitted from the inspection object to the outside (such as Figure 6 As shown, taking weld ABC as an example), and combining collision detection to remove blocked rays. Then, the collision between the remaining rays and the inspection space voxel grid is judged. If the rays emitted from different sources all hit the same voxel unit (such as Figure 6 If a voxel (marked in red) is located within the voxel, it is considered to have visibility of the target. Next, the corresponding pitch and azimuth angles are calculated for the initially selected viewpoints to ensure they fit within the drone's gimbal's rotation range. Once the viewpoints are determined, similar directions are clustered and merged, ultimately generating a comprehensive and efficient set of inspection viewpoints and directions.

[0140] It can be understood that by performing collision detection and line of sight clustering on the inspection space and inspection objects, and determining the inspection viewpoint and line of sight direction, the inspection viewpoint and line of sight direction can be set more accurately. The subsequent inspection by the drone based on the inspection viewpoint and line of sight direction can make the data collected by the drone more accurate, thereby improving the accuracy of the inspection detection results.

[0141] In some embodiments of the present application, S104 may be implemented through S401-S403 as follows:

[0142] S401. Add an obstacle avoidance turning point for flight based on obstacle distribution information corresponding to the patrol viewpoint and the inspection viewpoint.

[0143] S402 : Determine the path nodes based on the patrol viewpoints and the obstacle avoidance turning points; and calculate the shortest paths and distance matrices between the nodes in the path nodes using the A* algorithm to determine the initial patrol path.

[0144] S403 , optimizing the node access sequence of the initial inspection path through the ant colony algorithm and the distance matrix to determine the optimal inspection path.

[0145] For example, after completing the determination of the inspection viewpoint and the direction of sight, the next step is to generate a complete inspection mission path to ensure that the drone covers all target areas efficiently and safely. First, since the default waypoints are in a straight line, there may be a situation where the path passes through a gantry crane or other obstacles. Therefore, it is necessary to add obstacle avoidance turning points for the flight based on the distribution of the inspection viewpoint and surrounding obstacles to guide the path around obstacles and avoid potential collision risks. Then, the A* algorithm is used to perform a global search of all path nodes (including viewpoints and obstacle avoidance turning points), calculate the shortest path between each node, and construct a distance matrix as the basic data for path optimization. After obtaining the distance matrix, the ant colony algorithm (ACO) is introduced to optimize the node access order, and the pheromone iteration converges to the global optimal path (such as Figure 7 (as shown). After optimization is complete, the planning results are converted to geographic coordinates to ensure that the drone's flight path accurately matches the real-world coordinates. Subsequently, flight parameters, including coordinates, altitude, speed, and line of sight angle, are determined based on the optimized path to ensure smooth flight and complete data collection. Finally, an inspection task instruction set is generated and sent to the drone, initiating an autonomous inspection, achieving efficient coverage and precise inspection of the target area.

[0146] It's understandable that adding obstacle avoidance turning points during flight guides the path around obstacles, avoiding potential collision risks. Path planning using the A* algorithm and path optimization using the ant colony algorithm ensure smooth flight and complete data collection. This also improves gantry crane inspection efficiency by avoiding the reduction in inspection efficiency caused by repeated or long flight paths. Furthermore, by improving gantry crane inspection efficiency and the accuracy of inspection results, it facilitates timely identification of potential quality issues with gantry cranes and prevents accidents.

[0147] In some embodiments of the present application, generating a gantry crane inspection result based on the gantry crane data returned by the drone in S105 includes:

[0148] Obtain image data from gantry crane data;

[0149] Performing denoising, cropping, and color balancing preprocessing on the component image in the image data to obtain a preprocessed image; wherein the component image includes at least one of a weld image, a bolt image, and a pin image;

[0150] The preprocessed images are inspected using a gantry crane inspection model to obtain the gantry crane inspection results. The gantry crane inspection model is pre-trained on a YOLOv8 model based on sample weld images, sample bolt images, sample pin shaft images, and their corresponding annotation data.

[0151] It should be noted that the gantry crane detection model includes multiple different gantry crane detection models, and the gantry crane detection model is different for different component images.

[0152] For example, the gantry crane inspection model uses preprocessed images for inspection, resulting in inspection results that include: Weld inspection: Labelimg is used to annotate a pre-collected image dataset to determine weld location and the presence of defects (e.g., cracks, pores, etc.). The annotated dataset is used to train a YOLOv8 model to identify and locate welds and their defects. Weld images collected during inspections are input into the trained YOLOv8 model for real-time detection and identification, outputting information on weld location and defect presence. Bolt / pin inspection: Labelimg is used to annotate a pre-collected image dataset to indicate bolt location and status (e.g., normal, loose, corroded, etc.). The annotated dataset is used to train a YOLOv8 model, and cross-validation is performed to ensure model accuracy and generalization. In specific applications, preprocessing can be performed on inspection images, including denoising, sharpening, and contrast enhancement, to enhance bolt / pin visualization and meet the input requirements of the YOLOv8 model. Use the trained YOLOv8 model to detect bolts / pins in the captured images and identify their position and status.

[0153] In some embodiments of the present application, different gantry crane detection models pre-trained with the YOLOv8 model using different training data are used to identify corresponding inspection-collected image data. This can output more accurate gantry crane detection results, further improving the accuracy of gantry crane inspection results. This can also achieve precise detection of different inspection items for gantry cranes, resulting in more comprehensive and accurate detection results.

[0154] In some embodiments of the present application, the YOLOv8 model generally includes a backbone network (Backbone), a feature enhancement network (Neck), and a detection head unit (Head). In one example, the training process of the YOLOv8 model may include:

[0155] Construct a specified loss function of the YOLOv8 model, and process the training sample image data through the YOLOv8 model to obtain a detection result; determine the loss value of the specified loss function according to the detection result, and iteratively update the model parameters of the YOLOv8 model according to the loss value of the specified loss function, until the loss value of the specified loss function meets the preset conditions and the training is terminated; wherein, the specified loss function includes a first loss function corresponding to the YOLOv8 model and a second loss function set for the feature enhancement network; the loss value of the specified loss function meets the preset conditions including: the loss value of the first loss function is less than a first threshold, and the loss value of the second loss function is less than a second threshold.

[0156] It can be understood that in the training process of each gantry crane detection model, that is, the YOLOv8 model training, in addition to the constraints of the existing first loss function of the YOLOv8 model, by adding further constraints of the second loss function of the feature enhancement network, the feature enhancement network can obtain more effective and accurate image information from the feature map output by the backbone network to enhance the recognition effect of the detection head part, so that the trained YOLOv8 model, that is, the gantry crane detection model, can output more accurate tower crane detection result data.

[0157] After detecting the pre-processed image using the gantry crane detection model to obtain the gantry crane inspection result, the method further includes:

[0158] Match and associate the gantry crane inspection results with the corresponding components in the gantry crane inspection results BIM model to ensure that each test result data in the gantry crane inspection results accurately corresponds to the specific corresponding component in the gantry crane inspection results BIM model;

[0159] The severity of different inspection result data is visualized on the corresponding components in the BIM model of the gantry crane inspection results based on the preset color scheme to intuitively show the health status of each component.

[0160] For example, computer vision-based inspection and defect detection and BIM result visualization:

[0161] 1) Inspection image defect detection

[0162] The inspection image defect detection process begins by selecting an appropriate object detection model (YOLOv8) and collecting inspection image data from the construction site. The labeling tool (LabelImg) is then used to calibrate the defect type and location. The model's detection accuracy is then analyzed based on evaluation metrics, and model hyperparameters are optimized to ensure efficient defect identification. Furthermore, the image data undergoes preprocessing, including denoising, cropping, and color balancing, to optimize the model input and improve detection robustness and accuracy. Finally, the model annotates the defect information it outputs, generating detection results for defects such as cracks, corrosion, and deformation, providing data support for subsequent BIM integration.

[0163] 2) Results visualization

[0164] Results visualization deeply integrates defect information detected during inspections with the BIM model, using color coding (green: normal, yellow: minor defects, red: severe defects) to intuitively display the distribution and severity of defects. Leveraging Revit and Dynamo's view overlay technology, inspection results are simultaneously mapped to the BIM model for visual impact. Furthermore, the system generates inspection reports based on inspection data, detailing the specific location, type, and recommended remediation measures of the defects, helping construction managers quickly identify problems and develop maintenance plans.

[0165] It's understandable that the path planning provided by the integration of BIM models and oblique photography models enables comprehensive, multi-angle intelligent inspections of gantry cranes. This intelligent inspection model, which integrates building information models with real-world 3D data, not only enables accurate monitoring and quantitative assessment of equipment status, but also significantly improves the safety and efficiency of inspection operations, providing an innovative technical solution for preventive maintenance and full lifecycle management of gantry cranes.

[0166] In an embodiment of the present application, the method may further include: analyzing and evaluating the status of bolts / pins based on the inspection results of the gantry crane, evaluating the property parameters of cracks and rust, evaluating the status of welds, and generating inspection reports separately or together.

[0167] For example, a trained YOLOv8 model is used to detect bolts / pins in an image and identify their location and status. Further feature extraction and analysis are then performed on the detected partial images of the bolts / pins. Based on the trained YOLOv8 model's detection results and feature analysis, the condition of each bolt / pin is evaluated, ultimately generating a detailed inspection report that includes the bolt / pin's location, status, image evidence, and any necessary maintenance recommendations.

[0168] Use the trained YOLOv8 model to analyze captured crack and rust images to identify and locate cracks and rust. Based on the trained YOLOv8 model's detection results, evaluate crack attributes such as size, length, and distribution, and evaluate rust attributes such as area, severity, and distribution. Finally, prepare a detailed inspection report.

[0169] The captured weld images are fed into a trained YOLOv8 model for real-time detection and recognition. The trained YOLOv8 model's detection results assess the weld's condition, including defect type and severity. Finally, a detailed inspection report is compiled, including weld quality, defect type, and any necessary maintenance and repair recommendations.

[0170] It should be noted that the above-mentioned test reports can be written separately, or the contents of the above-mentioned test reports can be included in one test report.

[0171] The beneficial effects brought by this application:

[0172] (1) Security

[0173] Zero-contact inspection: Gantry cranes are inspected using drones equipped with high-definition cameras, completely avoiding accidents such as falls and electric shocks caused by manual climbing inspections.

[0174] (2) Efficiency and accuracy

[0175] Intelligent waypoint optimization: Based on the fusion data of BIM and oblique photography models, multi-view waypoints are automatically generated for the inspection object and the inspection path is calculated to ensure that all important inspection contents are inspected.

[0176] (3) Full life cycle data management

[0177] Dynamically associate defect data with BIM components and ultimately output a traceable report, enabling full-process intelligence from data collection to decision support.

[0178] Based on the above embodiment of the intelligent inspection method of gantry crane based on BIM and UAV, the embodiment of the present application also provides an intelligent inspection system of gantry crane based on BIM and UAV, such as Figure 8 As shown, Figure 8 The structural diagram of a gantry crane intelligent inspection system based on BIM and UAV provided in an embodiment of the present application is as follows. The gantry crane intelligent inspection system based on BIM and UAV 8 includes: a model building module 801, an inspection area generation module 802, an inspection viewpoint and line of sight generation module 803, an inspection path generation module 804 and a flight inspection module 805, wherein:

[0179] The model building module 801 is used to determine a gantry crane BIM model of the gantry crane and an oblique photography model of the construction site environment where the gantry crane is located; wherein the gantry crane BIM model is refined to a single component of the gantry crane;

[0180] The inspection area generation module 802 is configured to perform three-dimensional space alignment, model voxelization, and axis-aligned bounding box calculation based on the gantry crane BIM model and the oblique photography model to determine an initial inspection space; and perform a Boolean difference operation based on the initial inspection space and the oblique photography model to determine an inspection area;

[0181] The inspection viewpoint and sight line generation module 803 is configured to perform collision detection on the inspection area using a predetermined inspection object to determine an initial inspection viewpoint; and to determine the inspection viewpoint and sight line direction by calculating the pitch angle, azimuth angle, and sight line clustering of the initial inspection viewpoint; wherein the inspection object is at least one component of the gantry crane;

[0182] The inspection path generation module 804 is configured to perform path planning using the A* algorithm based on the inspection viewpoint and the obstacle distribution information corresponding to the inspection viewpoint to obtain an initial inspection path; and to optimize the node access sequence of the initial inspection path using the ant colony algorithm to obtain an optimal inspection path;

[0183] The flight inspection module 805 is used to generate an inspection task based on the inspection object, the line of sight direction and the optimal inspection path, so that the drone flies according to the line of sight direction and the optimal inspection path in the inspection task and collects gantry crane data, and generates gantry crane inspection results based on the gantry crane data returned by the drone.

[0184] In some embodiments of the present application, the inspection area generation module 802 is also used to obtain the three-dimensional coordinate information of multiple reference points in the gantry crane BIM model; based on the three-dimensional coordinate information, calculate the conversion matrix between the coordinates of the gantry crane BIM model and the coordinates of the oblique photography model; through the conversion matrix, perform three-dimensional spatial alignment on the gantry crane BIM model and the oblique photography model to obtain the gantry crane BIM model aligned with the oblique photography model; based on the aligned gantry crane BIM model, obtain the three-dimensional spatial coordinate information of the gantry crane, calculate the minimum and maximum values of the gantry crane in the three axes of X, Y, and Z, and determine the axis-aligned bounding box of the gantry crane; based on the axis-aligned bounding box corresponding to the gantry crane, perform expansion processing to determine the preliminary inspection space.

[0185] In some embodiments of the present application, the inspection area generation module 802 is also used to perform a first expansion process based on the axis-aligned bounding box corresponding to the gantry crane to obtain a first spatial bounding box; wherein, the first expansion process is used to increase the safety distance; based on the first spatial bounding box, a second expansion process is performed to obtain a second spatial bounding box; wherein, the spatial range increased by the second expansion process is within the preset imaging area of the drone camera; based on the axis-aligned bounding box corresponding to the gantry crane and the second spatial bounding box, the preliminary inspection space is determined.

[0186] In some embodiments of the present application, the inspection area generation module 802 is further used to perform model voxelization processing and spatial expansion on the oblique photography model to determine a voxelized model; wherein the voxelized model includes multiple voxel grids; a Boolean difference operation is performed on the three-dimensional voxel grid of the preliminary inspection space and the multiple voxel grids of the voxelized model to determine the obstacle area; and the obstacle area is eliminated from the preliminary inspection space to determine the inspection area.

[0187] In some embodiments of the present application, the inspection viewpoint and line of sight generation module 803 is also used to determine the inspection object of the drone; based on the inspection object, rays are emitted to the inspection space, and ray tracing analysis and collision detection are performed on the inspection object to determine the blocked rays; the blocked rays are removed from the emitted rays, and collision analysis is performed on the remaining rays and the voxel grid of the inspection space to determine the initial inspection viewpoint; wherein, the initial inspection viewpoint is the viewpoint corresponding to the same voxel unit hit by rays emitted from different sources; the pitch angle and azimuth angle of the initial inspection viewpoint are calculated, and the initial inspection viewpoint that meets the rotation range of the drone gimbal is determined as the inspection viewpoint; the inspection viewpoint is clustered and direction merged to determine the line of sight direction.

[0188] In some embodiments of the present application, the inspection path generation module 804 is also used to add obstacle avoidance turning points for flight based on the inspection viewpoint and the obstacle distribution information corresponding to the inspection viewpoint; determine the path nodes based on the inspection viewpoint and the obstacle avoidance turning points; and calculate the shortest path and distance matrix between each node in the path node through the A* algorithm to determine the initial inspection path; optimize the node access order of the initial inspection path through the ant colony algorithm and the distance matrix to determine the optimal inspection path.

[0189] In some embodiments of the present application, the flight inspection module 805 is also used to obtain image data in the gantry crane data; pre-process the component image in the image data by denoising, cropping and color balancing to obtain a pre-processed image; wherein the component image includes at least one of a weld image, a bolt image and a pin image; and detect the pre-processed image through a gantry crane detection model to obtain the gantry crane inspection result; wherein the gantry crane detection model is obtained by pre-training the YOLOv8 model based on sample weld images, sample bolt images, sample pin images and their respective corresponding annotation data.

[0190] In some embodiments of the present application, the flight inspection module 805 is also used to detect the preprocessed image through the gantry crane detection model, and after obtaining the gantry crane inspection result, match and associate the gantry crane inspection result with the corresponding component in the gantry crane inspection result BIM model to ensure that each detection result data in the gantry crane inspection result accurately corresponds to the specific corresponding component in the gantry crane inspection result BIM model; and visually present the severity of different detection result data on the specific corresponding component in the gantry crane inspection result BIM model based on a preset color scheme to intuitively display the health status of each component.

[0191] Based on the BIM and UAV-based gantry crane intelligent inspection method of the above embodiment, the embodiment of the present application also provides a BIM and UAV-based gantry crane intelligent inspection device, such as Figure 9 As shown, Figure 9 This is a schematic diagram of the structure of a BIM- and drone-based intelligent inspection device for gantry cranes provided in an embodiment of the present application. The BIM- and drone-based intelligent inspection device 9 for gantry cranes includes a processor 901 and a memory 902. The memory 902 is used to store a computer program; the processor 901 is used to load and execute the computer program from the memory to execute the BIM- and drone-based intelligent inspection method for gantry cranes described in the above embodiment.

[0192] In the embodiments of the present application, the processor 901 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0193] An embodiment of the present application provides a computer-readable storage medium storing a computer program for implementing the intelligent inspection method for gantry cranes based on BIM and drones as described in any of the above embodiments when executed by a processor.

[0194] For example, the program instructions corresponding to the intelligent inspection method for gantry cranes based on BIM and drones in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instructions corresponding to the intelligent inspection method for gantry cranes based on BIM and drones in the storage medium are read or executed by an electronic device, the intelligent inspection method for gantry cranes based on BIM and drones as described in any of the above embodiments can be implemented.

[0195] In addition, the functional modules in the embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional modules.

[0196] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0197] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other. For the sake of brevity, they will not be repeated here.

[0198] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed across multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of this embodiment.

[0199] In addition, all functional modules in the embodiments of the present application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the above-mentioned integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0200] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0201] The methods disclosed in the several method embodiments provided in the embodiments of this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0202] The features disclosed in several product embodiments provided in the embodiments of this application can be arbitrarily combined to obtain new product embodiments without conflict.

[0203] The features disclosed in several method or device embodiments provided in the embodiments of this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0204] The above is merely an implementation of the embodiments of the present application, but the scope of protection of the embodiments of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the embodiments of the present application. Therefore, the scope of protection of the embodiments of the present application should be based on the scope of protection of the claims.

Claims

1. A BIM and UAV-based intelligent inspection method for gantry cranes, characterized in that: The method comprises: Determining a gantry crane BIM model of a gantry crane and an oblique photography model of a construction site environment in which the gantry crane is located; wherein the gantry crane BIM model is refined down to individual components of the gantry crane; Based on the gantry crane BIM model and the oblique photography model, three-dimensional space alignment, model voxelization processing, and axis-aligned bounding box calculation are performed to determine an initial inspection space; and based on the initial inspection space and the oblique photography model, a Boolean difference operation is performed to determine an inspection area, including: performing model voxelization processing and spatial expansion on the oblique photography model to determine a voxelized model; wherein the voxelized model includes multiple voxel grids; performing a Boolean difference operation on the three-dimensional voxel grid of the initial inspection space and the multiple voxel grids of the voxelized model to determine an obstacle area; Eliminate the obstacle area from the initial inspection space to determine the inspection area; The inspection area is subjected to collision detection by a predetermined inspection object to determine an initial inspection viewpoint; and the inspection viewpoint and line of sight direction are determined by calculating the pitch angle, azimuth angle and line of sight clustering of the initial inspection viewpoint, including: determining the inspection object of the UAV; based on the inspection object, emitting rays to the inspection area, and performing ray tracing analysis and collision detection on the inspection object to determine the blocked rays; removing the blocked rays from the emitted rays, performing collision analysis on the remaining rays and the voxel grid of the inspection area to determine the initial inspection viewpoint; wherein, the initial inspection viewpoint is a viewpoint corresponding to the same voxel unit that all rays emitted from different sources hit; calculating the pitch angle and azimuth angle of the initial inspection viewpoint, and determining the initial inspection viewpoint that meets the rotation range of the UAV gimbal as the inspection viewpoint; performing line of sight clustering and direction merging on the inspection viewpoint to determine the line of sight direction; wherein, the inspection object is at least one component of the gantry crane; Based on the inspection viewpoint and the obstacle distribution information corresponding to the inspection viewpoint, path planning is performed using the A* algorithm to obtain an initial inspection path; and the node access sequence of the initial inspection path is optimized using the ant colony algorithm to obtain an optimal inspection path; Based on the inspection object, the line of sight direction and the optimal inspection path, an inspection task is generated so that the drone flies according to the line of sight direction and the optimal inspection path in the inspection task and collects gantry crane data, and a gantry crane inspection result is generated based on the gantry crane data returned by the drone.

2. The method according to claim 1, characterized in that The method of performing three-dimensional space alignment, model voxelization, and axis-aligned bounding box calculation based on the gantry crane BIM model and the oblique photography model to determine the initial inspection space includes: Obtaining three-dimensional coordinate information of multiple reference points in the gantry crane BIM model; Based on the three-dimensional coordinate information, calculating a conversion matrix between the coordinates of the gantry crane BIM model and the coordinates of the oblique photography model; Performing three-dimensional spatial alignment on the gantry crane BIM model and the oblique photography model using the conversion matrix to obtain the gantry crane BIM model aligned with the oblique photography model; Based on the aligned gantry crane BIM model, obtaining the three-dimensional spatial coordinate information of the gantry crane, calculating the minimum and maximum values of the gantry crane in the three axes of X, Y, and Z, and determining the axis-aligned bounding box of the gantry crane; An expansion process is performed based on the axis-aligned bounding box corresponding to the gantry crane to determine the initial inspection space.

3. The method according to claim 2, characterized in that The step of performing expansion processing based on the axis-aligned bounding box corresponding to the gantry crane to determine the initial inspection space includes: Performing a first dilation process based on the axis-aligned bounding box corresponding to the gantry crane to obtain a first spatial bounding box; wherein the first dilation process is used to increase the safety distance; Performing a second dilation process on the first spatial bounding box to obtain a second spatial bounding box; wherein the spatial range enlarged by the second dilation process is within a preset imaging area of the drone camera; The initial inspection space is determined based on the axis-aligned bounding box corresponding to the gantry crane and the second space bounding box.

4. The method according to claim 1, wherein The method of performing path planning based on the inspection viewpoint and the obstacle distribution information corresponding to the inspection viewpoint by using the A* algorithm to obtain an initial inspection path; The node access sequence of the initial inspection path is optimized by using an ant colony algorithm to obtain the optimal inspection path, including: Adding an obstacle avoidance turning point for flight based on the inspection viewpoint and the obstacle distribution information corresponding to the inspection viewpoint; Determine the path nodes based on the inspection viewpoint and the obstacle avoidance turning point; and calculate the shortest path and distance matrix between each node in the path nodes by the A* algorithm to determine the initial inspection path; The node access sequence of the initial inspection path is optimized by using the ant colony algorithm and the distance matrix to determine the optimal inspection path.

5. The method according to claim 1, wherein Generating a gantry crane inspection result according to the gantry crane data returned by the drone includes: Acquiring image data from the gantry crane data; Performing denoising, cropping, and color balancing preprocessing on the component image in the image data to obtain a preprocessed image; wherein the component image includes at least one of a weld image, a bolt image, and a pin image; The preprocessed image is detected by a gantry crane detection model to obtain the gantry crane inspection result; wherein the gantry crane detection model is pre-trained by a YOLOv8 model based on sample weld images, sample bolt images, sample pin shaft images and their corresponding annotated data; After detecting the pre-processed image using the gantry crane detection model to obtain the gantry crane inspection result, the method further includes: Matching and associating the gantry crane inspection results with corresponding components in the gantry crane BIM model to ensure that each test result data in the gantry crane inspection results accurately corresponds to the specific corresponding component in the gantry crane BIM model; The severity of different test result data is visually presented on the specific corresponding components in the gantry crane BIM model based on a preset color scheme to intuitively display the health status of each component.

6. An intelligent inspection system for gantry cranes based on BIM and drones, characterized by: The intelligent inspection system for gantry cranes based on BIM and UAVs includes: a model building module, an inspection area generation module, an inspection viewpoint and sight generation module, an inspection path generation module, and a flight inspection module, wherein: The model building module is used to determine a gantry crane BIM model of the gantry crane and an oblique photography model of the construction site environment in which the gantry crane is located; wherein the gantry crane BIM model is refined to a single component of the gantry crane; The inspection area generation module is configured to perform three-dimensional space alignment, model voxelization, and axis-aligned bounding box calculation based on the gantry crane BIM model and the oblique photography model to determine an initial inspection space; and perform a Boolean difference operation based on the initial inspection space and the oblique photography model to determine an inspection area; The inspection area generation module is further configured to perform voxelization and spatial expansion on the oblique photography model to determine a voxelized model; wherein the voxelized model includes a plurality of voxel grids; perform a Boolean difference operation on the three-dimensional voxel grid of the initial inspection space and the plurality of voxel grids of the voxelized model to determine an obstacle area; and eliminate the obstacle area from the initial inspection space to determine the inspection area; The inspection viewpoint and sight line generation module is used to perform collision detection on the inspection area through a predetermined inspection object to determine the initial inspection viewpoint; and to determine the inspection viewpoint and sight line direction by calculating the pitch angle and azimuth angle of the initial inspection viewpoint and sight line clustering; The inspection area generation module is further used to determine the inspection object of the UAV; based on the inspection object, emit rays to the inspection area, and perform ray tracing analysis and collision detection on the inspection object to determine the blocked rays; remove the blocked rays from the emitted rays, perform collision analysis on the remaining rays and the voxel grid of the inspection area, and determine the initial inspection viewpoint; wherein, the initial inspection viewpoint is the viewpoint corresponding to the same voxel unit hit by rays emitted from different sources; calculate the pitch angle and azimuth angle of the initial inspection viewpoint, and determine the initial inspection viewpoint that meets the rotation range of the UAV gimbal as the inspection viewpoint; perform line of sight clustering and direction merging on the inspection viewpoint to determine the line of sight direction; wherein, the inspection object is at least one component of the gantry crane; The inspection path generation module is used to perform path planning using the A* algorithm based on the inspection viewpoint and the obstacle distribution information corresponding to the inspection viewpoint to obtain an initial inspection path; and to optimize the node access sequence of the initial inspection path using the ant colony algorithm to obtain an optimal inspection path; The flight inspection module is used to generate an inspection task based on the inspection object, the line of sight direction and the optimal inspection path, so that the drone flies according to the line of sight direction and the optimal inspection path in the inspection task and collects gantry crane data, and generates gantry crane inspection results based on the gantry crane data returned by the drone.

7. An intelligent inspection device for gantry cranes based on BIM and drones, characterized by: include: processor and memory, wherein The memory is used to store computer programs; The processor is configured to call and run the computer program from the memory to perform the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that Executable instructions are stored, which are used to cause a processor to execute and implement the method according to any one of claims 1 to 5.

Images