Drone-based Ship Inspection Method, Device, System and Electronic Equipment

Abstract

The present application provides a method, device, system and electronic device for inspecting the integrity of ship intermediate products based on an unmanned aerial vehicle. The method for inspecting the integrity of ship intermediate products includes: acquiring three-dimensional point cloud data of ship intermediate products; removing noise points from the three-dimensional point cloud data; registering the denoised point cloud data with a theoretical three-dimensional model to obtain an overlapping area and a non-overlapping area; the non-overlapping area includes an unpaired point cloud area and an unpaired model area, pairwise registering a plurality of unpaired model blocks and a plurality of unpaired point cloud blocks, and removing misassembled unpaired point cloud blocks from the unpaired point cloud blocks to obtain overassembled unpaired point cloud blocks; removing misassembled unpaired model blocks from the unpaired model blocks to obtain missing unpaired model blocks; and generating an inspection report on the integrity of ship intermediate products. The method, device, system and electronic device for inspecting the integrity of ship intermediate products based on an unmanned aerial vehicle in the present application can improve the accuracy of inspection results.

Description

Technical Field

[0001] The present application relates to the field of intelligent ship manufacturing technology, and in particular to methods, devices, systems and electronic equipment for integrity inspection of intermediate ship products. Background Art

[0002] The integrity inspection of ship intermediate products is one of the important tasks in the shipbuilding process. The specific inspection content is to inspect whether the hull parts and outfitting parts on the intermediate products of each level of the ship, such as small assembly, medium assembly, large assembly, sub-section, and total section, are installed completely and correctly. The traditional method of inspecting the integrity of ship intermediate products is for inspectors to check the corresponding two-dimensional construction drawings, and then check the integrity of the assembly of each part on the intermediate product through spatial imagination based on the drawings. In recent years, some shipping companies and classification societies have successively carried out the application of drones to assist in ship inspections, but they have not been able to completely break away from traditional inspection methods. The existing ship integrity inspection methods have the following main shortcomings:

[0003] 1. The method of converting two-dimensional construction drawings into three-dimensional models through spatial imagination and comparing them with the actual hull cannot guarantee the accuracy of the imagined three-dimensional model and is prone to missed inspections.

[0004] Second, since most of the intermediate products of ships are large in size, the observation and comparison of the actual hull needs to be achieved by manually climbing the hull, resulting in low inspection efficiency and high risk.

[0005] 3. The existing method of using drones to inspect the integrity of intermediate products of ships mainly uses the camera carried by the drone to record the appearance of the intermediate products of ships. It is still necessary to manually observe the images sent back by the camera and compare them with the drawings. It does not completely liberate manual labor, and the accuracy of the inspection results is also affected by human factors.

[0006] Fourth, the existing 3D scanning modeling method generally uses laser scanners, visual sensors and other equipment to obtain 3D data of the hull structure in a non-contact manner, instead of using templates and sample boxes for contact inspection. It can only conclude that there are appearance differences between the actual hull and the standard model in certain areas, but it is impossible to inspect the integrity of the intermediate product of the ship.

[0007] As the shipbuilding industry develops towards digitalization and intelligence, how to reduce manual participation in the integrity inspection of ship intermediate products, reduce the danger of inspection operations, reduce errors in inspection results, ensure inspection quality, and improve inspection efficiency is a key issue that needs to be urgently addressed in the current intelligent development process of the shipbuilding industry. Summary of the invention

[0008] In view of the above-mentioned disadvantages of the prior art, the purpose of the present application is to provide a method, device, system and electronic device for inspecting the integrity of ship intermediate products, which can reduce human factors in the inspection process, enhance the degree of intelligence in the inspection process, and improve the accuracy of inspection results.

[0009] In a first aspect, the present application provides a method for inspecting the integrity of ship intermediate products based on an unmanned aerial vehicle. The method for inspecting the integrity of ship intermediate products based on an unmanned aerial vehicle includes the following steps:

[0010] Obtain the three-dimensional point cloud data of the ship intermediate product;

[0011] Remove the noise points from the three-dimensional point cloud data to obtain denoised point cloud data;

[0012] Register the denoised point cloud data with the theoretical three-dimensional model to obtain an overlapping area and a non-overlapping area; within the overlapping area, both the theoretical three-dimensional model and the corresponding denoised point cloud data exist; the non-overlapping area includes an unpaired point cloud area and an unpaired model area, and the unpaired model area includes a plurality of unpaired model blocks, and each unpaired model block corresponds to a part three-dimensional model or a component three-dimensional model;

[0013] Divide the point cloud data in the unpaired point cloud area into a plurality of unpaired point cloud blocks, and each unpaired point cloud block corresponds to a ship part or a ship component;

[0014] Register the plurality of unpaired model blocks and the plurality of unpaired point cloud blocks in pairs, and calculate the registration error. Mark a pair of unpaired model blocks and unpaired point cloud blocks with a registration error less than the first threshold as misassembled unpaired model blocks and misassembled unpaired point cloud blocks respectively;

[0015] Remove the misassembled unpaired point cloud blocks from the unpaired point cloud blocks to obtain overassembled unpaired point cloud blocks;

[0016] Remove the misassembled unpaired model blocks from the unpaired model blocks to obtain missing unpaired model blocks;

[0017] Generate an inspection report on the integrity of the ship intermediate product according to the misassembled unpaired model blocks, the misassembled unpaired point cloud blocks, the overassembled unpaired point cloud blocks and the missing unpaired model blocks.

[0018] In one implementation manner of the first aspect, the obtaining of the three-dimensional point cloud data of the ship intermediate product includes: scanning the physical object of the ship intermediate product by an SLAM three-dimensional laser scanner carried by an unmanned aerial vehicle to obtain the three-dimensional point cloud data.

[0019] In an implementation of the first aspect, the obtaining of the three-dimensional point cloud data of the ship intermediate product further includes: obtaining the color data of the ship intermediate product through an RGB sensor carried by a drone, and correlating the color data with the three-dimensional point cloud data;

[0020] The removing of the noise points from the three-dimensional point cloud data includes:

[0021] Subtracting the R, G, and B values of the color data from the R, G, and B values of a preset color respectively, taking the absolute value of the differences of R, G, and B and summing them to obtain a color difference;

[0022] Marking the three-dimensional point cloud data with the color difference greater than a second threshold as non-hull structure noise points;

[0023] Deleting the non-hull structure noise points from the three-dimensional point cloud data.

[0024] In an implementation of the first aspect, the drone is further equipped with a fill light, and the color data is obtained under the illumination of the fill light.

[0025] In an implementation of the first aspect, the removing of the noise points from the three-dimensional point cloud data further includes: removing isolated noise points from the three-dimensional point cloud data through Gaussian filtering.

[0026] In an implementation of the first aspect, the drone plans its flight route through the BUG1 algorithm.

[0027] In a second aspect, the present application provides an electronic device, including a memory and a processor, where the processor is configured to execute a computer program stored in the memory so that the electronic device executes the method for integrity inspection of ship intermediate products based on a drone.

[0028] In a third aspect, the present application provides an integrity inspection device for ship intermediate products based on a drone, characterized in that the integrity inspection device for ship intermediate products based on a drone includes:

[0029] A three-dimensional measurement module, configured to obtain three-dimensional point cloud data of the ship intermediate product;

[0030] A denoising module, configured to remove noise points from the three-dimensional point cloud data to obtain denoised point cloud data;

[0031] A primary registration module is used to register the denoised point cloud data with the theoretical three-dimensional model to obtain an overlapping area and a non-overlapping area; in the overlapping area, the theoretical three-dimensional model and the corresponding denoised point cloud data exist simultaneously; the non-overlapping area includes an unpaired point cloud area and an unpaired model area, the unpaired model area includes a plurality of unpaired model blocks, each of which corresponds to a part three-dimensional model or a component three-dimensional model;

[0032] A block division module, used for dividing the point cloud data of the unpaired point cloud area into a plurality of unpaired point cloud blocks by using a clustering algorithm, each unpaired point cloud block corresponding to a ship part or a ship component;

[0033] A secondary registration module is used to register the multiple unpaired model blocks and the multiple unpaired point cloud blocks in pairs, calculate the registration error, and mark a pair of unpaired model blocks and unpaired point cloud blocks whose registration error is less than a first threshold as a mis-installed unpaired model block and a mis-installed unpaired point cloud block respectively;

[0034] A multi-installation statistical module is used to remove the wrongly installed unpaired point cloud blocks from the unpaired point cloud blocks to obtain multi-installed unpaired point cloud blocks;

[0035] A missing assembly statistics module, used to remove the wrongly assembled unpaired model blocks from the unpaired model blocks to obtain missing unpaired model blocks;

[0036] A report generation module is used to generate a ship intermediate product integrity inspection report based on the mis-installed unpaired model block, the mis-installed unpaired point cloud block, the multiple-installed unpaired point cloud block and the missing unpaired model block.

[0037] In a fourth aspect, the present application provides a drone-based ship intermediate product integrity inspection system, the drone-based ship intermediate product integrity inspection system comprising:

[0038] A drone, wherein the drone is equipped with a SLAM three-dimensional laser scanner, and the SLAM three-dimensional laser scanner is used to obtain three-dimensional point cloud data of the ship intermediate product;

[0039] The edge server is connected to the drone for communication, and the edge server is used to: remove noise points in the three-dimensional point cloud data to obtain denoised point cloud data; align the denoised point cloud data with the theoretical three-dimensional model to obtain overlapping areas and non-overlapping areas; in the overlapping area, the theoretical three-dimensional model and the corresponding denoised point cloud data exist at the same time; the non-overlapping area includes an unpaired point cloud area and an unpaired model area, and the unpaired model area includes a plurality of unpaired model blocks, each of which corresponds to a three-dimensional model of a part or a three-dimensional model of a component; and the unpaired point cloud area is aligning the theoretical three-dimensional model with the theoretical three-dimensional model to obtain overlapping areas and non-overlapping areas. The point cloud data of the point cloud area is divided into a plurality of unpaired point cloud blocks, each of which corresponds to a ship part or a ship component; the plurality of unpaired model blocks and the plurality of unpaired point cloud blocks are registered in pairs, and the registration error is calculated, and a pair of unpaired model blocks and unpaired point cloud blocks whose registration error is less than a first threshold are marked as mis-installed unpaired model blocks and mis-installed unpaired point cloud blocks respectively; the mis-installed unpaired point cloud blocks are removed from the unpaired point cloud blocks to obtain multiple-installed unpaired point cloud blocks; the mis-installed unpaired model blocks are removed from the unpaired model blocks to obtain missed unpaired model blocks;

[0040] A display is used for three-dimensionally displaying the mis-installed unpaired model blocks, the mis-installed unpaired point cloud blocks, the over-installed unpaired point cloud blocks and the missing unpaired model blocks.

[0041] In an implementation of the fourth aspect, the edge server is connected to the drone via a 5G network.

[0042] As described above, the ship intermediate product integrity inspection method, device, system and electronic equipment described in the present application can reduce human factors in the inspection process, enhance the intelligence level of the inspection process, and improve the accuracy of the inspection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Shown is an application scenario diagram of the drone-based ship intermediate product integrity inspection method described in one embodiment of the present application.

[0044] Figure 2 Shown is a flow chart of a method for integrity inspection of intermediate ship products based on drones as described in one embodiment of the present application.

[0045] Figure 3 Display as Figure 1 Specific flow chart of step S200 in FIG.

[0046] Figure 4 Shown is a structural schematic diagram of an electronic device described in an embodiment of the present application.

[0047] Component number description

[0048] 100, Intermediate product of ship; 200, UAV; 300, Server; 400, Display. Specific implementation mode

[0049] The following uses specific specific examples to illustrate the implementation modes of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation modes. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0050] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner. Therefore, only the components related to the present application are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components during actual implementation. The type, quantity, and ratio of each component during actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0051] Figure 1 Shown is a schematic diagram of the application scenario of the integrity inspection method for the intermediate product of a ship based on a UAV in this embodiment. This application scenario includes an intermediate product 100 of a ship, a UAV 200, a server 300, and a display 400. The intermediate product 100 of the ship can be small assemblies, medium assemblies, large assemblies, sections, sub-assemblies, etc. during the shipbuilding process. The 3D scanning device carried by the UAV 200 can be an image acquisition device, a laser tracker, a laser scanner, etc. The server 300 can be a central server, an edge server, etc. During the process of the UAV 200 flying close to the intermediate product 100 of the ship, the 3D scanning device scans the internal and external surface structures of the intermediate product 100 of the ship, and sends the collected 3D point cloud data to the server 300 through a wireless network such as Bluetooth or WIFI. The server 300 obtains the point cloud data of the physical hull by performing processing such as denoising, smoothing, and stitching on the 3D point cloud data, and aligns and registers the point cloud data of the physical hull with the theoretical hull model, so as to visually display the missing, misassembled, and overassembled parts on the display 400, reduce errors caused by artificial imagination and comparison, and improve the intelligence and accuracy of the integrity inspection of the intermediate product of the ship.

[0052] The following will elaborate in detail on the principles and implementation modes of the integrity inspection method, device, system, and electronic device for the intermediate product of a ship in this embodiment, so that those skilled in the art can understand the integrity inspection method, device, system, and electronic device for the intermediate product of a ship in this embodiment without creative labor.

[0053] As Figure 2As shown in the figure, this embodiment provides a method for integrity inspection of ship intermediate products based on an unmanned aerial vehicle (UAV). The method for integrity inspection of ship intermediate products based on an UAV includes the following steps:

[0054] Step S100: Obtain the three-dimensional point cloud data of the ship intermediate product. The three-dimensional point cloud data is composed of millions to hundreds of millions of three-dimensional coordinate points on the surface of the ship intermediate product. In addition to the coordinate information of each three-dimensional coordinate point, the three-dimensional point cloud data may also include information such as light reflection intensity, point cloud density, transparency, time stamp, etc. The methods for obtaining the three-dimensional point cloud data include single-point laser detection, line laser scanning, structured light scanning, etc.

[0055] Step S200: Remove the noise points from the three-dimensional point cloud data to obtain denoised point cloud data. The three-dimensional point cloud data obtained by the scanning instrument may introduce various types of noise, such as environmental noise, non-hull structure noise, measurement noise, etc. Through denoising processing, the reliability of the three-dimensional point cloud data can be improved, and the possibility of misjudging noise points as overloaded or misassembled parts in the subsequent integrity detection process can be reduced.

[0056] Step S300: Register the denoised point cloud data with the theoretical three-dimensional model to obtain the overlapping area and the non-overlapping area; in the overlapping area, both the theoretical three-dimensional model and the corresponding denoised point cloud data exist simultaneously; the non-overlapping area includes the unpaired point cloud area and the unpaired model area. The unpaired model area includes multiple unpaired model blocks, and each unpaired model block corresponds to a part three-dimensional model or a component three-dimensional model. The existing point cloud data registration process is to obtain the shape deviation of the three-dimensional point cloud data relative to the theoretical three-dimensional model to further determine whether the produced parts are qualified. The purpose of the point cloud data registration in this embodiment is to obtain the macroscopic structural deviation of the ship intermediate product as a whole relative to the theoretical three-dimensional model, and then obtain information on misassembly, overloading, and missing assembly of parts. The specific algorithm for point cloud data registration in this embodiment can adopt existing technologies, such as the iterative closest point algorithm, k-d tree search method, etc. The theoretical three-dimensional model can be designed and completed by existing three-dimensional drawing software. The theoretical three-dimensional model of the ship intermediate product is generally assembled by multiple parts, so each unpaired model block can be directly obtained from the theoretical three-dimensional model. For example, the unpaired model blocks can be directly judged as components such as floor plates, brackets, longitudinal girders, ribs, and rib frames according to the theoretical three-dimensional model.

[0057] Step S400: Divide the point cloud data in the unpaired point cloud area into multiple unpaired point cloud blocks, with each unpaired point cloud block corresponding to a ship part or component. The three-dimensional scanning device obtains the overall shape data of the ship intermediate product. Therefore, it is necessary to segment each coordinate point in the three-dimensional point cloud data to distinguish different components. In this embodiment, the point cloud data in the unpaired point cloud area is segmented, rather than directly segmenting the denoised point cloud data. Compared with the denoised point cloud data, the unpaired point cloud area has more significant spatial discreteness, which is beneficial to improving the accuracy of the segmentation results. Specifically, the algorithms for segmenting the point cloud data in this embodiment can adopt the k-nearest neighbor fast search method, the edge detection algorithm based on gray value, the K-Means clustering algorithm, etc.

[0058] Step S500: Register the multiple unpaired model blocks and the multiple unpaired point cloud blocks pairwise, and calculate the registration error. Mark a pair of unpaired model blocks and unpaired point cloud blocks with a registration error less than the first threshold as the misinstalled unpaired model block and the misinstalled unpaired point cloud block respectively. If the registration error is less than the first threshold, it means that one of the unpaired model blocks and one of the unpaired point cloud blocks have shape similarity, and it is considered that the unpaired model block and the unpaired point cloud block are expressions of the same part or component, but there is a significant deviation between the spatial position or spatial attitude of the unpaired point cloud block and the unpaired model block, that is, the component corresponding to the unpaired point cloud block should be installed at the position where the unpaired model block is located, but in the actual product, it is not at the position where the unpaired model block is located. When a component is misinstalled, a misinstalled unpaired model block and a misinstalled unpaired point cloud block will appear in pairs. For the misinstalled unpaired point cloud block, the component corresponding to the misinstalled unpaired point cloud block should be removed and reinstalled at the position where the misinstalled unpaired model block is located.

[0059] Step S600: Remove the misinstalled unpaired point cloud blocks from the unpaired point cloud blocks to obtain the overinstalled unpaired point cloud blocks. The overinstalled unpaired point cloud blocks indicate that the components corresponding to the overinstalled unpaired point cloud blocks appear in the actual ship intermediate product, but do not appear at any position in the theoretical three-dimensional model. For the overinstalled unpaired point cloud blocks, the components corresponding to the overinstalled unpaired point cloud blocks should be removed from the actual ship intermediate product.

[0060] Step S700: Remove the misinstalled unpaired model blocks from the unpaired model blocks to obtain the missing unpaired model blocks. The missing unpaired model blocks indicate that the components corresponding to the missing unpaired model blocks exist in the theoretical three-dimensional model, but do not exist at any position in the actual ship intermediate product. For the missing unpaired model blocks, the components should be supplemented at the corresponding positions in the actual ship intermediate product according to the positions of the missing unpaired model blocks.

[0061] Step S800: Generate an integrity inspection report for the ship's intermediate products based on the misinstalled unpaired model blocks, misinstalled unpaired point cloud blocks, overinstalled unpaired point cloud blocks, and missing installed unpaired model blocks. The integrity inspection report for the ship's intermediate products may include a parts list, a 3D model with corresponding components marked in different colors, etc.

[0062] Specifically, in this embodiment, step S100 specifically includes: Scanning the physical object of the ship's intermediate products with a SLAM 3D laser scanner carried by a drone to obtain 3D point cloud data. The SLAM 3D laser scanner has the advantages of fast detection speed and high accuracy, and the obtained 3D point cloud data can be directly processed with 3D drawing software. In this embodiment, using the drone to capture the 3D point cloud data instead of manually carrying scanning instruments to climb large ship's intermediate products improves the safety of the inspectors and the inspection efficiency.

[0063] In order to reduce the influence of the point cloud data formed by non-ship parts such as scaffolding and brackets on the integrity detection results, in this embodiment, obtaining the 3D point cloud data of the ship's intermediate products further includes: Obtaining the color data of the ship's intermediate products through an RGB sensor carried by the drone, and correlating the color data with the 3D point cloud data.

[0064] As Figure 3 shown, removing the noise points from the 3D point cloud data includes:

[0065] Step S210: Subtract the R, G, and B values of the color data from the R, G, and B values of the preset color respectively, take the absolute value of the differences of R, G, and B and sum them to obtain the color difference.

[0066] Step S220: Mark the 3D point cloud data with a color difference greater than the second threshold as non-hull structure noise points.

[0067] Step S230: Delete the non-hull structure noise points from the 3D point cloud data.

[0068] In this embodiment, the color difference between the non-hull structure and the ship's intermediate products is used to remove the 3D point cloud formed by the non-hull structure, avoiding misidentifying the non-hull structure as overinstalled components during the subsequent registration process.

[0069] In actual ship's intermediate products, some components are in narrow or enclosed spaces, and indoor light may not reach, resulting in a large deviation between the color data obtained by the RGB sensor and the actual value. To improve the accuracy of color recognition, in this embodiment, the drone is also equipped with a fill light, and the color data is obtained under the illumination of the fill light.

[0070] In this embodiment, removing the noise points from the three-dimensional point cloud data further includes: step S240, removing the isolated noise points from the three-dimensional point cloud data through Gaussian filtering. The isolated noise points may come from the measurement noise of the three-dimensional scanning instrument. In this embodiment, by removing the isolated noise points, the accuracy of the integrity detection result is further improved.

[0071] To further reduce manual participation and improve the intelligence level of the integrity inspection of ship intermediate products, the unmanned aerial vehicle (UAV) plans the flight route through the BUG1 algorithm. In this embodiment, positioning markers can be set on the ship intermediate products to facilitate the UAV to establish the pose relationship relative to the physical hull according to the SLAM three-dimensional laser scanner, perceive the corresponding pose of the UAV relative to the actual environment, and thus automatically plan the route. The BUG1 algorithm will be repeatedly called until the complete three-dimensional point cloud data is constructed, and the UAV returns to the starting position.

[0072] Taking the integrity inspection of the 501 section of the ship as an example, the integrity inspection method of the ship intermediate products based on the UAV in this embodiment is described below.

[0073] First, mount the SLAM three-dimensional laser scanner and the RGB sensor on the UAV with the function of automatic collision avoidance, and ensure that the UAV is connected to the edge server through the workshop 5G network. In addition, set the marker pose for UAV positioning calibration in the theoretical three-dimensional model of the 501 section in the edge server, and attach markers at the corresponding positions on the physical hull of the 501 section.

[0074] It is determined that the RGB values of the actual hull outer surface color are Rs = 255, Gs = 100, Bs = 150, and the RGB values of the outfitting part outer surface are Ro = 100, Go = 180, Bo = 60. The coating colors of the non-ship intermediate product parts such as scaffolds and brackets are Rq = 150, Gq = 230, Bq = 0. The colors obtained by the RGB sensor are Rv, Gv, Bv.

[0075] Take the differences between the three values of Rv, Gv, Bv and Rs, Gs, Bs and Rq, Gq, Bq respectively, take the absolute values and sum them. When the sum of the absolute values of the two is greater than 100, the edge server retains the part belonging to the ship intermediate product in the model transmitted back by the RGB sensor on the UAV according to the color, and deletes the other parts.

[0076] After the scanning is completed, the three-dimensional point cloud data of the 501-section hull is transmitted to the edge server through the 5G network for point cloud data processing. First, remove the noise points with colors different from those of the hull and outfitting parts on the 501 section, then use methods such as Gaussian denoising to remove the isolated noise points, and finally use the depth data continuity clustering method to remove the noise points that do not belong to the hull again.

[0077] Use the built-in functions of software such as CATIA and Geomagic Verify to achieve the registration of denoised point cloud data and the theoretical 3D model. For example, in the production design model of the 501 section, there are two part models, namely the BK1A bracket and the TB252-A1 panel. In the superposition state of these two hull models, if there is no point cloud distribution around the production design models of the above two plates, it indicates that this part of the model belongs to the unpaired model area. Put the part models of the BK1A bracket and the TB252-A1 panel into the set A of unpaired model blocks.

[0078] If there is point cloud in a certain area but there is no part model at the corresponding position, it indicates that there are other parts on the physical hull that are wrongly assembled to this position. Then put all the point cloud blocks b1, b2... bj... bn in the areas with the above situation into the set B of unpaired point cloud blocks, where the point cloud blocks b1, b2... bj... bn can be formed by the depth continuity clustering method.

[0079] Pairwise register the elements in set A and set B, and incorporate the unpaired model blocks with a registration error less than the first threshold into set C. Set C contains 3D part models that are similar in shape to set A, that is, misassembled unpaired model blocks. Incorporate the unpaired point cloud blocks with a registration error less than the first threshold into set D. Set D contains misassembled unpaired point cloud blocks that are similar in shape to set B.

[0080] Find the difference set between set A and set D to obtain the overassembled unpaired point cloud blocks.

[0081] Find the difference set between set B and set C to obtain the missing-assembled unpaired model blocks.

[0082] After the inspector clicks to confirm, the missing-assembled, mispositioned, and overassembled parts will be recorded in the integrity inspection list of the edge server. The list will form a 501-section integrity inspection package together with the visualized model deviation analysis results and be sent to the inspection reviewer.

[0083] The protection scope of the integrity inspection method for ship intermediate products based on drones described in the embodiments of this application is not limited to the execution order of the steps listed in this embodiment. Any solutions achieved by adding or subtracting steps of the prior art and replacing steps according to the principles of this application are included in the protection scope of this application.

[0084] This embodiment provides an integrity inspection device for ship intermediate products based on drones. The integrity inspection device for ship intermediate products based on drones includes:

[0085] A 3D measurement module for acquiring the 3D point cloud data of the ship intermediate product;

[0086] A denoising module for removing the noise points in the 3D point cloud data to obtain denoised point cloud data;

[0087] A primary registration module, configured to register the denoised point cloud data with a theoretical three-dimensional model to obtain an overlapping area and a non-overlapping area; within the overlapping area, both the theoretical three-dimensional model and the corresponding denoised point cloud data exist; the non-overlapping area includes an unpaired point cloud area and an unpaired model area, and the unpaired model area includes a plurality of unpaired model blocks, and each unpaired model block corresponds to a three-dimensional model of a part or a component.

[0088] A block division module, configured to divide the point cloud data in the unpaired point cloud area into a plurality of unpaired point cloud blocks by using a clustering algorithm, and each unpaired point cloud block corresponds to a ship part or a ship component.

[0089] A secondary registration module, configured to register the plurality of unpaired model blocks and the plurality of unpaired point cloud blocks pairwise, calculate a registration error, and mark a pair of unpaired model blocks and unpaired point cloud blocks with a registration error less than a first threshold as misassembled unpaired model blocks and misassembled unpaired point cloud blocks respectively.

[0090] A multi-assembly statistics module, configured to remove the misassembled unpaired point cloud blocks from the unpaired point cloud blocks to obtain multi-assembled unpaired point cloud blocks.

[0091] A missing-assembly statistics module, configured to remove the misassembled unpaired model blocks from the unpaired model blocks to obtain missing-assembled unpaired model blocks.

[0092] A report generation module, configured to generate an integrity inspection report for a ship intermediate product according to the misassembled unpaired model blocks, the misassembled unpaired point cloud blocks, the multi-assembled unpaired point cloud blocks, and the missing-assembled unpaired model blocks.

[0093] This embodiment provides an integrity inspection system for a ship intermediate product based on a drone. The integrity inspection system for a ship intermediate product based on a drone includes:

[0094] A drone, which is equipped with a SLAM three-dimensional laser scanner, and the SLAM three-dimensional laser scanner is configured to obtain three-dimensional point cloud data of the ship intermediate product.

[0095] The edge server is connected to the drone for communication, and the edge server is used to: remove noise points in the three-dimensional point cloud data to obtain denoised point cloud data; align the denoised point cloud data with the theoretical three-dimensional model to obtain overlapping areas and non-overlapping areas; in the overlapping area, the theoretical three-dimensional model and the corresponding denoised point cloud data exist at the same time; the non-overlapping area includes an unpaired point cloud area and an unpaired model area, and the unpaired model area includes a plurality of unpaired model blocks, each of which corresponds to a three-dimensional model of a part or a three-dimensional model of a component; and the unpaired point cloud area is aligning the theoretical three-dimensional model with the theoretical three-dimensional model to obtain overlapping areas and non-overlapping areas. The point cloud data of the point cloud area is divided into a plurality of unpaired point cloud blocks, each of which corresponds to a ship part or a ship component; the plurality of unpaired model blocks and the plurality of unpaired point cloud blocks are registered in pairs, and the registration error is calculated, and a pair of unpaired model blocks and unpaired point cloud blocks whose registration error is less than a first threshold are marked as mis-installed unpaired model blocks and mis-installed unpaired point cloud blocks respectively; the mis-installed unpaired point cloud blocks are removed from the unpaired point cloud blocks to obtain multiple-installed unpaired point cloud blocks; the mis-installed unpaired model blocks are removed from the unpaired model blocks to obtain missed unpaired model blocks;

[0096] A display is used for three-dimensionally displaying the mis-installed unpaired model blocks, the mis-installed unpaired point cloud blocks, the over-installed unpaired point cloud blocks and the missing unpaired model blocks.

[0097] This embodiment can complete the remote detection of ship intermediate products and intuitively present the detection results, which is conducive to production management personnel to make production adjustments in a timely manner.

[0098] Specifically, in this embodiment, the edge server and the drone are connected via a 5G network to improve the real-time performance of data transmission and improve detection efficiency.

[0099] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0100] The module / unit described as a separate component may or may not be physically separated. The components shown as modules / units may or may not be physical modules, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected according to actual needs to achieve the objectives of the embodiments of the present application. For example, in various embodiments of the present application, the functional modules / units can be integrated into one processing module, or each module / unit can exist physically alone, or two or more modules / units can be integrated into one module / unit.

[0101] Those of ordinary skill in the art should also be able to further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0102] As Figure 4 shown, this embodiment also provides an electronic device, which is a mobile device such as a mobile phone, a PAD, a wearable device, an intelligent AI device, etc. for users; it includes a memory and a processor, and the processor is used to execute the computer program stored in the memory so that the electronic device executes the integrity inspection method of the ship intermediate product based on the drone in the above embodiments.

[0103] It should be further noted that the above-mentioned system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries).

[0104] The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0105] The above-mentioned storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid state drive, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0106] The descriptions of the processes or structures corresponding to the above respective drawings each have their own focuses. For parts not detailed in a certain process or structure, reference may be made to the relevant descriptions of other processes or structures.

[0107] The above embodiments are only illustrative of the principles and effects of the present application, and are not intended to limit the present application. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed by the present application should still be covered by the claims of the present application.

Claims

1. An integrity inspection method for ship intermediate products based on an unmanned aerial vehicle, characterized in that, the integrity inspection method for ship intermediate products based on an unmanned aerial vehicle comprises the following steps: Obtain the three-dimensional point cloud data of the ship intermediate product; Remove the noise points in the three-dimensional point cloud data to obtain denoised point cloud data; Register the denoised point cloud data with the theoretical three-dimensional model to obtain an overlapping area and a non-overlapping area; within the overlapping area, both the theoretical three-dimensional model and the corresponding denoised point cloud data exist; the non-overlapping area includes an unpaired point cloud area and an unpaired model area, and the unpaired model area includes a plurality of unpaired model blocks, and each unpaired model block corresponds to a part three-dimensional model or a component three-dimensional model; Divide the point cloud data in the unpaired point cloud area into a plurality of unpaired point cloud blocks, and each unpaired point cloud block corresponds to a ship part or a ship component; Register the plurality of unpaired model blocks and the plurality of unpaired point cloud blocks pairwise, and calculate the registration error. Mark a pair of unpaired model blocks and unpaired point cloud blocks with a registration error less than the first threshold as misassembled unpaired model blocks and misassembled unpaired point cloud blocks respectively; Remove the misassembled unpaired point cloud blocks from the unpaired point cloud blocks to obtain overassembled unpaired point cloud blocks; Remove the misassembled unpaired model blocks from the unpaired model blocks to obtain missing-assembled unpaired model blocks; Generate an integrity inspection report for the ship intermediate product according to the misassembled unpaired model blocks, the misassembled unpaired point cloud blocks, the overassembled unpaired point cloud blocks, and the missing-assembled unpaired model blocks.

2. The integrity inspection method for ship intermediate products based on an unmanned aerial vehicle according to claim 1, characterized in that, the obtaining of the three-dimensional point cloud data of the ship intermediate product includes: scanning the physical object of the ship intermediate product by a SLAM three-dimensional laser scanner carried by the unmanned aerial vehicle to obtain the three-dimensional point cloud data.

3. The integrity inspection method for ship intermediate products based on an unmanned aerial vehicle according to claim 2, characterized in that, the obtaining of the three-dimensional point cloud data of the ship intermediate product further includes: obtaining the color data of the ship intermediate product by an RGB sensor carried by the unmanned aerial vehicle, and correlating the color data with the three-dimensional point cloud data; the removing of the noise points in the three-dimensional point cloud data includes: Subtract the R, G, and B values of the color data from the R, G, and B values of a preset color respectively, take the absolute value of the difference of R, G, and B and sum them to obtain a color difference; Mark the three-dimensional point cloud data with a color difference greater than the second threshold as non-hull structure noise points; Delete the non-hull structure noise points from the three-dimensional point cloud data.

4. The integrity inspection method for ship intermediate products based on an unmanned aerial vehicle according to claim 3, characterized in that, the unmanned aerial vehicle is further equipped with a fill light, and the color data is obtained under the illumination of the fill light.

5. The integrity inspection method for ship intermediate products based on an unmanned aerial vehicle according to claim 2, characterized in that, Removing the noise points in the three-dimensional point cloud data further includes: removing isolated noise points in the three-dimensional point cloud data through Gaussian filtering.

6. The method for integrity inspection of ship intermediate products based on an unmanned aerial vehicle according to claim 2, wherein, the unmanned aerial vehicle plans a flight route through the BUG1 algorithm.

7. An electronic device, including a memory and a processor, wherein, the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method for integrity inspection of ship intermediate products based on an unmanned aerial vehicle according to any one of claims 1 to 6.

8. An integrity inspection device for ship intermediate products based on an unmanned aerial vehicle, wherein, the integrity inspection device for ship intermediate products based on an unmanned aerial vehicle includes: a three-dimensional measurement module, configured to obtain three-dimensional point cloud data of the ship intermediate product; a denoising module, configured to remove noise points in the three-dimensional point cloud data to obtain denoised point cloud data; a primary registration module, configured to register the denoised point cloud data with a theoretical three-dimensional model to obtain an overlapping area and a non-overlapping area; within the overlapping area, both the theoretical three-dimensional model and the corresponding denoised point cloud data exist; the non-overlapping area includes an unpaired point cloud area and an unpaired model area, and the unpaired model area includes a plurality of unpaired model blocks, and each unpaired model block corresponds to a part three-dimensional model or a component three-dimensional model; a block division module, configured to divide the point cloud data in the unpaired point cloud area into a plurality of unpaired point cloud blocks by using a clustering algorithm, and each unpaired point cloud block corresponds to a ship part or a ship component; a secondary registration module, configured to register the plurality of unpaired model blocks and the plurality of unpaired point cloud blocks in pairs, calculate the registration error, and mark a pair of unpaired model blocks and unpaired point cloud blocks with a registration error less than a first threshold as misassembled unpaired model blocks and misassembled unpaired point cloud blocks respectively; a multi-installation statistics module, configured to remove the misassembled unpaired point cloud blocks from the unpaired point cloud blocks to obtain multi-installation unpaired point cloud blocks; a missing-installation statistics module, configured to remove the misassembled unpaired model blocks from the unpaired model blocks to obtain missing-installation unpaired model blocks; a report generation module, configured to generate an integrity inspection report for the ship intermediate product according to the misassembled unpaired model blocks, the misassembled unpaired point cloud blocks, the multi-installation unpaired point cloud blocks, and the missing-installation unpaired model blocks.

9. An integrity inspection system for ship intermediate products based on an unmanned aerial vehicle, wherein, the integrity inspection system for ship intermediate products based on an unmanned aerial vehicle includes: an unmanned aerial vehicle, which is equipped with a SLAM three-dimensional laser scanner, and the SLAM three-dimensional laser scanner is used to obtain three-dimensional point cloud data of the ship intermediate product; An edge server, communicatively connected to the drone, is configured to: remove noise points from the three-dimensional point cloud data to obtain denoised point cloud data; register the denoised point cloud data with a theoretical three-dimensional model to obtain an overlapping region and a non-overlapping region; within the overlapping region, both the theoretical three-dimensional model and the corresponding denoised point cloud data exist; the non-overlapping region includes an unpaired point cloud area and an unpaired model area, the unpaired model area includes a plurality of unpaired model blocks, and each unpaired model block corresponds to a part three-dimensional model or a component three-dimensional model; use a clustering algorithm to divide the point cloud data in the unpaired point cloud area into a plurality of unpaired point cloud blocks, and each unpaired point cloud block corresponds to a ship part or a ship component; register the plurality of unpaired model blocks and the plurality of unpaired point cloud blocks pairwise, and calculate the registration error, and mark a pair of unpaired model blocks and unpaired point cloud blocks with a registration error less than a first threshold as a misloaded unpaired model block and a misloaded unpaired point cloud block respectively; remove the misloaded unpaired point cloud block from the unpaired point cloud blocks to obtain overloaded unpaired point cloud blocks; remove the misloaded unpaired model block from the unpaired model blocks to obtain missing unpaired model blocks; A display, configured to perform three-dimensional display on the misloaded unpaired model block, the misloaded unpaired point cloud block, the overloaded unpaired point cloud block, and the missing unpaired model block.

10. The integrity inspection system for ship intermediate products based on drones according to claim 9, wherein, the edge server is connected to the drone through a 5G network.

Images