Cabin modeling method and system based on three-dimensional positioning and two-dimensional mapping

Abstract

The invention relates to the technical field of cabin modeling, in particular to a cabin modeling method and system based on three-dimensional positioning and two-dimensional mapping, and the method comprises the following steps: carrying out the registration reconstruction of local coal pile point cloud data of each modeling region, and obtaining a cabin coal pile three-dimensional static model; identifying the relevance of the local coal pile point cloud data of each modeling area by using a neural network to obtain a dynamic relevance model of the local coal pile point cloud data of each modeling area; and combining a cabin coal pile three-dimensional static model with the dynamic association model to obtain a cabin coal pile three-dimensional dynamic model for dynamically sensing the cabin coal pile. According to the method, the dynamic correlation model is combined to the cabin coal pile three-dimensional static model to obtain the cabin coal pile three-dimensional dynamic model, so that the coal pile three-dimensional model can adapt to the dynamic change of the coal pile when the coal pile changes in the cabin cleaning operation process.

Description

Technical Field

[0001] The present invention relates to the technical field of cabin modeling, and in particular to a cabin modeling method and system based on three-dimensional positioning and two-dimensional mapping. Background Art

[0002] Nowadays, the main tools for cargo hold cleaning in terminal operations are ship unloaders and pushers. In cargo hold cleaning operations, in order to facilitate ship unloader workers to accurately perceive the three-dimensional shape of the coal pile in the cabin, the remaining coal volume and the remaining working time of the cargo hold cleaning operation can be calculated during the operation of the pusher. A three-dimensional model of the coal pile is constructed through three-dimensional reconstruction to avoid repeated visual grabbing and raking of the coal pile inside the cabin to complete the cleaning work, and avoid the limitations of non-closed-loop operations and manual visual capabilities, which leads to reduced efficiency of cargo hold cleaning operations.

[0003] At present, cabin modeling usually uses laser radar to perform laser scanning of coal piles, obtain point cloud data of coal piles in the cabin, and then reconstruct the three-dimensional model of the coal pile based on the point cloud data of the coal pile. However, this modeling method that only uses laser scanning can only obtain the three-dimensional coordinates of the coal pile, and it is difficult to grasp the color and texture of the modeling object. At the same time, the modeled three-dimensional model of the coal pile is the modeling result of a fixed moment. During the cabin cleaning operation, when the coal pile changes, the three-dimensional model of the coal pile is difficult to adapt to the dynamic changes of the coal pile, accurately display the current actual shape of the coal pile, and lacks dynamic perception. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for cabin modeling based on three-dimensional positioning and two-dimensional mapping, so as to solve the technical problems in the prior art that the modeling method using laser scanning can only obtain the three-dimensional coordinates of the coal pile, it is difficult to grasp the color and texture of the modeling object, and the dynamic perception of the modeled three-dimensional model of the coal pile is insufficient.

[0005] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0006] A cabin modeling method based on three-dimensional positioning and two-dimensional mapping includes the following steps:

[0007] Divide areas such as the cabin area into multiple modeling areas;

[0008] In each modeling area, the laser radar and industrial camera mounted on the mobile platform are used to perform three-dimensional positioning and two-dimensional mapping of the coal pile in the modeling area, and obtain the local coal pile point cloud data of each modeling area;

[0009] The local coal pile point cloud data of each modeling area is registered and reconstructed to obtain a three-dimensional static model of the coal pile in the cabin;

[0010] The three-dimensional static model of the coal pile in the ship cabin is artificially adjusted to obtain multiple groups of local coal pile adjustment point cloud data in each modeling area;

[0011] Using a neural network to adjust point cloud data based on multiple groups of local coal piles in each modeling area, the correlation of the local coal pile point cloud data in each modeling area is identified to obtain a dynamic correlation model of the local coal pile point cloud data in each modeling area;

[0012] The three-dimensional static model of the coal pile in the ship's hold is combined with the dynamic association model to obtain a three-dimensional dynamic model of the coal pile in the ship's hold for dynamic perception of the coal pile in the ship's hold.

[0013] As a preferred solution of the present invention, a method for three-dimensionally locating a coal pile in a modeling area by using a laser radar in the modeling area includes:

[0014] In the modeling area, a laser radar is used to perform laser scanning on a coal pile in the modeling area to obtain laser scanning data of the coal pile in the modeling area, including a laser range measurement value, a horizontal angle value, a vertical angle value, and a horizontal offset angle of a laser channel;

[0015] The three-dimensional coordinate data of the coal pile in the modeling area are converted by the laser scanning data in the modeling area;

[0016] The conversion formula of the three-dimensional coordinate data is:

[0017]

[0018] In the formula, x kj is the x-axis coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area, and y kj is the y-axis coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area, z kj is the z-axis coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area, r kj is the laser ranging value obtained by the jth channel of the laser radar in the kth modeling area, α kj is the horizontal angle value obtained by the jth channel of the lidar in the kth modeling area, δ kj is the horizontal offset angle value of the jth channel of the laser radar in the kth modeling area, β kj The vertical angle value obtained by the jth channel of the lidar in the kth modeling area;

[0019] Among them, the three-dimensional coordinate data of the coal pile in the kth modeling area is {(x kj ,y kj ,z kj )|j∈[1,n]}, where (x kj ,ykj ,z kj ) is the three-dimensional coordinate of the coal pile corresponding to the j-th channel of the laser radar in the k-th modeling area, and n is the number of channels of the laser radar.

[0020] As a preferred solution of the present invention, a method for constructing a two-dimensional map of a coal pile in a modeling area using an industrial camera in the modeling area includes:

[0021] Adjust the sampling frequency of the industrial camera to be consistent with the sampling frequency of the lidar;

[0022] An industrial camera is used in the modeling area to collect images of the coal pile in the modeling area to obtain two-dimensional image data of the coal pile in the modeling area.

[0023] As a preferred solution of the present invention, a method for constructing local coal pile point cloud data by using three-dimensional coordinate data and two-dimensional image data of the coal pile in the modeling area includes:

[0024] The three-dimensional coordinate data and two-dimensional image data of the coal pile in the modeling area are arranged into a coordinate system;

[0025] The three-dimensional coordinate data and two-dimensional image data of the coal pile in the modeling area after the coordinate system is set are combined according to the corresponding coordinates to obtain the local point cloud data of the coal pile in the modeling area;

[0026] The formula of the coordinate system 1 is:

[0027]

[0028] In the formula, (u kj ,v kj ) is (x kj ,y kj ,z kj ) is converted to the pixel coordinates in the pixel coordinate system where the two-dimensional image data is located, M is the parameter matrix of the industrial camera, R0 is the rotation matrix between the laser radar coordinate system and the industrial camera coordinate system, t0 is the translation vector between the laser radar coordinate system and the industrial camera coordinate system, (x kj ,y kj ,z kj ) is the three-dimensional coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area.

[0029] As a preferred solution of the present invention, the registration and reconstruction method of the three-dimensional static model of the coal pile in the ship cabin includes:

[0030] The point cloud data located in the adjacent edge areas of the local coal pile point cloud data of two adjacent modeling areas are respectively used as the to-be-registered point cloud set A and the target point cloud set B;

[0031] The ICP registration algorithm is used to perform registration operations on the point cloud set A to be registered and the target point cloud set B to obtain a rotation matrix and translation vector that unifies the coordinates of two adjacent modeling areas;

[0032] The rotation matrix and translation vector are used to unify the coordinates of two adjacent modeling areas in turn, and the two adjacent modeling areas after the coordinate unification are fused in the adjacent edge area according to the corresponding coordinates to obtain the three-dimensional static model of the coal pile in the cabin {P k |k∈[1,N]}, where P k is the local coal pile point cloud data of the kth modeling area, and N is the total number of modeling areas;

[0033] The registration equation is:

[0034]

[0035] In the formula, F(R AB ,t AB ) is the objective function of the ICP registration algorithm, R AB is the rotation matrix between the point cloud set A to be registered and the target point cloud set B, t AB is the translation vector between the point cloud set A to be registered and the target point cloud set B, b i is the i-th point cloud data in the target point cloud set B, a i is the i-th point cloud data in the point cloud set A to be registered, and m is the total number of point cloud data in the point cloud set A to be registered and the target point cloud set B.

[0036] As a preferred solution of the present invention, a method for acquiring multiple groups of local coal pile adjustment point cloud data in each modeling area includes:

[0037] The shapes in the three-dimensional static model of the coal pile in the ship's hold are artificially deleted for multiple times to obtain multiple three-dimensional missing models of the coal pile in the ship's hold;

[0038] Extract the local coal pile adjustment point cloud data of each modeling area from the three-dimensional missing model of each ship hold coal pile;

[0039] The local coal pile adjustment point cloud data of each modeling area is extracted from the three-dimensional missing model of the coal pile in each cabin, and combined into multiple groups of local coal pile adjustment point cloud data of each modeling area.

[0040] As a preferred solution of the present invention, the method for constructing the dynamic association model includes:

[0041] Multiple groups of local coal pile adjustment point cloud data in two adjacent modeling areas are used as data samples for training the mapping relationship between the input and output of the BP neural network;

[0042] Using a BP neural network to establish a mapping relationship between an input and an output of the BP neural network based on the data samples, and obtaining the dynamic correlation model that characterizes the correlation between the local coal pile point cloud data of two adjacent modeling areas;

[0043] The dynamic association model is:

[0044]

[0045] Where P (k+1)o is the local coal pile point cloud data of the k+1th modeling area output by the dynamic association model, P ko is the local coal pile point cloud data of the kth modeling area output by the dynamic association model, P k is the local coal pile point cloud data of the kth modeling area, P (k+1) is the local coal pile point cloud data of the k+1th modeling area, and BP is a neural network;

[0046] Among them, the loss function for training the dynamic association model is:

[0047]

[0048] Where loss is the loss function for training the dynamic association model. P (k+1)o and P (k+1) The L2 norm between P ko and P k The L2 norm between .

[0049] As a preferred embodiment of the present invention, the method for constructing the three-dimensional dynamic model of the coal pile in the ship cabin comprises:

[0050] The three-dimensional dynamic model of coal pile in the ship's hold is combined with the dynamic correlation model to obtain the three-dimensional dynamic model of coal pile in the ship's hold;

[0051] The three-dimensional dynamic model of the coal pile in the ship's hold is:

[0052] {P 1o ,…,P (h-1)o ,P h ,P (h+1)o ,P (h+2)o ,…,P No};

[0053] P lo =BP(P (l+1)o ),l∈[1,(h-1)];

[0054] P (r+1)o =BP(P ro ),r∈[h,(N-1)];

[0055] Where P h is the point cloud data of the local coal pile that undergoes dynamic changes in the hth modeling area, P 1o , P (h-1)o , P (h+1)o , P (h+2)o and P No The dynamic association model is based on P h Output the local coal pile point cloud data of the 1st, (h-1), (h+1), (h+2) and Nth modeling areas, where N is the total number of modeling areas, P lo and P (l+1)o is the local coal pile point cloud data of the lth and (l+1)th modeling areas located on the left side of the hth modeling area output by the dynamic association model, P ro and P (r+1)o It is the local coal pile point cloud data of the rth and (r+1)th modeling areas on the right side of the hth modeling area output by the dynamic association model, and BP is a neural network.

[0056] As a preferred solution of the present invention, point cloud data of a local coal pile that undergoes dynamic changes is obtained by re-positioning the coal pile in the modeling area in three dimensions and re-mapping it in two dimensions using a laser radar and an industrial camera mounted on a mobile platform.

[0057] As a preferred solution of the present invention, the present invention provides a cabin modeling system based on three-dimensional positioning and two-dimensional mapping, which is applied to the cabin modeling method based on three-dimensional positioning and two-dimensional mapping, and the system includes:

[0058] A region division unit, used to divide a cabin area and the like into a plurality of modeling areas;

[0059] The regional modeling unit is used to perform three-dimensional positioning and two-dimensional mapping of the coal pile in each modeling area by using the laser radar and industrial camera mounted on the mobile platform, so as to obtain the local coal pile point cloud data of each modeling area;

[0060] The regional combination unit is used to register and reconstruct the local coal pile point cloud data of each modeling area to obtain a three-dimensional static model of the coal pile in the cabin;

[0061] A dynamic association unit is used to artificially adjust the morphology of the three-dimensional static model of the coal pile in the ship cabin to obtain multiple groups of local coal pile adjustment point cloud data in each modeling area;

[0062] and for using a neural network to adjust point cloud data based on multiple groups of local coal piles in each modeling area, identifying the correlation of the local coal pile point cloud data in each modeling area, and obtaining a dynamic correlation model of the local coal pile point cloud data in each modeling area;

[0063] The dynamic modeling unit is used to combine the three-dimensional static model of the coal pile in the ship's hold with the dynamic association model to obtain a three-dimensional dynamic model of the coal pile in the ship's hold for dynamic perception of the coal pile in the ship's hold.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] The present invention utilizes a laser radar and an industrial camera mounted on a mobile platform to perform three-dimensional positioning and two-dimensional mapping of the coal pile in the modeling area. On the three-dimensional coordinates of the coal pile obtained by the laser scanning modeling method, the color and texture of the modeling object are obtained through image acquisition. At the same time, a dynamic association model that characterizes the correlation of local coal pile point cloud data in each modeling area is constructed by using a neural network. The model is combined with the three-dimensional static model of the coal pile in the cabin to obtain a three-dimensional dynamic model of the coal pile in the cabin. When the coal pile changes during the cabin cleaning operation, the three-dimensional model of the coal pile can adapt to the dynamic changes of the coal pile and accurately display the current actual shape of the coal pile. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0067] Figure 1 A flow chart of a method for cabin modeling based on three-dimensional positioning and two-dimensional mapping provided by an embodiment of the present invention;

[0068] Figure 2 A block diagram of a cabin modeling system based on three-dimensional positioning and two-dimensional mapping provided by an embodiment of the present invention;

[0069] Figure 3 A schematic diagram of local coal pile point cloud data provided by an embodiment of the present invention;

[0070] Figure 4 A schematic diagram of a local coal pile real scene in a modeling area provided by an embodiment of the present invention;

[0071] Figure 5 A schematic diagram of a three-dimensional model of a coal pile provided in an embodiment of the present invention;

[0072] Figure 6 A schematic diagram of the overall real scene of a coal pile in a ship's hold provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] like Figure 1 As shown, the present invention provides a cabin modeling method based on three-dimensional positioning and two-dimensional mapping, comprising the following steps:

[0075] Divide areas such as the cabin area into multiple modeling areas;

[0076] In each modeling area, the laser radar and industrial camera mounted on the mobile platform are used to perform three-dimensional positioning and two-dimensional mapping of the coal pile in the modeling area, and obtain the local coal pile point cloud data of each modeling area;

[0077] The local coal pile point cloud data of each modeling area is registered and reconstructed to obtain a three-dimensional static model of the coal pile in the cabin;

[0078] The three-dimensional static model of the coal pile in the ship cabin is artificially adjusted to obtain multiple groups of local coal pile adjustment point cloud data in each modeling area;

[0079] Using a neural network to adjust point cloud data based on multiple groups of local coal piles in each modeling area, the correlation of the local coal pile point cloud data in each modeling area is identified to obtain a dynamic correlation model of the local coal pile point cloud data in each modeling area;

[0080] The three-dimensional static model of the coal pile in the ship's hold is combined with the dynamic association model to obtain a three-dimensional dynamic model of the coal pile in the ship's hold for dynamic perception of the coal pile in the ship's hold. (To achieve rapid coal pile modeling under changes in the coal pile in the ship's hold)

[0081] The present invention divides the overall modeling of the coal pile in the hold into multiple local areas and uses point cloud data to perform modeling. After completing the modeling of the coal pile in the local area (i.e., obtaining the point cloud data of the local coal pile in the local area), it is aligned and combined into an overall three-dimensional model of the coal pile in the hold. Usually, the overall three-dimensional model is a three-dimensional model of the coal pile before any cabin cleaning operation is performed, and is an initial static model of the coal pile (i.e., a three-dimensional static model of the coal pile in the hold). The present invention allows the constructed three-dimensional model of the coal pile to be dynamically updated as the cabin cleaning operation proceeds, reflecting the actual form of the coal pile formed by the changes in the coal pile during the cabin cleaning operation.

[0082] Therefore, the present invention establishes the correlation between the point cloud data used for three-dimensional modeling of coal piles in each local area (modeling area), that is, the correlation between the local coal pile point cloud data of each modeling area, and encapsulates it into a neural network to form a dynamic correlation model. It can directly derive the local coal pile point cloud data of the adjacent modeling area based on the local coal pile point cloud data of the modeling area, thereby achieving the goal of rapid modeling of coal piles in adjacent modeling areas.

[0083] The present invention combines the dynamic association model with the three-dimensional static model of the coal pile in the cabin, so that in the process of cabin cleaning, the coal pile morphology of a certain modeling area changes due to the operation of coal pile grabbing and raking to clear out, and at this time, it is only necessary to re-acquire the local coal pile point cloud data of the modeling area, that is, to obtain a new local three-dimensional model of the coal pile in the area. At this time, it is only necessary to input the new local coal pile point cloud data of the modeling area into the dynamic association model, and use the correlation between adjacent modeling areas to obtain the new local coal pile point cloud data of all remaining modeling areas. After that, the new local coal pile point cloud data of all remaining modeling areas and the new local coal pile point cloud data of the modeling area are aligned to form a three-dimensional model of the coal pile after the operation of coal pile grabbing and raking to clear out. Therefore, after the dynamic association model is combined with the three-dimensional static model of the coal pile in the cabin, a three-dimensional dynamic model of the coal pile in the cabin can be formed, which has dynamic perception ability, can be adaptively updated according to the change of coal pile morphology, accurately perceive various changes in the coal pile, and construct a three-dimensional model of the coal pile that reflects the actual morphology of the coal pile.

[0084] When acquiring point cloud data of coal piles in each modeling area, the present invention utilizes laser radar and industrial cameras to perform three-dimensional positioning and two-dimensional mapping of the coal piles in the modeling area, respectively. Among them, the laser radar scans the coal pile to obtain point cloud data, which is actually three-dimensional coordinates, that is, it reflects the spatial structure of the coal pile, and the industrial camera can collect the color and texture characteristics of the coal pile, so that the point cloud data obtained by the laser radar is combined with the pixel color and texture, so that the coal pile modeling effect is better and the details are richer.

[0085] When acquiring point cloud data of coal piles in each modeling area, the present invention uses laser radar and industrial cameras to perform three-dimensional positioning and two-dimensional mapping of the coal piles in the modeling area, as follows:

[0086] The method for three-dimensionally locating a coal pile in a modeling area by using a laser radar includes:

[0087] In the modeling area, laser scanning is performed on the coal pile in the modeling area by using a laser radar to obtain laser scanning data of the coal pile in the modeling area, laser ranging values, horizontal angle values, vertical angle values ​​of the laser scanning data, and horizontal offset angles of the laser channels;

[0088] The three-dimensional coordinate data of the coal pile in the modeling area is converted through the laser scanning data in the modeling area;

[0089] The conversion formula for three-dimensional coordinate data is:

[0090]

[0091] In the formula, x kj is the x-axis coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area, and y kj is the y-axis coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area, z kj is the z-axis coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area, r kj is the laser ranging value obtained by the jth channel of the laser radar in the kth modeling area, α kj is the horizontal angle value obtained by the jth channel of the lidar in the kth modeling area, δ kj is the horizontal offset angle value of the jth channel of the laser radar in the kth modeling area, β kj The vertical angle value obtained by the jth channel of the lidar in the kth modeling area;

[0092] Among them, the three-dimensional coordinate data of the coal pile in the kth modeling area is {(x kj ,y kj ,z kj )|j∈[1,n]}, where (x kj ,y kj ,z kj ) is the three-dimensional coordinate of the coal pile corresponding to the j-th channel of the laser radar in the k-th modeling area, and n is the number of channels of the laser radar.

[0093] And, a method for constructing a two-dimensional map of a coal pile in a modeling area by using an industrial camera in the modeling area includes:

[0094] Adjust the sampling frequency of the industrial camera to be consistent with the sampling frequency of the lidar;

[0095] An industrial camera is used in the modeling area to collect images of the coal pile in the modeling area to obtain two-dimensional image data of the coal pile in the modeling area.

[0096] The method for constructing local coal pile point cloud data by using the three-dimensional coordinate data and two-dimensional image data of the coal pile in the modeling area includes:

[0097] The three-dimensional coordinate data and two-dimensional image data of the coal pile in the modeling area are arranged into a coordinate system;

[0098] The three-dimensional coordinate data and two-dimensional image data of the coal pile in the modeling area after the coordinate system is set are combined according to the corresponding coordinates to obtain the local point cloud data of the coal pile in the modeling area;

[0099] The formula for coordinate system 1 is:

[0100]

[0101] In the formula, (u kj ,v kj ) is (x kj ,y kj ,z kj ) is converted to the pixel coordinates in the pixel coordinate system where the two-dimensional image data is located, M is the parameter matrix of the industrial camera, R0 is the rotation matrix between the laser radar coordinate system and the industrial camera coordinate system, t0 is the translation vector between the laser radar coordinate system and the industrial camera coordinate system, (x kj ,y kj ,z kj ) is the three-dimensional coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area.

[0102] Since the three-dimensional coordinate data and two-dimensional image data obtained by the laser radar have different coordinate systems and cannot be directly combined, the present invention first combines the three-dimensional coordinate data and two-dimensional image data of the coal pile in the modeling area into a coordinate system, as follows:

[0103] The three-dimensional coordinate data of the coal pile in the modeling area is converted from the coordinate system of the laser radar to the coordinate system of the industrial camera. The conversion formula is:

[0104]

[0105] In the formula, .x ′ kj ,y ′ kj ,z ′ kj / is (x kj ,y kj ,z kj ) is converted to the coordinates in the industrial camera coordinate system, R0 is the rotation matrix between the laser radar coordinate system and the industrial camera coordinate system, t0 is the translation vector between the laser radar coordinate system and the industrial camera coordinate system, (x kj ,y kj ,z kj ) is the three-dimensional coordinate of the coal pile corresponding to the j-th channel of the laser radar in the k-th modeling area;

[0106] The three-dimensional coordinate data of the coal pile in the modeling area is converted from the coordinate system of the industrial camera to the pixel coordinate system. The conversion formula is:

[0107]

[0108] In the formula, (u kj ,v kj ) is .x ′ kj ,y ′ kj ,z ′ kj / Convert to the pixel coordinates in the pixel coordinate system where the two-dimensional image data is located, M is the parameter matrix of the industrial camera, f x and f y It is an inherent parameter of the industrial camera and depends on the industrial camera. (u0, v0) is the offset coordinate between the optical center of the industrial camera and the far point of the pixel coordinate system.

[0109] Therefore, according to the above steps, the three-dimensional coordinate data and two-dimensional image data of the coal pile in the modeling area can be realized to form a coordinate system, that is:

[0110]

[0111] In the formula, (u kj ,v kj ) is (x kj ,y kj ,z kj ) is converted to the pixel coordinates in the pixel coordinate system where the two-dimensional image data is located, M is the parameter matrix of the industrial camera, R0 is the rotation matrix between the laser radar coordinate system and the industrial camera coordinate system, t0 is the translation vector between the laser radar coordinate system and the industrial camera coordinate system, (x kj ,y kj ,z kj ) is the three-dimensional coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area.

[0112] The present invention converts pixel coordinates (u kj ,v kj ) at the pixel RGB((u kj ,v kj )) and superimposed onto the three-dimensional coordinate data (x kj ,y kj ,z kj ), forming a colorful point cloud data {(x kj ,y kj ,z kj ),RGB((u kj ,v kj ))}, where RGB((u kj ,v kj )) is the two-dimensional image data, and the pixel coordinates (u kj ,vkj ), (x kj ,y kj ,z kj ) is the three-dimensional coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area.

[0113] Therefore, the local coal pile point cloud data of the kth modeling area is: {(x kj ,y kj ,z kj ),RGB.(u kj ,v kj ) / |j∈[1,n]}, n is the number of channels of the laser radar, such as Figure 3 As shown, the actual coal pile picture of the kth modeling area is as follows Figure 4 shown.

[0114] The registration and reconstruction method of the 3D static model of the coal pile in the ship's hold includes:

[0115] The point cloud data located in the adjacent edge areas of the local coal pile point cloud data of two adjacent modeling areas are respectively used as the to-be-registered point cloud set A and the target point cloud set B;

[0116] The ICP registration algorithm is used to perform registration operations on the point cloud set A to be registered and the target point cloud set B to obtain a rotation matrix and translation vector that unifies the coordinates of two adjacent modeling areas;

[0117] The rotation matrix and translation vector are used to unify the coordinates of two adjacent modeling areas in turn, and the two adjacent modeling areas after the coordinate unification are fused in the adjacent edge area according to the corresponding coordinates to obtain the three-dimensional static model of the coal pile in the ship cabin {P k |k∈[1,N]}, where P k is the local coal pile point cloud data of the kth modeling area, and N is the total number of modeling areas, such as Figure 6 As shown in the figure, the overall real picture of the coal pile in the ship's cabin is as follows Figure 5 As shown;

[0118] The registration formula is:

[0119]

[0120] In the formula, F(R AB ,t AB ) is the objective function of the ICP registration algorithm, R AB is the rotation matrix between the point cloud set A to be registered and the target point cloud set B, t AB is the translation vector between the point cloud set A to be registered and the target point cloud set B, b i is the i-th point cloud data in the target point cloud set B, a iis the i-th point cloud data in the point cloud set A to be registered, and m is the total number of point cloud data in the point cloud set A to be registered and the target point cloud set B.

[0121] Since the modeling area is divided from the entire cabin area, the edge areas of the modeling area are relatively overlapping, so they can be used for registration. The ICP algorithm can be used to make the Euclidean distance of the point cloud data of the corresponding points in the adjacent modeling edge areas the shortest, and the accurate rotation matrix and translation vector can be obtained to complete the fine registration of the local coal pile point cloud of the adjacent modeling areas.

[0122] Since the local coal pile point cloud data contains three-dimensional coordinate data and two-dimensional image data, when the ICP algorithm is used for solving, it can ensure that the solved rotation matrix and translation vector can make the coordinates of the corresponding points closest, and at the same time ensure that the solved rotation matrix and translation vector can make the corresponding point pixels closest, which is in line with practical significance, that is, adjacent points in two adjacent modeling areas should have similar three-dimensional coordinates and similar pixel values, so that high-precision alignment results can be obtained based on the local coal pile point cloud data obtained by the present invention. Therefore, it can be ensured that although the present invention divides the area for modeling, it is possible to obtain accurate three-dimensional modeling results of the coal pile through high-precision alignment, that is, the construction accuracy of the three-dimensional static model of the coal pile and the three-dimensional dynamic model of the coal pile is high.

[0123] Using singular value decomposition method Solve and obtain the rotation matrix R between the point cloud set A to be registered and the target point cloud set B AB And the translation vector t between the point cloud set A to be registered and the target point cloud set B AB , thereby obtaining the rotation matrix and translation vector for unifying the coordinates of the local coal pile point cloud data of the two adjacent modeling areas. The point cloud registration of the two modeling areas is completed according to the solved rotation matrix and translation vector. The registered point cloud data are combined to form a three-dimensional static model of the coal pile in the cabin.

[0124] The method for obtaining multiple groups of local coal pile adjustment point cloud data in each modeling area includes:

[0125] The shapes in the three-dimensional static model of the coal pile in the ship's hold are processed with artificial deletion multiple times to obtain multiple three-dimensional missing models of the coal pile in the ship's hold;

[0126] Extract the local coal pile adjustment point cloud data of each modeling area from the three-dimensional missing model of each ship hold coal pile;

[0127] The local coal pile adjustment point cloud data of each modeling area is extracted from the three-dimensional missing model of the coal pile in each cabin, and combined into multiple groups of local coal pile adjustment point cloud data of each modeling area.

[0128] The construction method of the dynamic association model includes:

[0129] Multiple groups of local coal pile adjustment point cloud data in two adjacent modeling areas are used as data samples for training the mapping relationship between the input and output of the BP neural network;

[0130] The BP neural network is used to establish a mapping relationship between the input and output of the BP neural network based on data samples, and a dynamic correlation model representing the correlation between the local coal pile point cloud data of two adjacent modeling areas is obtained;

[0131] The dynamic association model is:

[0132]

[0133] Where P (k+1)o is the local coal pile point cloud data of the k+1th modeling area output by the dynamic association model, P ko is the local coal pile point cloud data of the kth modeling area output by the dynamic association model, P k is the local coal pile point cloud data of the kth modeling area, P (k+1) is the local coal pile point cloud data of the k+1th modeling area, and BP is a neural network;

[0134] Among them, the loss function for training the dynamic association model is:

[0135]

[0136] Where loss is the loss function for training the dynamic association model. P (k+1)o and P (k+1) The L2 norm between P ko and P k The L2 norm between .

[0137] The present invention divides the overall modeling of the coal pile in the hold into multiple local areas and uses point cloud data to perform modeling. After completing the modeling of the coal pile in the local area (i.e., obtaining the point cloud data of the local coal pile in the local area), it is aligned and combined into an overall three-dimensional model of the coal pile in the hold. Usually, the overall three-dimensional model is a three-dimensional model of the coal pile before any cabin cleaning operation is performed, and is an initial static model of the coal pile (i.e., a three-dimensional static model of the coal pile in the hold). The present invention allows the constructed three-dimensional model of the coal pile to be dynamically updated as the cabin cleaning operation proceeds, reflecting the actual form of the coal pile formed by the changes in the coal pile during the cabin cleaning operation.

[0138] Therefore, the present invention establishes the correlation between the point cloud data used for three-dimensional modeling of coal piles in each local area (modeling area), that is, the correlation between the local coal pile point cloud data of each modeling area, and encapsulates it into a neural network to form a dynamic correlation model. It can directly derive the local coal pile point cloud data of the adjacent modeling area based on the local coal pile point cloud data of the modeling area, thereby achieving the goal of rapid modeling of coal piles in adjacent modeling areas.

[0139] The dynamic correlation model constructed by the present invention can realize the correlation of local coal pile point cloud data of adjacent modeling areas obtained from two directions, that is, it can construct k To P k+1 The data association in the direction is obtained, so as to realize the k Calculate P k+1 , can be constructed from P k+1 To P k The data association in the direction is obtained, so as to realize the k+1 Calculate P k , it is possible to obtain the local coal pile point cloud data of another modeling area through the local coal pile point cloud data of any modeling area in the adjacent modeling areas.

[0140] At the same time, when training the dynamic association model, the k Calculate P k+1 Accuracy P k+1 The error between the model output value and the true value) and the ... k+1 Calculate P k Accuracy P k The error between the model output value and the true value) is combined into a loss function, which can make the trained dynamic association model have high accuracy in obtaining the local coal pile point cloud data of another modeling area through the local coal pile point cloud data of any modeling area in the adjacent modeling areas.

[0141] The highly accurate dynamic association model can ensure the high accuracy of the three-dimensional dynamic model of the coal pile in the ship's hold, and the dynamic association model can realize rapid modeling of coal piles in adjacent modeling areas, which can ensure the modeling efficiency of the three-dimensional dynamic model of the coal pile in the ship's hold.

[0142] The construction method of the three-dimensional dynamic model of the coal pile in the ship's hold includes:

[0143] The three-dimensional dynamic model of coal pile in the ship's hold is combined with the dynamic correlation model to obtain the three-dimensional dynamic model of coal pile in the ship's hold;

[0144] The three-dimensional dynamic model of the coal pile in the ship's hold is:

[0145] {P 1o ,…,P(h-1)o ,P h ,P (h+1)o ,P (h+2)o ,…,P No};

[0146] P lo =BP(P (l+1)o ),l∈[1,(h-1)];

[0147] P (r+1)o =BP(P ro ),r∈[h,(N-1)];

[0148] Where P h is the point cloud data of the local coal pile that undergoes dynamic changes in the hth modeling area, P 1o , P (h-1)o , P (h+1)o , P (h+2)o and P No The dynamic association model is based on P h Output the local coal pile point cloud data of the 1st, (h-1), (h+1), (h+2) and Nth modeling areas, where N is the total number of modeling areas, P lo and P (l+1)o is the local coal pile point cloud data of the lth and (l+1)th modeling areas located on the left side of the hth modeling area output by the dynamic association model, P ro and P (r+1)o is the local coal pile point cloud data of the rth and (r+1)th modeling areas located on the right side of the hth modeling area output by the dynamic association model, BP is a neural network, P ho =P h .

[0149] The point cloud data of the dynamically changing local coal pile is obtained by re-positioning the coal pile in the modeling area in three dimensions and re-mapping it in two dimensions using the laser radar and industrial camera mounted on a mobile platform.

[0150] The present invention provides an embodiment:

[0151] Divide the cabin area and other areas into 4 modeling areas. Here, 4 is for illustration purposes only. The actual use can be selected according to the actual scenario.

[0152] The local coal pile point cloud data P1, P2, P3, P4 of the four modeling areas are established, and the three-dimensional static model of the coal pile {P1, P2, P3, P4} is reconstructed by registration;

[0153] A dynamic association model is trained through P1, P2, P3, and P4;

[0154] When the coal pile shape changes in the second modeling area, the new local coal pile point cloud data P of the second modeling area is re-acquired. 2new ;

[0155] P 2new Input into the dynamic association model, which can be obtained by P 1o =BP(P 2new ), and get P 1o , and P 3o =BP(P 2new ), and get P 3o , and then by P 4o =BP(P 3o ), and get P 4o , according to P 2new The local coal pile point cloud data of the 1st, 3rd and 4th modeling areas are obtained, and the 3D dynamic model of the coal pile is reconstructed by re-registration {P 1o , P 2new , P 3o , P 4o}.

[0156] BP (back propagation) neural network is a concept proposed by scientists led by Rumelhart and McClelland. It is a multi-layer feedforward neural network trained according to the error back propagation algorithm and is one of the most widely used neural network models.

[0157] BP neural network has the ability to classify arbitrarily complex patterns and excellent multi-dimensional function mapping capabilities. It can learn certain rules through its own training and obtain the result closest to the expected output value when the input value is given, that is, determine the mapping relationship between input and output.

[0158] By utilizing the function mapping property of the BP neural network, the present invention applies the BP neural network to the dynamic association construction process, and trains the BP neural network to learn the mapping relationship of the local coal pile point cloud data between adjacent modeling areas.

[0159] like Figure 2 As shown, the present invention provides a cabin modeling system based on three-dimensional positioning and two-dimensional mapping, which is applied to a cabin modeling method based on three-dimensional positioning and two-dimensional mapping. The system includes:

[0160] A region division unit, used to divide a cabin area and the like into a plurality of modeling areas;

[0161] The regional modeling unit is used to perform three-dimensional positioning and two-dimensional mapping of the coal pile in each modeling area by using the laser radar and industrial camera mounted on the mobile platform, so as to obtain the local coal pile point cloud data of each modeling area;

[0162] The regional combination unit is used to register and reconstruct the local coal pile point cloud data of each modeling area to obtain a three-dimensional static model of the coal pile in the cabin;

[0163] A dynamic association unit is used to artificially adjust the morphology of the three-dimensional static model of the coal pile in the ship cabin to obtain multiple groups of local coal pile adjustment point cloud data in each modeling area;

[0164] and for using a neural network to adjust point cloud data based on multiple groups of local coal piles in each modeling area, identifying the correlation of the local coal pile point cloud data in each modeling area, and obtaining a dynamic correlation model of the local coal pile point cloud data in each modeling area;

[0165] The dynamic modeling unit is used to combine the three-dimensional static model of the coal pile in the ship's hold with the dynamic association model to obtain a three-dimensional dynamic model of the coal pile in the ship's hold for dynamic perception of the coal pile in the ship's hold.

[0166] The present invention utilizes a laser radar and an industrial camera mounted on a mobile platform to perform three-dimensional positioning and two-dimensional mapping of the coal pile in the modeling area. On the three-dimensional coordinates of the coal pile obtained by the laser scanning modeling method, the color and texture of the modeling object are obtained through image acquisition. At the same time, a dynamic association model that characterizes the correlation of local coal pile point cloud data in each modeling area is constructed by using a neural network. The dynamic model is combined with the three-dimensional static model of the coal pile in the cabin to obtain a three-dimensional dynamic model of the coal pile in the cabin. When the coal pile changes during the cabin cleaning operation, the three-dimensional model of the coal pile can adapt to the dynamic changes of the coal pile and accurately display the current actual shape of the coal pile.

[0167] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A cabin modeling method based on three-dimensional positioning and two-dimensional mapping, characterized in that: The following steps are involved: Divide areas such as the cabin area into multiple modeling areas; In each modeling area, the laser radar and industrial camera mounted on the mobile platform are used to perform three-dimensional positioning and two-dimensional mapping of the coal pile in the modeling area, and obtain the local coal pile point cloud data of each modeling area; The local coal pile point cloud data of each modeling area is registered and reconstructed to obtain a three-dimensional static model of the coal pile in the cabin; The three-dimensional static model of the coal pile in the ship cabin is artificially adjusted to obtain multiple groups of local coal pile adjustment point cloud data in each modeling area; Using a neural network to adjust point cloud data based on multiple groups of local coal piles in each modeling area, the correlation of the local coal pile point cloud data in each modeling area is identified to obtain a dynamic correlation model of the local coal pile point cloud data in each modeling area; The three-dimensional static model of the coal pile in the ship's hold is combined with the dynamic association model to obtain a three-dimensional dynamic model of the coal pile in the ship's hold for dynamic perception of the coal pile in the ship's hold.

2. A method for cabin modeling based on three-dimensional positioning and two-dimensional mapping according to claim 1, characterized in that: The method for three-dimensionally locating a coal pile in a modeling area by using a laser radar includes: In the modeling area, a laser radar is used to perform laser scanning on the coal pile in the modeling area to obtain laser scanning data of the coal pile in the modeling area, including laser ranging values, horizontal angle values, vertical angle values, and horizontal offset angles of the laser channels; The three-dimensional coordinate data of the coal pile in the modeling area are converted by the laser scanning data in the modeling area; The conversion formula of the three-dimensional coordinate data is: In the formula, x kj is the x-axis coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area, and y kj is the y-axis coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area, z kj is the z-axis coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area, r kj is the laser ranging value obtained by the jth channel of the laser radar in the kth modeling area, α kj is the horizontal angle value obtained by the jth channel of the lidar in the kth modeling area, δ kj is the horizontal offset angle value of the jth channel of the laser radar in the kth modeling area, β kj The vertical angle value obtained by the jth channel of the lidar in the kth modeling area; Among them, the three-dimensional coordinate data of the coal pile in the kth modeling area is {(x kj ,y kj , z kj )|j∈[1,n]}, where (x kj ,y kj , z kj ) is the three-dimensional coordinate of the coal pile corresponding to the j-th channel of the laser radar in the k-th modeling area, and n is the number of channels of the laser radar.

3. The method for cabin modeling based on three-dimensional positioning and two-dimensional mapping according to claim 2 is characterized in that: The method for constructing a two-dimensional map of a coal pile in a modeling area by using an industrial camera includes: Adjust the sampling frequency of the industrial camera to be consistent with the sampling frequency of the lidar; An industrial camera is used in the modeling area to collect images of the coal pile in the modeling area to obtain two-dimensional image data of the coal pile in the modeling area.

4. The method for cabin modeling based on three-dimensional positioning and two-dimensional mapping according to claim 3 is characterized in that: The method for constructing local coal pile point cloud data by using the three-dimensional coordinate data and two-dimensional image data of the coal pile in the modeling area includes: The three-dimensional coordinate data and two-dimensional image data of the coal pile in the modeling area are arranged into a coordinate system; The three-dimensional coordinate data and two-dimensional image data of the coal pile in the modeling area after the coordinate system is set are combined according to the corresponding coordinates to obtain the local point cloud data of the coal pile in the modeling area; The formula of the coordinate system 1 is: In the formula, (u kj , v kj ) is (x kj ,y kj , z kj ) is converted to the pixel coordinates in the pixel coordinate system where the two-dimensional image data is located, M is the parameter matrix of the industrial camera, R0 is the rotation matrix between the laser radar coordinate system and the industrial camera coordinate system, t0 is the translation vector between the laser radar coordinate system and the industrial camera coordinate system, (x kj ,y kj , z kj ) is the three-dimensional coordinate of the coal pile corresponding to the j-th channel of the lidar in the k-th modeling area.

5. The method for cabin modeling based on three-dimensional positioning and two-dimensional mapping according to claim 4 is characterized in that: The registration and reconstruction method of the three-dimensional static model of the coal pile in the ship cabin comprises: The point cloud data located in the adjacent edge areas of the local coal pile point cloud data of two adjacent modeling areas are respectively used as the to-be-registered point cloud set A and the target point cloud set B; The ICP registration algorithm is used to perform registration operations on the point cloud set A to be registered and the target point cloud set B to obtain a rotation matrix and translation vector that unifies the coordinates of two adjacent modeling areas; The rotation matrix and translation vector are used to unify the coordinates of two adjacent modeling areas in turn, and the two adjacent modeling areas after the coordinate unification are fused in the adjacent edge area according to the corresponding coordinates to obtain the three-dimensional static model of the coal pile in the cabin {P k |k∈[1,N]}, where P k is the local coal pile point cloud data of the kth modeling area, and N is the total number of modeling areas; The registration equation is: In the formula, F(R AB , t AB ) is the objective function of the ICP registration algorithm, R AB is the rotation matrix between the point cloud set A to be registered and the target point cloud set B, t AB is the translation vector between the point cloud set A to be registered and the target point cloud set B, b i is the i-th point cloud data in the target point cloud set B, a i is the i-th point cloud data in the point cloud set A to be registered, and m is the total number of point cloud data in the point cloud set A to be registered and the target point cloud set B.

6. A method for cabin modeling based on three-dimensional positioning and two-dimensional mapping according to claim 5, characterized in that: The method for obtaining multiple groups of local coal pile adjustment point cloud data in each modeling area includes: The shapes in the three-dimensional static model of the coal pile in the ship's hold are artificially deleted for multiple times to obtain multiple three-dimensional missing models of the coal pile in the ship's hold; Extract the local coal pile adjustment point cloud data of each modeling area from the three-dimensional missing model of each ship hold coal pile; The local coal pile adjustment point cloud data of each modeling area is extracted from the three-dimensional missing model of the coal pile in each cabin, and combined into multiple groups of local coal pile adjustment point cloud data of each modeling area.

7. A method for cabin modeling based on three-dimensional positioning and two-dimensional mapping according to claim 6, characterized in that: The method for constructing the dynamic association model includes: Multiple groups of local coal pile adjustment point cloud data in two adjacent modeling areas are used as data samples for training the mapping relationship between the input and output of the BP neural network; Using a BP neural network to establish a mapping relationship between an input and an output of the BP neural network based on the data samples, and obtaining the dynamic correlation model that characterizes the correlation between the local coal pile point cloud data of two adjacent modeling areas; The dynamic association model is: Where P (k+1)o is the local coal pile point cloud data of the k+1th modeling area output by the dynamic association model, P ko is the local coal pile point cloud data of the kth modeling area output by the dynamic association model, P k is the local coal pile point cloud data of the kth modeling area, P (k+1) is the local coal pile point cloud data of the k+1th modeling area, and BP is a neural network; Among them, the loss function for training the dynamic association model is: Where loss is the loss function for training the dynamic association model. P (k+1)o and P (k+1) The L2 norm between P ko and P k The L2 norm between .

8. The method for cabin modeling based on three-dimensional positioning and two-dimensional mapping according to claim 7 is characterized in that: The method for constructing the three-dimensional dynamic model of the coal pile in the ship's hold comprises: The three-dimensional dynamic model of coal pile in the ship's hold is combined with the dynamic correlation model to obtain the three-dimensional dynamic model of coal pile in the ship's hold; The three-dimensional dynamic model of the coal pile in the ship's hold is: {P 1o ,...,P (h-1)o ,P h ,P (h+1)o ,P (h+2)o ,...,P No }; P lo =BP(P (l+1)o ),l∈[1,(h-1)]; P (r+1)o =BP(P ro ),r∈[h,(N-1)]; Where P h is the point cloud data of the local coal pile that undergoes dynamic changes in the hth modeling area, P 1o , P (h-1)o , P (h+1)o , P (h+2)o and P No The dynamic association model is based on P h Output the local coal pile point cloud data of the 1st, (h-1), (h+1), (h+2) and Nth modeling areas, where N is the total number of modeling areas, P lo and P (l+1)o is the local coal pile point cloud data of the lth and (l+1)th modeling areas located on the left side of the hth modeling area output by the dynamic association model, P ro and P (r+1)o It is the local coal pile point cloud data of the rth and (r+1)th modeling areas on the right side of the hth modeling area output by the dynamic association model, and BP is a neural network.

9. The method for cabin modeling based on three-dimensional positioning and two-dimensional mapping according to claim 8, characterized in that: The point cloud data of the dynamically changing local coal pile is obtained by re-positioning the coal pile in the modeling area in three dimensions and re-mapping it in two dimensions using the laser radar and industrial camera mounted on a mobile platform.

10. A cabin modeling system based on three-dimensional positioning and two-dimensional mapping, characterized in that: A cabin modeling method based on three-dimensional positioning and two-dimensional mapping as described in any one of claims 1 to 9, the system comprising: A region division unit, used to divide a cabin area and the like into a plurality of modeling areas; The regional modeling unit is used to perform three-dimensional positioning and two-dimensional mapping of the coal pile in each modeling area by using the laser radar and industrial camera mounted on the mobile platform, so as to obtain the local coal pile point cloud data of each modeling area; The regional combination unit is used to register and reconstruct the local coal pile point cloud data of each modeling area to obtain a three-dimensional static model of the coal pile in the cabin; A dynamic association unit is used to artificially adjust the morphology of the three-dimensional static model of the coal pile in the ship cabin to obtain multiple groups of local coal pile adjustment point cloud data in each modeling area; and for using a neural network to adjust point cloud data based on multiple groups of local coal piles in each modeling area, identifying the correlation of the local coal pile point cloud data in each modeling area, and obtaining a dynamic correlation model of the local coal pile point cloud data in each modeling area; The dynamic modeling unit is used to combine the three-dimensional static model of the coal pile in the ship's hold with the dynamic association model to obtain a three-dimensional dynamic model of the coal pile in the ship's hold for dynamic perception of the coal pile in the ship's hold.

Images