Method for efficient utilization of point cloud data and its application in overhead crane operation program

Abstract

The present invention relates to a method for efficiently utilizing point cloud data and its application in an overhead crane operation program. Valid point cloud data is converted into a GBMPCD file for storage. When in use, the obtained GBMPCD file is parsed into valid point cloud data and then utilized. The file header of the GBMPCD file includes the following fields arranged in sequence: Version, Height, Width, IsOrder, Type, and Config. For discrete goods, goods identification is performed based on the material yard image, and each piece of goods is regarded as a point. For continuous materials, the material yard is divided into several continuously distributed small rectangular areas, and each small rectangular area is regarded as a point. Based on the constructed new point cloud data file format, the present invention realizes the efficient storage and utilization of point cloud data in overhead crane automation and other suitable fields.

Description

Technical Field

[0001] The present invention relates to a method for efficiently utilizing and storing point cloud data and its application in an overhead crane operation program, belonging to the technical field of information processing and intelligent control. Background Art

[0002] Software design for automated overhead cranes and other applications requires extensive processing of point cloud data acquired from scanning devices. This data is then stored and retrieved and recalculated at appropriate times. This leads to duplication of effort and code, resulting in low operational efficiency and hindering both maintenance and scalability.

[0003] Existing technologies allow point cloud data to be stored and processed using certain data file formats, enabling sharing and multiple reuse of point cloud data. Existing file formats include PLY, PCD, STL, OBJ, and X3D. A typical data format is the PCD format, adopted by the open source PCL (PointCloud Library). This file format offers advantages such as the ability to store point clouds, support for multiple data structures, efficient read and write performance, and flexible dimension settings. Each file format has its own unique characteristics and is effectively applied in various adaptive scenarios. However, each file format also has limitations and cannot meet all point cloud workloads. For example, in overhead crane automation software, point cloud data does not require certain redundant operations and requires customization for different scanner devices. The PCD format requires specification of each dimension, along with its data type and size. Furthermore, it lacks some properties necessary for overhead crane automation program design. Therefore, it is necessary to develop more suitable file types tailored to the application requirements of overhead crane automation programs. Summary of the Invention

[0004] The purpose of the present invention is to achieve efficient storage and utilization of point cloud data in overhead crane automation and other suitable fields based on the constructed new point cloud data file format.

[0005] The technical solution of the present invention is: a method for efficiently utilizing point cloud data, collecting / acquiring original point cloud data, obtaining valid (useful) point cloud data based on the original point cloud data, converting the valid point cloud data into (recording it as) a GBMPCD file for storage, and when the point cloud data needs to be used, obtaining the stored GBMPCD file, parsing (restoring) the obtained GBMPCD file into valid point cloud data and then utilizing it, wherein the file header of the GBMPCD file includes the following fields arranged in sequence: Version, Height, Width, IsOrder, Type and Config, wherein Version indicates the current file version; Height indicates the height of the point cloud; Width indicates the width of the point cloud; IsOrder indicates whether it is an ordered point cloud; Type indicates the adaptive data type of the stored point; and Config is an additional configuration.

[0006] Furthermore, for an ordered point cloud, the Height in the file header of the GBMPCD file is the number of rows, the Width is the number of columns, and IsOrder is Yes.

[0007] Furthermore, for unordered point clouds, kd-Tree is used to construct the topological relationship between points, and all node data of the kd-Tree are recorded in the GBMPCD file. The node data includes the parent node number of the node, whether it is a leaf node, and the point data of the leaf node (if it is a leaf node) (the content is the same as the point data of the ordered point cloud).

[0008] For the GBMPCD file that records all node data of the kd-Tree, the Height in the file header is 1, Width is the number of nodes, and IsOrder is No.

[0009] Furthermore, the additional information involved in Config is written in json format.

[0010] The application of any method for efficiently utilizing point cloud data disclosed in the present invention in an overhead crane operation program.

[0011] Furthermore, the original point cloud data is the image data of the material yard involved in the operation of the overhead crane, and the effective point cloud data is the data of the points (or program control points) that the overhead crane operation program in the material yard focuses on.

[0012] Preferably, the stockyard image data is preprocessed, and the preprocessing includes filtering.

[0013] Preferably, the preprocessing includes multi-image fusion.

[0014] Furthermore, for continuous materials (for example, powder materials, granular materials), the material field can be divided into several continuously distributed small rectangles (for example, squares, or rectangular areas. The division of small rectangular areas can be set according to the characteristics of the overhead crane / grab bucket or actual operation requirements. The divided small rectangular area (the location of the small rectangular area) is regarded as a point contained in the valid point cloud data. Accordingly, the point cloud is an ordered point cloud. The Type in the GBMPCD file header uses PointXYZ, where XY is the X coordinate (the number of columns can be used) and Y coordinate (the number of rows can be used) of the center or a corner of the small rectangular area, and Z is the material height of the small rectangular area (it can be the average height of the top surface of the material in the small rectangular area, or the highest point height, etc., which can be set according to the overhead crane operation control requirements).

[0015] Furthermore, for discrete goods (non-continuous materials, such as cargo boxes), goods identification is performed based on the material yard image, and each identified piece of goods (the location of the goods) is regarded as a point contained in the valid point cloud data. Accordingly, the point cloud is an unordered point cloud, and the Type in the GBMPCD file header uses PointXY or PointXYZ, where XY is the X and Y coordinates of the center of the goods, respectively, and Z is the height coordinate of the goods, which can be expressed by / converted into the number of cargo layers.

[0016] Furthermore, after each cargo pickup or placement by the overhead crane, the GBMPCD file for the relevant material yard is updated. The GBMPCD file update is based on the original point cloud data collected after the action is completed. Generally, no GBMPCD file update (including the data collection and processing required for the update) should be performed between two updates implemented in accordance with the above requirements.

[0017] The beneficial effects of the present invention are as follows: in addition to the necessary information such as the width and height of the point cloud, the file header also includes the IsOrder field, which can be used to directly determine whether the point cloud stored in the file is ordered, so as to adopt different rules for parsing; the Config field is a field for storing a json format string, which is used for customized format information. For example, the angular resolution of the scanner or the scanning starting angle that the program needs to obtain in certain scenarios can be saved through the Config field. Since different scanner devices often have different parameters, even the same device may have different requirements in different scenarios. By using Conf The ig field can achieve the purpose of customization; the file format information can be customized with the help of Config, and can only be customized with the help of the Config field, which can give developers a certain design space and prevent other information from being abused; the type system is limited. Unlike the existing general point cloud files, the files of the present invention only support the point cloud types involved / commonly used in the automated overhead crane software, which greatly simplifies the complexity of the point cloud file and saves redundant space occupied by irrelevant information; the kd-Tree of the disordered point cloud can be directly stored when the point cloud is disordered, and can be directly restored when needed later, which simplifies the calculation and saves computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the present invention;

[0019] Figure 2 It is an example of logical point position relationship / point distribution of an ordered point cloud;

[0020] Figure 3 This is an example of a point cloud / point distribution of an unordered point cloud;

[0021] Figure 4 yes Figure 3 kd-Tree illustration of an example point cloud. DETAILED DESCRIPTION

[0022] See also Figure 1-Figure 4 In response to the application requirements of overhead crane automation programs, the present invention constructs a new file format (or file type). This new file format can be called GBMPCD, which can better adapt to the program (operation / working program) design of automated overhead cranes.

[0023] The existing PCD file header includes the following fields arranged in sequence: VERSION, FIELDS, SIZE, TYPE, COUNT, WIDTH, HEIGHT, VIEWPOINT, POINTS, and DATA. These attributes can better adapt to general or common situations. However, for point cloud data involved in automated overhead crane operation programs or other similar applications (for example, material yard point clouds involved in overhead crane material transportation), this point cloud data file type cannot yet meet the usage requirements well.

[0024] In the programming of automated overhead cranes, a large amount of point cloud data obtained from scanning devices needs to be processed, stored, and retrieved and recalculated at the appropriate time. Without point cloud data in a suitable format, there will be the dilemma of repeated work and code duplication, resulting in low operating efficiency, and is not conducive to maintenance and expansion.

[0025] The present invention constructs a new point cloud data file format by improving the file header. The file header determines and declares relevant characteristics / attributes of the point cloud data stored in the file through a number of fields arranged in sequence.

[0026] The file header is encoded in ASCII code, and each file header field and ASCII point data specified in the file are separated by a new line (\n).

[0027] In version 1.0 (basic version) of the GBMPCD file proposed in the present invention, the file header contains 6 fields (6 fields) and their order is: Version, Height, Width, IsOrder, Type, and Config, where Version indicates the version of the current file; Height indicates the height of the point cloud; Width indicates the width of the point cloud; IsOrder indicates whether the point cloud is an ordered point cloud; Type is used to indicate what data type is used to store the points in this file; and the Config field is used to write some additional information, which can be determined according to the needs of the program / application.

[0028] The following is an example of a GBMPCD file:

[0029] Version 1.0 / / File version

[0030] Height 100 / / Point cloud height

[0031] Width 100 / / Point cloud width

[0032] IsOrder yes / / Is it an ordered point cloud?

[0033] Type PointXY / / Adaptive data type for storing points

[0034] Config / / Additional configuration is written in json format 32.3 21.2 123.5 34.5 23.8 94.0 ...

[0039] 939.2 1004.4 / / From the Config field to this point belongs to the specific data range

[0040] The function for reading a file is defined as ReadPcdFrom(), the function for writing a file is defined as WritePcdTo(), and the data type used to store the point cloud is defined as PointCloudData.

[0041] Figure 1 The dotted line in the figure shows the data transmission / processing process under the existing technology, in which the original point cloud data is obtained from the scanner device through the network, and is stored in the memory after processing for use by the point cloud data processing program.

[0042] Based on the above processing flow, you can consider storing the data in the memory directly to the disk, or you can further process the point cloud data (such as filtering, k-nearest neighbor search, etc.) and / or save it to the disk. You can read the stored point cloud data from the disk and load it into the memory so that the application can read and use the data in the memory.

[0043] Figure 1 The solid line path in the figure represents the control process for the point cloud of the present invention. Instead of storing the initial data in memory, the processed data stored on disk is selected, and the data is restored from the disk in its original state (this process differs for ordered and unordered point clouds). Therefore, the results of a single processing can be reused multiple times, although generally speaking, if the point cloud data changes during this process, the point cloud stored on disk becomes outdated and unsuitable for subsequent use. However, the present invention relates to automated cranes and other similar applications. Since the material field often does not change during automated operation, or in other words, the point cloud data reflecting the material field state only changes when the overhead crane (grab bucket) grabs or releases material, the movement of large machines such as overhead cranes is relatively slow, especially compared to the computing speed of computers. During the several seconds or even longer operation of the overhead crane, the computer does not need to perform the same calculation hundreds or thousands of times, which helps save computing resources. The point cloud is only recalculated after determining that the material field (point cloud data) has changed. Therefore, this process is feasible and efficient during the operation of automated overhead cranes.

[0044] For ordered point clouds, we only need to parse the multi-dimensional data into one-dimensional data. Figure 2 For the point cloud / point distribution shown, the GBMPCD file can be:

[0045] Version 1.0 / / File version

[0046] Height 100 / / Point cloud height

[0047] Width 100 / / Point cloud width

[0048] IsOrder yes / / Ordered point cloud

[0049] Type PointXY / / 2D point data type

[0050] Config / / No additional configuration 1 1 1 2 1 3 twenty one twenty two twenty three 3 1 3 2

[0059] 3 3 / / 9 points in total

[0060] The point cloud height and point cloud width are both counted in points (or, in other words, the number of rows and columns, respectively); IsOrder takes the value yes for an ordered point cloud and no for an unordered point cloud; Type can usually be PointXY or PointXYZ.

[0061] Figure 2 The example is an ordered point cloud. The points form an array of multiple rows and columns, or a rectangular row and column distribution. When stored in a file, the points are stored in the order of rows first and then columns. The points in the first row are written to the file in the order of columns (that is, (1, 1) first, then (1, 2) and then (1, 3)), followed by the second row, the third row, and so on until the last row. The final file is shown above.

[0062] For unordered point clouds (see Figure 3 For example, there are several points (A, B, C, D, E, F, G). Appropriate methods can be introduced to maintain the topological relationship between point clouds. A more suitable choice is to build / maintain the topological relationship between point clouds based on the kd-Tree data structure. The point cloud area can be gradually divided by a horizontal or vertical dividing line passing through one or more points so that all points are located on the dividing line. The kd-Tree is established based on the topological relationship reflected by the dividing line (see Figure 4).

[0063] According to the numbering rules, all nodes of the kd-Tree are numbered to determine the coordinates of the parent node and leaf node of each node. The node numbering rules can be numbered sequentially from top to bottom, and the nodes on the same layer can be numbered sequentially from left to right. Figure 4 For example, the node number and related information are as follows:

[0064]

[0065]

[0066] The information in the above table is stored in the GBMPCD file. When the application reads the point cloud data file, it parses / restores it based on this information.

[0067] Figure 4 and Figure 3 An example GBMPCD file might be:

[0068] Version 1.0 / / File version

[0069] Height 1 / / The value of unordered point cloud is 1

[0070] Width 7 / / Point cloud width, i.e. the number of points

[0071] IsOrder no / / Unordered point cloud

[0072] Type PointXY / / 2D point data type

[0073] Config / / No additional configuration 0 0 1 0 1 0 2 0 2 0 3 0

[0080] 3 1Xd Yd

[0081] 4 1Xg Yg

[0082] 4 1Xb Yb

[0083] 5 1Xa Ya

[0084] 5 1Xc Yc

[0085] 6 1Xf Yf

[0086] 6 1Xe Ye / / There are 7 points in total, (Xe, Ye) represents the coordinates of point e

[0087] This is Figure 4 The file after the kd-Tree is stored in the example. Based on the above file, the data can be easily restored to the original kd-Tree, so that the data can be used directly in subsequent programs, instead of starting from the most original point cloud data every time and then repeatedly processing it.

[0088] Restoration process: First read the file. If it is determined to be an unordered point cloud, restore it according to the storage rules of the unordered point cloud when reading the data area. Taking the above file as an example: if the first row of data is 0 0, it means that this point is the root node and not a leaf node. So continue to go deeper and find that the second and third rows of data are both 1 0, which means that they both have the first node as the parent node and are not leaf nodes. So continue to search down and find the leaf node later. Then restore the point coordinates to memory. Until all points are restored, it becomes the original kd-Tree.

[0089] In an unordered point cloud, each node has a number to distinguish different points; except for the root node, each node has a parent node number that can be used to establish the parent-child relationship of all points; each node has a field to indicate whether the current node is a leaf node, which can be used to determine whether the current node needs to establish a data type to store the data of a specific point; all leaf nodes contain specific point data, and this information combined can completely establish all the point cloud data in the program.

[0090] Because the file header specifies Type as PointXY, leaf nodes have more PointXY data than non-leaf nodes. When the program reads data from the file, it writes this extra data into the PointXY data type. When PointXY is defined as shown in the following code, the X and Y values ​​of the corresponding PointXY data type are assigned to the corresponding data in the file when the leaf node is initialized.

[0091]

[0092] In the field of automated driving, working conditions are complex and changeable. For example, the three data 54 100 43 may represent the three-dimensional coordinates of a point, or a two-dimensional coordinate plus a temperature value, color value, depth value, etc., which is not conducive to type inference. This problem is well solved by directly specifying the data type of the received data in the file.

[0093] Unless otherwise specified or when one preferred or optional technical means is a further limitation of another technical means, the preferred and optional technical means disclosed in the present invention can be arbitrarily combined to form several different specific implementation methods.

Claims

1. A method for efficiently utilizing point cloud data, characterized by Collect / acquire original point cloud data, obtain valid point cloud data based on the original point cloud data, convert the valid point cloud data into GBMPCD files for storage, obtain the stored GBMPCD files when point cloud data is needed, parse the obtained GBMPCD files into valid point cloud data and use them. The file header of the GBMPCD file includes the following fields arranged in sequence: Version, Height, Width, IsOrder, Type and Config, where Version indicates the current file version; Height indicates the point cloud height; Width indicates the point cloud width; IsOrder indicates whether it is an ordered point cloud; Type indicates the adapted data type of the stored point; Config is an additional configuration; for ordered point clouds, the file header of the GBMPCD file contains The Height is the number of rows, Width is the number of columns, and IsOrder is Yes; for unordered point clouds, kd-Tree is used to construct the topological relationship between points, and all node data of kd-Tree are recorded in the GBMPCD file. The node data includes the parent node number of the node, whether it is a leaf node, and the point data of the leaf node; the Height in the file header of the GBMPCD file is 1, Width is the number of nodes, and IsOrder is No; in an unordered point cloud, each node has a number to distinguish different points; except for the root node, each node has a parent node number, which can be used to establish the parent-child relationship of all points; each node has a field to indicate whether the current node is a leaf node; all leaf nodes contain specific point data, and this information combined is enough to build all the point cloud data; The restoration process includes: reading the file stored by kd-Tree. If it is determined to be an unordered point cloud, it is restored according to the storage rules of the unordered point cloud when reading the data area. The first row of data in the file stored by kd-Tree is the node data of the root node. Starting from the first row of data, it is judged whether it is a leaf node based on the node data of each row. After the leaf node is found, the coordinates of the leaf node are restored to the memory until all leaf nodes are restored.

2. The method for efficiently utilizing point cloud data according to claim 1, characterized in that The additional information involved in Config is written in json format.

3. Application of the method for efficiently utilizing point cloud data according to any one of claims 1 to 2 in an overhead crane operation program.

4. Application of the method for efficiently utilizing point cloud data according to claim 3 in an overhead crane operation program, characterized in that The original point cloud data is the image data of the material yard involved in the operation of the overhead crane, and the effective point cloud data is the point data that the overhead crane operation program in the material yard pays attention to.

5. The application of the method for efficiently utilizing point cloud data according to claim 4 in an overhead crane operation program is characterized in that For continuous materials, the material yard is divided into several continuously distributed small rectangular areas, and the divided small rectangular areas are regarded as a point contained in the valid point cloud data. Accordingly, the point cloud is an ordered point cloud, and the Type in the GBMPCD file header uses PointXYZ, where XY is the X coordinate and Y coordinate of the center or a corner of the small rectangular area, and Z is the material height of the small rectangular area.

6. Application of the method for efficiently utilizing point cloud data according to claim 4 in an overhead crane operation program, characterized in that For discrete goods, goods identification is performed based on the material yard image, and each identified piece of goods is regarded as a point contained in the valid point cloud data. Accordingly, the point cloud is an unordered point cloud, and the Type in the GBMPCD file header uses PointXY or PointXYZ, where XY is the X and Y coordinates of the center of the goods, respectively, and Z is the height coordinate of the goods.

7. Application of the method for efficiently utilizing point cloud data according to any one of claims 3 to 6 in an overhead crane operation program, characterized in that After each overhead crane completes a cargo grabbing or placing, the GBMPCD file of the material yard involved is updated once. The GBMPCD file update is based on the original point cloud data collected after the action is completed. The GBMPCD file update will not be performed between two updates implemented according to the above requirements.

Images