A stockyard point cloud efficient access method and a point cloud management system

Abstract

The application provides a stockyard point cloud efficient access method and a point cloud management system. The stockyard point cloud efficient access method comprises the following steps: S1, stockyard partitioning: according to the actual situation of the stockyard, the stockyard is partitioned with a track as the partition basis; S2, point cloud storage structure design: according to the characteristics of the stockyard, combining the point cloud data characteristics, using the characteristics of the efficient access memory database redis, the point cloud data is accessed one by one according to the node structure; S3, point cloud read-write scheme: according to the discrete characteristics of the point cloud data, combining the point cloud storage structure design scheme, using the redis access NOSQL characteristics, a two-level cache scheme is used to realize access. The application is based on the efficient point cloud data processing, a multi-level processing design scheme, a unique cache technology and a special data structure are used to solve the problems of large instantaneous amount of point cloud data writing and reading, a point cloud full life cycle management system is designed based on the discrete characteristics of the point cloud, and flexible calling and system smooth expansion are realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method for efficient storage and retrieval of point clouds in a material yard and a point cloud management system. Background Technology

[0002] Currently, the point cloud system for thermal power plant material yards suffers from problems such as cumbersome deployment, complex debugging, long debugging cycles, and difficulty in verifying its accuracy against actual conditions. In addition, the following main issues exist:

[0003] Problem 1: Instantaneous massive point cloud data reading and writing. In a 300-meter-long strip material yard, 6 scanners need to be set up. During the scanning process, the 6 scanners operate concurrently, with an instantaneous point cloud data volume of 100,000 per second, which places high demands on the real-time processing of point clouds.

[0004] Question 2: Limited application of point cloud data platform management. Currently, most manufacturers are still at the stage of acquiring and displaying point cloud data from material yards, and have not yet addressed the management of point clouds. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a method for efficient storage and retrieval of point cloud data in a material yard and a point cloud management system, which addresses the above-mentioned shortcomings of existing point cloud systems in terms of difficulty in processing massive amounts of data. Based on a solution for instantaneous high-concurrency point cloud data calculation and processing, it can realize efficient reading and writing of point cloud data, flexible real-time access, meet the requirements of instantaneous high-concurrency data processing, and enable the system to have smooth expansion capabilities.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for efficient storage and retrieval of point clouds in a material yard includes the following steps:

[0008] S1. Material Yard Zoning: Based on the actual situation of the material yard, the material yard is divided according to the track, that is, the track divides the material yard into two parts, namely the first logical zone and the second logical zone.

[0009] S2. Point Cloud Storage Structure Design: Based on the characteristics of the material yard and the features of point cloud data, and utilizing the high-efficiency storage and retrieval features of the in-memory database Redis, the point cloud data is stored and retrieved one by one according to the node structure. The basic principle is that the root node is the material yard ID, the second node is the partition ID, and the third node is the point cloud data (i.e., the key value). Its structure is: x_y_z data are concatenated with underscores, and the z value of the point cloud is the data value (i.e., the value value).

[0010] S3. Point Cloud Read / Write Solution: Because point cloud read / write operations require peak access times exceeding 500KB QPS, and considering the discrete nature of point cloud data, a two-level caching scheme is employed, leveraging Redis's NoSQL access capabilities. The specific steps are as follows:

[0011] S31: Data writing process:

[0012] i) All data is first cached in memory in batches, and the data is marked as hot data;

[0013] ii) If the data in the memory cache does not exceed 10MB, then write the data to Redis in batches;

[0014] iii) Peak shaving and valley smoothing: If the data cached in memory exceeds 10MB, it is first written to a temporary file; at the same time, the data write QPS is monitored in real time. If the QPS exceeds 100K, the data is first written to a local file and cached locally; if the QPS is less than 100K, the file data is periodically pulled into memory, marked as hot data, and stored according to the logic of step ii).

[0015] S32: Data Reading Process

[0016] i) Prioritize reading hot data in memory;

[0017] ii) Batch fetch Redis data. The fetching rule is that x_y_z form a unique key. With the X-axis as the axis, the minimum unit is accumulated sequentially along the Y-axis to obtain the X and Y values. All key-value values ​​(x_y_z) are batched and then parsed one by one according to the data storage structure.

[0018] iii) After the fetch is complete, clear all partition-related hot data, temporary cache files, and Redis data.

[0019] According to the above scheme, in step S1, the first logical partition and the second logical partition are further divided into several business partitions based on the user's actual usage needs.

[0020] This invention also designs a material yard point cloud management system suitable for the above-mentioned efficient point cloud storage and retrieval method. According to logic, it mainly includes a six-layer architecture: front-end device layer, data acquisition layer, data layer, computing layer, application layer and client (presentation layer).

[0021] The front-end device layer mainly includes location-type devices, control-type devices, and scanning-type devices;

[0022] The data acquisition layer is connected to the front-end device layer and the data layer. The data acquisition layer adopts an Internet of Things architecture and uses the above-mentioned efficient point cloud storage method to realize point cloud storage, while providing basic data for the data layer.

[0023] The data layer stores the data collected by the data acquisition layer to form a point cloud basic data service, and at the same time provides core retrieval capabilities for the upper computing layer;

[0024] The computing layer is based on the data layer and calculates point cloud features according to the business requirements of the application layer to form business-related point cloud object modeling (such as merging adjacent similar features of the entire material yard, removing some features of the material yard, and point cloud prediction and compensation, etc.), as well as point cloud classification and labeling, and establishing a point cloud retrieval index sequence (realizing point cloud classification and point cloud file index management).

[0025] The application layer is mainly responsible for service-oriented management and provides business data transmission to the presentation layer. The application layer includes a point cloud management service module, a point cloud analysis service module, and a point cloud application service module.

[0026] The client mainly includes a 3D point cloud display module, a report export module, a historical record query module, and a device operation and maintenance module.

[0027] According to the above scheme, the location-related devices include GPS, BeiDou, inclinometers, and other positioning devices. GPS and BeiDou are used to provide precise positioning information for control devices (pan-tilt units) and scanning devices (scanners) to ensure the accuracy and precision when point clouds from multiple scanners intersect. Inclinometers are used to measure the horizontal angles of each device during installation and use. The location-related devices are the basic equipment for real-time precise calculation of point clouds. The core device of the control devices is the pan-tilt unit controller. The scanning devices mainly include laser scanners.

[0028] This invention also provides a method for transmitting business data in the aforementioned point cloud management system for material yards. In the application layer, business data transmission adopts an MQTT+RPC processing scheme. Asynchronous MQTT is used for data requests and responses at the application layer, and RPC is used between the point cloud management service module and the point cloud analysis service module. The main business data transmission process includes:

[0029] 1) Data Request

[0030] In principle, the client actively sends data requests;

[0031] 2) Data forwarding

[0032] The MQTT protocol is used as a message middleware to forward data to the point cloud management service module;

[0033] 3) Business Processing

[0034] After receiving a business request, the point cloud management service module processes the business request and outputs the specified request data.

[0035] 4) Data return

[0036] After receiving a message, the MQTT service pushes the data to the subscribers.

[0037] 5) Data retrieval

[0038] After subscribing to a specified topic, the client pulls point cloud data in real time.

[0039] 6) Point cloud data request

[0040] The client requests the specified point cloud data and pulls the point cloud file using HTTP. After receiving the point cloud file, the client sends a point cloud data request to the point cloud management service module.

[0041] 7) Scheduled Task Management

[0042] The point cloud management service module configures scheduled tasks based on the point cloud application service module. The main functions of the scheduled tasks include: scheduled task distribution, scheduled check of expired records, and management of the point cloud computing service status.

[0043] 8) Task Issuance

[0044] The point cloud management service module uses RPC to encrypt and send tasks to the point cloud analysis service module.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] 1. Based on the instantaneous high-concurrency point cloud data computing and processing scheme, this invention provides an efficient point cloud storage and retrieval method. It adopts a unique caching technology and designs a proprietary data structure, which solves the problems of large instantaneous quantity, strong correlation, and difficulty in processing massive amounts of point cloud data in the field of point cloud processing.

[0047] 2. Based on the discrete nature of point clouds, a complete point cloud lifecycle management system was designed. It innovatively draws on the core concept of big data MapReduce, enabling flexible and real-time access to point cloud data, while also allowing the system to scale smoothly. Attached Figure Description

[0048] Figure 1 This is a schematic diagram illustrating how the logical partitioning of the material yard is further divided into several business partitions according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the point cloud storage structure design according to an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram illustrating the data writing process of point cloud data according to an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram illustrating the process of batch retrieving Redis data in an embodiment of the present invention.

[0052] Figure 5 This is a schematic diagram of the six-layer logical architecture of the point cloud management system according to an embodiment of the present invention;

[0053] Figure 6 This is a business data flow diagram of the point cloud management system according to an embodiment of the present invention. Detailed Implementation

[0054] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0055] This invention provides an efficient method for storing and retrieving point clouds in a material yard, comprising the following steps:

[0056] S1. Material Yard Zoning: Based on the actual situation of the material yard, the yard is divided according to the tracks, that is, the tracks divide the material yard into two parts, namely logical zone 01 and logical zone 02; in addition, according to the actual usage needs of users, logical zone 01 and logical zone 02 are further divided into several business zones, such as... Figure 1 As shown ( Figure 1 Logical partition 02 is further divided into partition 03 and partition 04.

[0057] S2. Point Cloud Storage Structure Design: Based on the characteristics of the material yard and the features of point cloud data, and utilizing the high-efficiency in-memory database Redis, the point cloud data is stored and retrieved node by node according to the node structure. The basic principle is that the root node is the material yard ID, the second node is the partition ID, and the third node is the point cloud data (i.e., the key value). Its structure is as follows: x_y_z data are concatenated with underscores, and the z-value of the point cloud is the data value (i.e., the value). Specifically, as shown... Figure 2 As shown;

[0058] S3. Point Cloud Read / Write Solution: Because point cloud read / write operations require peak access times exceeding 500KB QPS, and considering the discrete nature of point cloud data, a two-level caching scheme is adopted, leveraging Redis's NoSQL access capabilities. The specific steps are as follows:

[0059] S31: Data writing process:

[0060] i) All data is first cached in memory in batches, and the data is marked as hot data;

[0061] ii) If the data in the memory cache does not exceed 10MB, then write the data to Redis in batches;

[0062] iii) Peak shaving and valley smoothing: If the data cached in memory exceeds 10MB, it is first written to a temporary file; simultaneously, the data write QPS is monitored in real time. If the QPS exceeds 100K, the data is first written to a local file and cached locally; if the QPS is less than 100K, the file data is periodically pulled into memory, marked as hot data, and stored according to the logic of step ii). A detailed flowchart is shown below. Figure 3 As shown;

[0063] S32: Data Reading Process

[0064] i) Prioritize reading hot data in memory;

[0065] ii) Batch fetch Redis data, using the rule that x, y, and z form a unique key. Using the X-axis as the axis, increment the Y-axis by the smallest unit to obtain the X and Y values. Then, batch-fetch all corresponding key-value pairs (x, y, z), and parse them one by one according to the data storage structure (see diagram). Figure 4 (as shown);

[0066] iii) After the fetch is complete, clear all partition-related hot data, temporary cache files, and Redis data.

[0067] The above design enables efficient reading and writing of point clouds. Verified in a real-world project, it can handle instantaneous high-concurrency data processing. This design innovatively solves the problem of handling massive amounts of data in the field of point cloud processing.

[0068] In conjunction with the above-mentioned efficient point cloud storage and retrieval methods, this invention also designs a point cloud management system, which innovatively draws on the core concept of big data MapReduce to solve the problems of large number of point clouds and strong correlation.

[0069] The basic principles of a point cloud management system include the following:

[0070] 1) Divide the material yard into logical zones of 1 meter each, based on length;

[0071] 2) The minimum logical point cloud size is 10cm*10cm;

[0072] 3) A partition is formed by concatenating multiple logical partitions;

[0073] 4) Rules need to be specified when partitioning; the partition size must be a multiple of 1MB.

[0074] 5) The smallest unit of update is the unit partition, which is 1M * width of data.

[0075] like Figure 5 As shown, the point cloud management system of this invention is mainly divided into six layers according to logic:

[0076] (1) Front-end device layer

[0077] The front-end device layer mainly includes position devices, control devices, and scanning devices:

[0078] Positioning devices include GPS, BeiDou, inclinometers, and other positioning equipment. GPS and BeiDou are used to provide precise positioning information for control equipment (pan-tilt units) and scanning equipment (scanners) to ensure the accuracy and precision of point cloud intersections when multiple scanners are in use. Inclinometers are used to measure the horizontal angles of each device during installation and use. Positioning devices are the basic equipment for real-time accurate point cloud calculation.

[0079] The core component of control equipment is the pan-tilt controller.

[0080] Scanning equipment mainly includes laser scanners.

[0081] (2) Data Acquisition Layer

[0082] Its main responsibility is to interface with the front-end device layer, collect point cloud data in real time, and perform preliminary processing. It adopts an IoT architecture, collects data based on the standard TCP / IP protocol, and uses the aforementioned efficient storage methods to achieve point cloud storage, providing basic data for the data layer.

[0083] (3) Data layer

[0084] By storing the collected data, a point cloud basic data service is formed, while providing core retrieval capabilities for the upper computing layer.

[0085] (4) Computation layer

[0086] Based on the data layer, point cloud features are calculated according to the business requirements of the application layer to form business-related point cloud object modeling (such as merging adjacent similar features of the entire material yard, removing some features of the material yard, and point cloud prediction and compensation, etc.), as well as point cloud classification and annotation, and establishing a point cloud retrieval index sequence (realizing point cloud classification and point cloud file index management).

[0087] (5) Application Layer

[0088] It is primarily responsible for service-oriented management, including point cloud management service module, point cloud analysis service module, and point cloud application service module. The application layer is used to provide business data transmission to the presentation layer.

[0089] (6) Client-side (presentation layer)

[0090] The main functions include a 3D point cloud display module, a report export module, a historical record query module, and an equipment operation and maintenance module.

[0091] The main business data flow of this invention is as follows: Figure 6 As shown:

[0092] The entire system uses an MQTT+RPC+HTTP message passing scheme. Asynchronous MQTT is used for data requests and responses at the application layer, while RPC is used between the point cloud management service module and the point cloud analysis service module. The main business data transmission flow includes:

[0093] 1) Data Request

[0094] In principle, the client actively sends the request.

[0095] 2) Data forwarding

[0096] The MQTT protocol is used as a message middleware to forward data to the point cloud management service module.

[0097] 3) Business Processing

[0098] After receiving a business request, the point cloud management service module processes the request and outputs the specified request data.

[0099] 4) Data return

[0100] After receiving a message, the MQTT service pushes the data to the subscribers.

[0101] 5) Data retrieval

[0102] After subscribing to a specified topic, the client pulls point cloud data in real time.

[0103] 6) Point cloud data request

[0104] The client requests the specified point cloud data and pulls the point cloud file using HTTP. After receiving the point cloud file, the client sends a point cloud data request to the point cloud management service module.

[0105] 7) Scheduled Task Management

[0106] The point cloud management service module configures scheduled tasks based on the point cloud application service module. The main functions of the scheduled tasks include: issuing tasks on a scheduled basis, checking expired records on a scheduled basis, and managing the status of point cloud computing services.

[0107] 8) Task Issuance

[0108] The point cloud management service module uses RPC to encrypt and send tasks to the point cloud analysis service module.

[0109] The above embodiments are merely examples for clearly illustrating the present invention and are not intended to limit the implementation of the present invention. Obvious variations or modifications derived from the spirit of the present invention will be apparent to those skilled in the art and are still within the scope of protection of the present invention.

Claims

1. A method for efficient storage and retrieval of point clouds in a material yard, characterized in that, Includes the following steps: S1. Material Yard Zoning: Based on the actual situation of the material yard, the material yard is divided according to the track, that is, the track divides the material yard into two parts, namely the first logical zone and the second logical zone. S2. Point cloud storage structure design: Based on the characteristics of the material yard and the characteristics of point cloud data, and utilizing the characteristics of the in-memory database Redis, the point cloud data is stored and retrieved one by one according to the node structure. The basic principle is that the root node is the material yard ID, the second node is the partition ID, and the third node is the point cloud data. Its structure is: x_y_z data are concatenated with underscores, and the z value of the point cloud is the data value. S3. Point Cloud Read / Write Solution: Because point cloud read / write operations require peak access times exceeding 500KB QPS, and considering the discrete nature of point cloud data, a two-level caching scheme is employed, leveraging Redis's NoSQL access capabilities. The specific steps are as follows: S31: Data writing process: i) All data is first cached in batches in memory, and the data is marked as hot data; ii) If the data in the memory cache does not exceed 10MB, then write the data to Redis in batches; iii) Peak shaving and valley smoothing: If the data cached in memory exceeds 10MB, it is first written to a temporary file; at the same time, the data write QPS is monitored in real time. If the QPS exceeds 100K, the data is first written to a local file and cached locally; if the QPS is less than 100K, the file data is periodically pulled into memory, marked as hot data, and stored according to the logic of step ii). S32: Data Reading Process i) Prioritize reading hot data in memory; ii) Batch pull Redis data. The pull rule is that x_y_z form a unique key. With the X-axis as the axis, the minimum unit is accumulated sequentially along the Y-axis to obtain the X and Y values. All key-value values ​​are batched and then parsed one by one according to the data storage structure. iii) After the fetch is complete, clear all partition-related hot data, temporary cache files, and Redis data.

2. The efficient storage and retrieval method for point clouds in a material yard according to claim 1, characterized in that, In step S1, the first logical partition and the second logical partition are further divided into several business partitions according to the user's actual usage needs.

3. A material yard point cloud management system applicable to the efficient storage and retrieval method for material yard point clouds according to any one of claims 1 to 2, characterized in that, The architecture mainly consists of six layers: front-end device layer, data acquisition layer, data layer, computing layer, application layer, and client. The front-end device layer mainly includes location-type devices, control-type devices, and scanning-type devices; The data acquisition layer is connected to the front-end device layer and the data layer. The data acquisition layer adopts an Internet of Things architecture and uses the above-mentioned efficient point cloud storage method to realize point cloud storage, while providing basic data for the data layer. The data layer stores the data collected by the data acquisition layer and performs point cloud feature recognition and calculation, thereby realizing point cloud structured processing and point cloud classification feature storage, forming basic point cloud data services, and providing core retrieval capabilities for the upper computing layer. The computing layer is based on the data layer and calculates point cloud features according to the business requirements of the application layer to form business-related point cloud object modeling, point cloud classification and annotation, and establish a point cloud retrieval index sequence. The application layer is mainly responsible for service-oriented management and provides business data transmission to the client. The application layer includes a point cloud management service module, a point cloud analysis service module, and a point cloud application service module. The client mainly includes a 3D point cloud display module, a report export module, a historical record query module, and a device operation and maintenance module.

4. The material yard point cloud management system according to claim 3, characterized in that, The location-related devices include GPS, BeiDou, and inclinometers. GPS and BeiDou provide precise positioning information for control and scanning devices to ensure accuracy and precision when point clouds from multiple scanners intersect. Inclinometers measure the horizontal angles of each device during installation and use. These location-related devices are fundamental for real-time precise point cloud calculations. The core device of the control devices is a pan-tilt controller. The scanning devices mainly include laser scanners.

5. A method for transmitting business data in a material yard point cloud management system as described in any one of claims 3 to 4, characterized in that, In the application layer, business data transmission adopts an MQTT+RPC processing scheme. Asynchronous MQTT is used for data requests and responses at the application layer, while RPC is used between the point cloud management service module and the point cloud analysis service module. The main business data transmission process includes: 1) Data Request The client actively sends a data request; 2) Data forwarding The MQTT protocol is used as a message middleware to forward data to the point cloud management service module; 3) Business Processing After receiving a business request, the point cloud management service module processes the business request and outputs the specified request data. 4) Data return After receiving a message, the MQTT service pushes the data to the subscribers; 5) Data retrieval After subscribing to a specified topic, the client pulls point cloud data in real time; 6) Point cloud data request The client requests the specified point cloud data and pulls the point cloud file using HTTP. After receiving the point cloud file, the client sends a point cloud data request to the point cloud management service module. 7) Scheduled Task Management The point cloud management service module configures scheduled tasks based on the point cloud application service module. The main functions of the scheduled tasks include: scheduled task distribution, scheduled check of expired records, and management of the point cloud computing service status. 8) Task Issuance The point cloud management service module uses RPC to encrypt and send tasks to the point cloud analysis service module.

Images