Robot environment simulation reconstruction method and device based on point cloud, equipment and medium

Abstract

The embodiment of the invention discloses a robot environment simulation reconstruction method and device based on point cloud, equipment and a medium. The method comprises the following steps: acquiring an original point cloud of a surrounding environment through at least one laser radar configured by a robot; preprocessing the original point cloud to obtain a reference environment point cloud; each input reference environment point cloud is recognized through a trained OccupanyNetworks model, reference environment point cloud type information is obtained, and the OccupanyNetworks model is of a multi-layer perceptron structure and comprises a Set Abstraction layer and a feature extraction layer; and loading on a simulation platform according to the point cloud type information of each reference environment to obtain a simulation reconstruction environment of the robot. According to the technical scheme, the environment point cloud type can be rapidly and accurately determined, the simulation environment identical to the actual operation environment of the robot can be rapidly reconstructed in simulation software, the test result can better reflect the performance of the robot in the actual operation scene, test errors caused by environment differences are reduced, and the credibility of the test result is improved.

Description

Technical Field

[0001] The present invention relates to the field of robot simulation technology, and in particular to a point cloud-based robot environment simulation reconstruction method, device, equipment and medium. Background Art

[0002] In the modern warehousing and logistics industry, the performance and reliability of autonomous mobile robot equipment such as AMR and AGV are crucial to ensure the safe and stable operation of logistics robots. However, testing robot performance in a real operating environment often requires a lot of resources, including equipment, labor and time costs. On the other hand, in some extreme scenarios, such as testing the robot's fall prevention function, it is risky to test in real scenarios;

[0003] In contrast, simulation platforms allow robots to be extensively tested and optimized in a virtual environment without occupying actual warehouse or logistics sites, significantly reducing the cost of robot testing. It is crucial to quickly and accurately recreate the same environment as the actual operation scenario in the simulation scenario.

[0004] For those skilled in the art, how to effectively extract environmental information from point cloud data and quickly and accurately reconstruct a simulation environment that is the same as the actual operating environment on a simulation platform is a challenging task. Summary of the invention

[0005] The present invention provides a point cloud-based robot environment simulation and reconstruction method, device, equipment and medium, which utilizes a trained machine model to identify the environmental point cloud, can quickly and accurately determine the type of environmental point cloud, and quickly rebuild a simulation environment that is the same as the actual operating environment of the robot in the simulation software, so that the test results can better reflect the performance of the robot in the actual operating scenario, reduce the errors caused by environmental differences, and improve the credibility of the test results.

[0006] According to one aspect of the present invention, a method for reconstructing a robot environment simulation based on point cloud is provided, comprising:

[0007] Acquire the original point cloud of the surrounding environment through at least one laser radar configured by the robot;

[0008] Preprocess the original point cloud to obtain the reference environment point cloud; use the trained Occupancy Networks model to identify each input reference environment point cloud to obtain the reference environment point cloud type information, where the Occupancy Networks model adopts a multi-layer perceptron structure, including a SetAbstraction layer and a feature extraction layer;

[0009] According to the point cloud type information of each reference environment, the simulation reconstruction environment of the robot is obtained by loading it on the simulation platform.

[0010] Optionally, the original point cloud is preprocessed to obtain a reference environment point cloud, including:

[0011] Based on the time series algorithm, the dynamic obstacle point cloud in the original point cloud is detected and filtered to obtain the static environment point cloud;

[0012] Mapping the static environment point cloud into the first grid map to obtain a static environment point cloud grid;

[0013] Calculate the point cloud density of each static environment point cloud grid, filter the grid with a point cloud density less than a preset threshold, and obtain a reference environment grid, which includes a reference environment point cloud;

[0014] The trained OccupancyNetworks model is used to identify the reference environment point cloud within the reference environment grid and determine the reference environment point cloud type information.

[0015] Optionally, the reference environment point cloud in the reference environment grid is identified using the trained OccupancyNetworks model to determine the reference environment point cloud type information; including:

[0016] Randomly select any reference environment point cloud in the reference environment grid as the center point, save the position information of the center point, and use the sliding window method to slide along the X-axis, Y-axis and Z-axis directions of the world coordinate system according to the preset step length; each time sliding, obtain the point cloud information within the first preset radius as the reference environment point cloud sub-dataset until all reference environment point cloud data are counted; wherein the preset step length is less than the first preset radius value;

[0017] Identify the reference environment point cloud sub-dataset based on the trained OccupancyNetworks model and determine the reference environment point cloud type;

[0018] Different types of reference environment point clouds are converted into mesh format through the Poisson reconstruction algorithm.

[0019] Optionally, different types of reference environment point clouds can be converted to mesh format using the Poisson reconstruction algorithm, including:

[0020] Construct a vector field: Construct a vector field v based on each point and normal vector in the reference environment point cloud;

[0021] Vector Field

[0022] Where v(q) is the vector field at q, pi is the position of the i-th point in the input point cloud, ni is the normal vector of point pi, F is the direction-sensitive kernel function, (q-pi) represents the vector from point pi to point q, d is the dimension of the kernel function, ||q-pi|| represents the Euclidean distance from point q to pi, and N is the total number of points in the point cloud;

[0023] F is the direction-sensitive kernel function, F(r) = max(0,Rr) 3 / R 3 ,R is the support radius, r represents the distance from point q to point pi;

[0024] Solve Poisson's equation: Take the vector field as the source term and construct Poisson's equation ▽ 2 χ=▽·v, determine the approximate value of the indicator function χ, where χ is positive inside the surface, negative outside the surface, and zero or close to zero on the surface; where ▽ 2 represents the Laplace operator, and ▽· represents the divergence operator.

[0025] Isosurface extraction: The Marching Cubes algorithm is used to extract isosurfaces from the solved indicator function χ and store them as the final mesh model.

[0026] Optionally, before using the trained OccupancyNetworks model to identify each input reference environment point cloud, the following steps are included:

[0027] Use simulation software to build a robot simulation environment, including at least walls, pillars, trays, and a robot equipped with a laser radar;

[0028] Acquiring type information of the simulation environment point cloud through simulation software;

[0029] Mapping the simulation environment point cloud to a second grid to obtain a second grid with simulation environment point cloud type information;

[0030] A training data set is constructed based on the simulation environment point cloud and a second grid with type information of the simulation environment point cloud.

[0031] Optionally, after the training data set is obtained according to the simulation environment point cloud and the second grid with the type information of the simulation environment point cloud, the following steps are included:

[0032] Randomly select any point cloud in the training data set, take the point cloud as the center point, use the sliding window method to slide along the X-axis, Y-axis and Z-axis directions of the world coordinate system according to the second preset step size, obtain the point cloud within the second radius as the input point of the OccupancyNetworks model, obtain the type of each grid within the second radius, and perform supervised training through the annotated labels;

[0033] Repeatedly select point clouds for model training and iteratively optimize model parameters until the model output meets the requirements of the loss function.

[0034] Optional, OccupancyNetworks model, including:

[0035] A point cloud input module is used to receive a point cloud of the robot environment, including an original point cloud of the surrounding environment and / or a point cloud of the simulated environment;

[0036] The Set Abstraction layer includes multiple downsampling layers and feature extraction layers, among which:

[0037] The downsampling layer uses the farthest point sampling algorithm to select key points;

[0038] The feature extraction layer includes multiple connection layers and pooling layers, and selects neighborhood points around each key point to extract local features from the feature set output by the Set Abstraction layer through the PointNet++ structure;

[0039] An encoder module, using a global pooling operation, converts the feature map into a global feature vector;

[0040] A multi-layer perceptron (MLP) layer for fusing the query point (x, y, z) coordinates and the global feature vector;

[0041] The output layer outputs the type prediction probability of each query point to form a predicted type probability map in three-dimensional space;

[0042] The loss calculation and optimization module uses the cross entropy loss function to evaluate the difference between the predicted type and the actual type, and adjusts the network parameters through the optimization algorithm.

[0043] According to another aspect of the present invention, a point cloud-based robot environment simulation and reconstruction device is provided, comprising:

[0044] An environmental point cloud acquisition module, used to acquire raw point cloud data of the surrounding environment through at least one laser radar configured by the robot;

[0045] Point cloud processing module, used to pre-process the original point cloud data to obtain the reference environment point cloud;

[0046] The point cloud type prediction module is used to use the trained OccupancyNetworks model to identify each input reference environment point cloud data and obtain the reference environment point cloud type prediction information;

[0047] The environment loading determination module is used to load the simulation platform according to the prediction information of each reference environment point cloud type to obtain the simulation reconstruction environment of the robot.

[0048] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0049] at least one processor; and

[0050] a memory communicatively connected to at least one processor; wherein,

[0051] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the point cloud-based robot environment simulation and reconstruction method described in any embodiment of the present invention.

[0052] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the point cloud-based robot environment simulation and reconstruction method described in any embodiment of the present invention when executed.

[0053] The technical solution of the embodiment of the present invention is to obtain the original point cloud of the surrounding environment through at least one laser radar configured by the robot; pre-process the original point cloud to obtain the reference environment point cloud; use the trained OccupancyNetworks model to identify each input reference environment point cloud to obtain the reference environment point cloud type information, wherein the OccupancyNetworks model adopts a multi-layer perceptron structure, including a Set Abstraction layer and a feature extraction layer; according to the type information of each reference environment point cloud, load it on the simulation platform to obtain the simulated reconstruction environment of the robot. This technical solution is a point cloud-based robot environment simulation and reconstruction method, which uses a trained machine model to identify the environment point cloud, can quickly and accurately determine the type of the environment point cloud, and realize the rapid reconstruction of the simulation environment that is the same as the actual operating environment of the robot in the simulation software, so that the test results can better reflect the performance of the robot in the actual operating scenario, reduce the error caused by environmental differences, and improve the credibility of the test results.

[0054] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 is a flow chart of a point cloud-based robot environment simulation and reconstruction method provided according to Embodiment 1 of the present invention;

[0057] Figure 2 is a flow chart of a method for preprocessing an original point cloud provided in accordance with the first embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of an Occupancy Networks network model architecture provided according to Embodiment 1 of the present invention;

[0059] Figure 4 is a flow chart of a method for converting a point cloud into a mesh format and saving it according to the second embodiment of the present invention;

[0060] Figure 5 is a flow chart of a method for point cloud-based robot environment simulation and reconstruction according to Embodiment 3 of the present invention;

[0061] Figure 6 is a schematic structural diagram of a point cloud-based robot environment simulation and reconstruction device provided according to a fourth embodiment of the present invention;

[0062] Figure 7 It is a structural schematic diagram of an electronic device for implementing a point cloud-based robot environment simulation and reconstruction method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0064] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0065] Embodiment 1

[0066] Figure 1 This is a flowchart of a method for simulating and reconstructing a robot environment based on point cloud provided in the first embodiment of the present invention. This embodiment is applicable to the situation of quickly reconstructing the actual operating environment of the robot. The method can be executed by a simulation environment reconstruction device of the robot. The simulation environment reconstruction device of the robot can be implemented in the form of hardware and / or software. The simulation environment reconstruction device of the robot can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0067] S110, obtaining an original point cloud of the surrounding environment through at least one laser radar configured for the robot.

[0068] In this embodiment, it is necessary to obtain the laser radar point cloud map of the actual operation scene of the robot, reconstruct the simulation environment according to the actual operation environment of the robot, and reduce the deviation of the test results caused by environmental differences. Specifically, the original point cloud of the surrounding environment can be obtained by at least one laser radar configured by the robot. The laser radar can be selected as a single-line or multi-line laser radar according to actual needs. The laser radar can be configured on the top of the robot to collect the environmental point cloud.

[0069] The laser radar emits laser pulses at a certain frequency. The laser pulses are reflected back when they encounter objects in the environment. The laser radar receives the reflected signals to measure the distance and angle information of objects in the environment. This information is the original point cloud data of the robot's surroundings.

[0070] S120, preprocessing the original point cloud of the surrounding environment to obtain a reference environment point cloud.

[0071] In this embodiment, the original point cloud of the surrounding environment collected by the laser radar needs to be preprocessed to obtain the reference environment point cloud. Specifically, the method flow of preprocessing the original point cloud is as follows: Figure 2 As shown, including:

[0072] Based on the time series algorithm, S1211 detects and filters the dynamic obstacle point cloud in the original point cloud to obtain the static environment point cloud.

[0073] The position of static objects in the environment is fixed. In this embodiment, the simulation environment is reconstructed for static objects in the actual operating environment of the robot, such as walls, pillars, trays or ceilings, and the point cloud information of dynamic objects is filtered. Specifically, a laser radar is used to scan the surrounding environment at a fixed frequency, and multiple frames of point cloud data are continuously acquired. The point cloud data of different time frames are aligned to the same coordinate system using the ICP algorithm. The point clouds belonging to the same object are clustered together using the point cloud clustering DBSCAN algorithm, and then the object motion features of each cluster, such as speed or acceleration information, are extracted to determine dynamic obstacles, and the point clouds of dynamic obstacles are removed from the original point clouds, and the point clouds of static obstacles are retained.

[0074] S1212 maps the static environment point cloud into the first grid map to obtain a static environment point cloud grid.

[0075] Specifically, a 3D grid map is initialized. The size of the 3D grid map and the size of each grid unit are determined according to the environmental range. In this embodiment, the size of the grid can be selected as 10cm*10cm*10cm. The three-dimensional coordinates (x, y, z) in the point cloud data are converted to the coordinate system of the 3D grid map. Each point in the static environment point cloud data set is traversed and assigned to the corresponding grid unit in the 3D grid map according to its coordinates.

[0076] S1213 calculates the point cloud density of each static environment point cloud grid, performs point cloud filtering on the grids whose point cloud density is less than a preset threshold, and obtains a reference environment grid, which includes a reference environment point cloud.

[0077] Specifically, each grid cell in the 3D grid image is used to store the feature values ​​of the point cloud data within the corresponding grid space, such as point cloud density, average height, highest / lowest point, etc. Filtering out grids with low point cloud density can reduce the amount of static environment point cloud data. These low-density grids contain less environmental information or noise and have less impact on the overall environmental information.

[0078] S1214 obtains a reference environment grid.

[0079] The above method is used to preprocess the original point cloud of the surrounding environment collected by the robot, retain the static object point cloud and map the static environment point cloud to a 3D grid map to obtain a reference environment grid, which contains the reference environment point cloud.

[0080] S130 uses the trained Occupancy Networks model to identify the reference environment point cloud in the reference environment grid and determine the reference environment point cloud type information.

[0081] In this embodiment, after obtaining the reference environment grid and the reference environment point cloud, any reference environment point cloud in the reference environment grid is randomly selected as the center point, the position information of the center point is recorded and saved, and the sliding window method is used to slide along the X-axis, Y-axis and Z-axis directions of the world coordinate system according to the preset step size; each time a slide is performed, the point cloud information within the first preset radius is obtained as a reference environment point cloud sub-dataset until all reference environment point cloud data are counted; wherein the preset step size is less than the first preset radius value.

[0082] Input each reference environment point cloud into the trained OccupancyNetworks model, use the trained OccupancyNetworks model to identify each input reference environment point cloud, obtain the reference environment point cloud type information, identify the reference environment point cloud sub-dataset according to the trained OccupancyNetworks model, and determine the reference environment point cloud type. According to the location information of each center point in each saved point cloud sub-dataset, combined with the determined reference environment point cloud type, splice to obtain the reference environment point cloud type map.

[0083] The OccupancyNetworks network model first determines the occupancy state of the input environmental point cloud in the 3D grid and extracts features from the grid occupancy information. The grid occupancy features are used to classify the point cloud type, determine the object type to which each grid cell belongs based on the grid occupancy information, determine the point cloud semantic label, and thus identify the point cloud type.

[0084] In this embodiment, the Occupancy Networks model adopts a multi-layer perceptron (MPL) structure, specifically including:

[0085] A point cloud input module is used to receive a point cloud of the robot environment, including an original point cloud of the surrounding environment and / or a point cloud of the simulated environment;

[0086] The Set Abstraction layer includes multiple downsampling layers and feature extraction layers, among which:

[0087] The downsampling layer uses the farthest point sampling algorithm to select key points;

[0088] The feature extraction layer contains multiple connection layers and pooling layers. It selects neighborhood points around each key point and uses the PointNet++ structure to extract multi-scale feature maps from the feature set output by the SetAbstraction layer to extract local features.

[0089] The encoder module uses a global pooling operation to convert the feature map into a global feature vector;

[0090] A multi-layer perceptron (MLP) layer to fuse the query point cloud (x, y, z) coordinates and the global feature vector;

[0091] The output layer outputs the type prediction probability of each query point to form a three-dimensional prediction type probability map;

[0092] The loss calculation and optimization module uses the cross entropy loss function to evaluate the difference between the predicted type and the actual type, and adjusts the network parameters through the optimization algorithm to minimize the loss.

[0093] Specifically, Figure 3 As shown in the figure, the OccupancyNetworks model inputs point cloud data from the outside. The point cloud data can be point cloud data collected by the robot in the actual operating environment, or point cloud of the simulation environment collected in the built simulation environment. The point cloud input module filters the input point cloud for dynamic object point cloud and maps the filtered point cloud to the 3D grid.

[0094] The Set Abstraction layer contains multiple downsampling layers and feature extraction layers, among which:

[0095] The downsampling layer can use the farthest point sampling algorithm to select key points. In specific implementation, the downsampling layer can use the farthest point sampling algorithm to select key points to ensure that the selected key points are evenly distributed in space and better preserve the geometric structure of the original point cloud.

[0096] The feature extraction layer extracts multi-scale feature maps from the feature set output from the SetAbstraction layer. In specific implementation, the feature extraction layer may include multiple connection layers and at least two maximum pooling layers. The connection layer extracts between the connection layers through convolution kernels of different sizes to reduce feature dimensions and extract the most significant features. Key feature points such as object shape, position, orientation and other information features are extracted from the original point cloud. Each Set Abstraction layer reduces the resolution of the original point cloud to 1 / 4, 1 / 16, 1 / 64 of the original to extract more semantic point cloud features. After each SetAbstraction layer, a connection layer is used to map the point cloud features to a new space; the features are further downsampled through the pooling layer to reduce the number of features. The feature extraction layer can use the Ball Query algorithm to select a fixed number of neighborhood points around each key point, and use the PointNet++ structure to extract local features for each neighborhood point, where the PointNet++ structure extracts the features of the point set through a multi-layer perceptron and maximum pooling operations.

[0097] The outputs of all pooling layers are combined and then passed through the global feature extraction module to extract global point cloud features. These global features are output through the encoder to obtain global features. After obtaining the global features, they are fused with the query point (x, y, z) features. The results of feature fusion are input into multiple multi-layer perceptrons (MPL) for processing point cloud features and point cloud type prediction. The final output point cloud type probability prediction determines the type of the input point cloud.

[0098] It should be noted that when the model is identifying the point cloud type, it first obtains the initial type identification based on the feature information of the point cloud. On the basis of the initial category information, it will repeatedly combine the types of adjacent point clouds to confirm the previously identified point cloud type information. Finally, the model will output the type with the highest prediction probability as the final point cloud type.

[0099] As described above, the sliding window slides along the X-axis, Y-axis and Z-axis of the world coordinate system according to the preset step length; each time it slides, the point cloud information within the first preset radius is obtained as the reference environment point cloud subset data set until all the reference environment point cloud data are statistically completed; wherein the preset step length is less than the first preset radius value. That is to say, each time the sliding window slides, there will be overlapping areas, so there will be repeated determination of the point cloud type, that is, point cloud data will be repeated in different point cloud data subsets obtained. In specific implementation, the overlapping areas can be deduplicated according to the central point cloud recorded in each point cloud data subset and the first preset radius value.

[0100] S140 loads the environment point cloud type information identified by the model into the simulation platform to obtain a simulated reconstruction environment of the robot.

[0101] According to the identified environment point cloud type information, the constructed environment model is imported into the simulation platform (such as Gazebo, V-REP, etc.) to obtain the simulation reconstruction environment of the robot's operating environment.

[0102] The embodiment of the present invention obtains the original point cloud of the surrounding environment through at least one laser radar configured by the robot; pre-processes the original point cloud to obtain the reference environment point cloud; uses the trained Occupancy Networks model to identify each input reference environment point cloud to obtain the reference environment point cloud type information, wherein the OccupancyNetworks model adopts a multi-layer perceptron structure, including a SetAbstraction layer and a feature extraction layer; according to each reference environment point cloud type information, it is loaded on the simulation platform to obtain the simulated reconstruction environment of the robot. This technical solution can quickly and accurately determine the type of environmental point cloud, and quickly reconstruct the simulation environment that is the same as the actual operating environment of the robot in the simulation software. The test results can better reflect the performance of the robot in the actual operating scenario, reduce the test errors caused by environmental differences, and improve the credibility of the test results.

[0103] Embodiment 2

[0104] Figure 4 This is a flow chart of a method for simulating and reconstructing a robot environment based on point cloud provided in the second embodiment of the present invention. This embodiment is optimized based on the above embodiment. The specific optimization is: using the Poisson reconstruction algorithm to convert different types of reference environment point clouds identified by the Occupancy Networks network model into mesh format for storage.

[0105] like Figure 4 As shown in the figure, different types of reference environment point clouds are converted into mesh format through the Poisson reconstruction algorithm, including:

[0106] S210 constructing a vector field: constructing a vector field v according to each point and normal vector in the reference environment point cloud;

[0107] Vector Field

[0108] Where v(q) is the vector field at q, pi is the position of the i-th point in the input point cloud, ni is the normal vector of point pi, F is the direction-sensitive kernel function, (q-pi) represents the vector from point pi to point q, d is the dimension of the kernel function, ||q-pi|| represents the Euclidean distance from point q to pi, N is the total number of points in the point cloud, F is the direction-sensitive kernel function, and F(r)=max(0,Rr) 3 / R 3 ,R is the supported search radius, and r represents the distance from point q to point pi.

[0109] S220 Solve Poisson's equation: Take the vector field as the source term and construct Poisson's equation ▽ 2 χ=▽·v, determine the approximate value of the indicator function χ, where χ is positive inside the surface, negative outside the surface, and zero or close to zero on the surface; where ▽2 = represents the Laplace operator, which is used to smooth the point cloud, reduce noise, improve the surface quality of the point cloud, and generate a continuous surface; ▽ represents the divergence operator, which is used to identify the boundary or surface changes of an object and evaluate the distribution density of the point cloud. By solving the Poisson equation, a smooth and continuous surface can be obtained from the original point cloud data. The Laplace operator used in the present invention ▽ 2 and divergence operator ▽·. The relevant contents can be referred to the prior art and will not be elaborated here.

[0110] S230 Isosurface extraction: Using the Marching Cubes algorithm, isosurfaces are extracted from the solved indicator function χ and stored as the final mesh model.

[0111] In this embodiment, the point cloud is saved in mesh format through the Poisson reconstruction algorithm, which can be quickly loaded in the simulation software and significantly reduce the storage space; and the mesh format provides a continuous surface representation, no longer a discrete point cloud, so that the loaded image is closer to the real-world object and provides a more accurate test environment in the simulation software.

[0112] Embodiment 3

[0113] Figure 5 A flowchart of a method for simulating and reconstructing a robot environment based on point cloud is provided in the third embodiment of the present invention. This embodiment is optimized based on the above embodiment. The specific optimization is: before using the trained OccupancyNetworks model to perform the recognition step for each input reference environment point cloud, the model is supervisedly trained using a simulation data set, specifically including constructing a data set and performing supervised training.

[0114] The process of constructing the dataset is as follows: Figure 5 As shown:

[0115] S310 builds a simulation environment for the robot through simulation software. In specific implementation, the robot simulation environment pre-installed in the simulation software can be equipped with various scene types as needed, such as walls, pillars, pallets, robots equipped with laser radars, etc.

[0116] S320 obtains the simulation environment point cloud and point cloud type information; the simulation environment point cloud can be obtained through a robot equipped with a laser radar, and the point cloud type can be obtained through a simulation software interface.

[0117] S330 maps the simulation environment point cloud to a second grid to obtain a second grid with simulation environment point cloud type information;

[0118] Similar to the method of mapping the static environment point cloud into the first grid map described above, first initialize a second 3D grid map. The size of the second 3D grid map and the size of each grid cell are determined according to the environmental range. In this embodiment, the size of the grid can be selected as 10cm*10cm*10cm. Convert the three-dimensional coordinates (x, y, z) in the simulation environment point cloud data to the coordinate system of the second 3D grid map. Traverse each point in the simulation environment point cloud data set and assign it to the corresponding grid cell in the second 3D grid map according to its coordinates to obtain a second grid with Fengzhen environment point cloud type information.

[0119] S340 constructs a training data set according to the simulation environment point cloud and the second grid with the simulation environment point cloud type information.

[0120] After the training data set is obtained according to the simulation environment point cloud and the second grid with the simulation environment point cloud type information, it includes:

[0121] Randomly select any point cloud in the training data set, take the point cloud as the center point, use the sliding window method to slide along the X-axis, Y-axis and Z-axis directions of the world coordinate system according to the second preset step size, obtain the point cloud within the second radius as the input point of the Occupancy Networks model, and perform supervised training through the annotated labels;

[0122] The point cloud is repeatedly selected for model training, and the model parameters are iteratively optimized until the model output result meets the loss function requirements. The loss function used in training the point cloud type recognition model can be a cross entropy loss function L, and the calculation formula of the cross entropy loss function L is:

[0123]

[0124] N represents the number of point clouds, p_ic represents the predicted probability that the model predicts that point i belongs to type c, and y_ic represents the true pre-labeled label probability that point i belongs to category c.

[0125] The technical solution of this embodiment utilizes the point cloud of the constructed simulation environment and the pre-labeled point cloud types to construct a simulation data set, which can provide a large amount of training data for model training; and through the simulation software, point cloud data for various scenarios can be constructed as needed, thereby improving resource utilization and data generation efficiency, and the cross-entropy function can be used to control the model training accuracy and improve the model recognition accuracy.

[0126] Embodiment 4

[0127] Figure 6This is a schematic diagram of the structure of a point cloud-based robot simulation environment reconstruction device provided in the fourth embodiment of the present invention. The device can execute the point cloud-based robot environment simulation reconstruction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Figure 6 As shown, the device comprises:

[0128] The environment point cloud acquisition module 610 is used to acquire the original point cloud data of the surrounding environment through at least one laser radar configured by the robot;

[0129] The point cloud processing module 620 is used to pre-process the original point cloud data to obtain a reference environment point cloud;

[0130] The point cloud type prediction module 630 is used to identify each reference environment point cloud data input by using the trained OccupancyNetworks model to obtain the reference environment point cloud type information, wherein the OccupancyNetworks model adopts a multi-layer perceptron structure, including a SetAbstraction layer and a feature extraction layer;

[0131] The environment loading determination module 640 is used to load the point cloud type information of each reference environment on the simulation platform to obtain the simulation reconstruction environment of the robot.

[0132] Optionally, the point cloud processing module 620 further includes:

[0133] A static environment point cloud determination unit, configured to detect and filter the dynamic obstacle point cloud in the original point cloud based on a time series algorithm to obtain a static environment point cloud;

[0134] A static environment point cloud grid determination unit, used for mapping the static environment point cloud into a first grid map to obtain a static environment point cloud grid;

[0135] A reference environment grid determination unit is used to calculate the point cloud density of each of the static environment point cloud grids, perform point cloud filtering on grids with point cloud density less than a preset threshold, and obtain a reference environment grid, wherein the reference environment grid includes a reference environment point cloud;

[0136] The reference environment grid type identification unit is used to identify the reference environment point cloud in the reference environment grid by using the trained Occupancy Networks model to determine the reference environment point cloud type information.

[0137] Optionally, the point cloud type prediction module 630 includes:

[0138] A center point cloud determination module is used to randomly select any reference environment point cloud in the reference environment grid as the center point and save the position information of the center point;

[0139] A reference environment sub-dataset determination module is used to slide along the X-axis, Y-axis and Z-axis directions of the world coordinate system according to a preset step length using a sliding window method; each time the sliding is performed, point cloud information within a first preset radius is obtained as a reference environment point cloud sub-dataset until the statistics of all reference environment point cloud data are completed; wherein the preset step length is less than the first preset radius value;

[0140] The reference environment point cloud sub-dataset recognition module is used to identify the reference environment point cloud sub-dataset according to the trained Occupancy Networks model and determine the reference environment point cloud type;

[0141] The format conversion and saving module is used to convert different types of reference environment point clouds into mesh format through the Poisson reconstruction algorithm.

[0142] The optional point cloud type prediction module 630 also includes:

[0143] A vector field construction module, used to calculate the vector field v according to each point and normal vector in the reference environment point cloud according to the vector field calculation formula;

[0144] Vector Field

[0145] Where v(q) is the vector field at q, pi is the position of the i-th point in the input point cloud, ni is the normal vector of point pi, F is the direction-sensitive kernel function, (q-pi) represents the vector from point pi to point q, d is the dimension of the kernel function, ||q-pi|| represents the Euclidean distance from point q to pi, and N is the total number of points in the point cloud;

[0146] F is the direction-sensitive kernel function, F(r) = max(0,Rr) 3 / R 3 ,R is the supported search radius, r represents the distance from point q to point pi;

[0147] Poisson equation solver module, used to construct Poisson equation using vector field as source term▽ 2 χ=▽·v, determine the approximate value of the indicator function χ, where χ is positive inside the surface, negative outside the surface, and zero or close to zero on the surface; where ▽ 2 represents the Laplace operator, and ▽· represents the divergence operator.

[0148] Isosurface extraction and determination module: used to extract isosurfaces from the solved indicator function χ using the Marching Cubes algorithm and store them as the final mesh model.

[0149] Optionally, the device further comprises:

[0150] A simulation environment building module, used to build a simulation environment for the robot through simulation software, including at least walls, pillars, trays, and a robot equipped with a laser radar;

[0151] A simulation environment point cloud type acquisition module is used to obtain type information of the simulation environment point cloud through simulation software;

[0152] A simulation environment point cloud type information grid determination module is used to map the simulation environment point cloud to a second grid to obtain a second grid with simulation environment point cloud type information;

[0153] The training data set determination module is used to construct a training data set according to the simulation environment point cloud and the second grid with the simulation environment point cloud type information.

[0154] Optionally, the device further comprises:

[0155] An OccupancyNetworks model input point determination module is used to randomly select any point cloud in the training data set, take the point cloud as the center point, use a sliding window method to slide along the X-axis, Y-axis and Z-axis directions of the world coordinate system according to a second preset step length, and obtain a point cloud within the second radius as the input point of the OccupancyNetworks model;

[0156] The supervised training module is used to perform supervised training according to the type of each grid within the second radius through the marked labels, repeatedly select the point cloud for model training, and iteratively optimize the model parameters until the model output results meet the loss function requirements.

[0157] Optionally, the point cloud type prediction module 630 further includes:

[0158] A point cloud input module, used for receiving the robot environment point cloud, including the surrounding environment original point cloud and / or the simulated environment point cloud;

[0159] The point cloud conversion module includes the Set Abstraction layer, which contains multiple downsampling layers and feature extraction layers, among which:

[0160] The downsampling layer uses the farthest point sampling algorithm to select key points;

[0161] The feature extraction layer selects neighborhood points around each key point and extracts local features through the PointNet++ structure;

[0162] A feature extraction layer, comprising a plurality of connection layers and a pooling layer, extracts a multi-scale feature map from the feature set output by the SetAbstraction layer;

[0163] The encoder module converts the feature map into a global feature vector through global feature abstraction processing;

[0164] The feature fusion module includes a multi-layer perceptron (MLP) layer to fuse the query point (x, y, z) and global feature features;

[0165] A spatial type determination module is used to output the type prediction probability of each query point to form a type map in three-dimensional space;

[0166] The loss calculation and optimization module uses a loss function to evaluate the difference between the predicted type and the actual type, and adjusts the network parameters through an optimization algorithm to minimize the loss. The loss function used when training the point cloud type recognition model can be a cross entropy loss function L, and the calculation formula of the cross entropy loss function L is:

[0167]

[0168] N represents the number of point clouds, p_ic represents the predicted probability that the model predicts that point i belongs to type c, and y_ic represents the true pre-labeled label probability that point i belongs to category c.

[0169] The point cloud-based robot environment simulation and reconstruction device provided in the embodiment of the present invention can execute the point cloud-based robot environment simulation and reconstruction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0170] Embodiment 5

[0171] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0172] like Figure 7As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0173] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0174] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a point cloud-based robot environment simulation and reconstruction method.

[0175] In some embodiments, the point cloud-based robot environment simulation and reconstruction method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the point cloud-based robot environment simulation and reconstruction method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the point cloud-based robot environment simulation and reconstruction method by any other appropriate means (for example, by means of firmware).

[0176] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0177] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0178] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0179] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0180] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0181] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0182] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0183] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A point cloud-based robot environment simulation and reconstruction method, characterized in that: The method comprises: Acquire the original point cloud of the surrounding environment through at least one laser radar configured by the robot; Preprocessing the original point cloud to obtain a reference environment point cloud; using the trained Occupancy Networks model to identify each input reference environment point cloud to obtain reference environment point cloud type information, wherein the Occupancy Networks model adopts a multi-layer perceptron structure, including a Set Abstraction layer and a feature extraction layer; According to the point cloud type information of each reference environment, the simulation reconstruction environment of the robot is obtained by loading it on the simulation platform.

2. The method according to claim 1, characterized in that The original point cloud is preprocessed to obtain a reference environment point cloud, including: Based on a time series algorithm, the dynamic obstacle point cloud in the original point cloud is detected and filtered to obtain a static environment point cloud; Mapping the static environment point cloud into a first grid map to obtain a static environment point cloud grid; Calculating the point cloud density of each of the static environment point cloud grids, performing point cloud filtering on the grids whose point cloud density is less than a preset threshold, and obtaining a reference environment grid, wherein the reference environment grid includes a reference environment point cloud; The trained Occupancy Networks model is used to identify the reference environment point cloud in the reference environment grid to determine the reference environment point cloud type information.

3. The method according to claim 2, characterized in that Using the trained Occupancy Networks model to identify the reference environment point cloud in the reference environment grid, and determining the reference environment point cloud type information; including: Randomly select any reference environment point cloud in the reference environment grid as the center point, save the position information of the center point, and use the sliding window method to slide along the X-axis, Y-axis and Z-axis directions of the world coordinate system according to the preset step length; each time sliding, obtain the point cloud information within the first preset radius as the reference environment point cloud sub-dataset until the statistics of all the reference environment point cloud data are completed; wherein the preset step length is less than the first preset radius value; Identify the reference environment point cloud sub-dataset according to the trained Occupancy Networks model to determine the reference environment point cloud type; The reference environment point clouds of different types are converted into mesh format through the Poisson reconstruction algorithm.

4. The method according to claim 3, characterized in that Different types of reference environment point clouds are converted into mesh format using the Poisson reconstruction algorithm, including: Constructing a vector field: constructing a vector field v according to each point and normal vector in the reference environment point cloud; Vector Field Where v(q) is the vector field at q, pi is the position of the i-th point in the input point cloud, ni is the normal vector of point pi, F is the direction-sensitive kernel function, (q-pi) represents the vector from point pi to point q, d is the dimension of the kernel function, ‖q-pi‖ represents the Euclidean distance from point q to pi, and N is the total number of points in the point cloud; F is the direction-sensitive kernel function, F(r) = max(0,Rr) 3 R3, R is the supported search radius, r represents the distance from point q to point pi; Solve Poisson's equation: Use the vector field as the source term to construct Poisson's equation Determine an approximate value of the indicator function χ, where χ is positive inside the surface, negative outside the surface, and zero or close to zero on the surface; where represents the Laplace operator, represents the divergence operator. Isosurface extraction: The Marching Cubes algorithm is used to extract isosurfaces from the solved indicator function χ and store them as the final mesh model.

5. The method according to claim 1, characterized in that Before using the trained Occupancy Networks model to identify each reference environment point cloud input, the following steps are included: Use simulation software to build a robot simulation environment, including at least walls, pillars, trays, and a robot equipped with a laser radar; Acquire the simulation environment point cloud and type information of the simulation environment point cloud; Mapping the simulation environment point cloud to a second grid to obtain a second grid with simulation environment point cloud type information; A training data set is constructed based on the simulation environment point cloud and the second grid with the simulation environment point cloud type information.

6. The method according to claim 5, characterized in that After the training data set is obtained according to the simulation environment point cloud and the second grid with the simulation environment point cloud type information, the method includes: Randomly select any point cloud in the training data set, take the point cloud as the center point, use the sliding window method to slide along the X-axis, Y-axis and Z-axis directions of the world coordinate system according to the second preset step size, obtain the point cloud within the second radius as the input point of the OccupancyNetworks model, obtain the type of each grid within the second radius, and perform supervised training through the annotated labels; Repeatedly select the simulation environment point cloud for model training, and iteratively optimize model parameters until the model output results meet the loss function requirements.

7. The method according to any one of claims 1 to 6, characterized in that: The Occupancy Networks model includes: A point cloud input module, used for receiving the robot environment point cloud, including the surrounding environment original point cloud and / or the simulated environment point cloud; The Set Abstraction layer includes multiple downsampling layers and feature extraction layers, among which: The downsampling layer selects key points using the farthest point sampling algorithm; The feature extraction layer includes a plurality of connection layers and a pooling layer, and selects neighborhood points around each key point to extract local features from the feature set output by the Set Abstraction layer through the PointNet++ structure; An encoder module, using a global pooling operation, converts the feature map into a global feature vector; A multi-layer perceptron (MLP) layer for fusing the query point (x, y, z) coordinates and the global feature vector; The output layer outputs the type prediction probability of each query point to form a predicted type probability map in three-dimensional space; The loss calculation and optimization module uses the cross entropy loss function to evaluate the difference between the predicted type and the actual type, and adjusts the network parameters through the optimization algorithm.

8. A point cloud-based robot environment simulation and reconstruction device, characterized in that: The device comprises: An environmental point cloud acquisition module, used to acquire raw point cloud data of the surrounding environment through at least one laser radar configured by the robot; A point cloud processing module, used for preprocessing the original point cloud data to obtain a reference environment point cloud; The point cloud type prediction module is used to use the trained OccupancyNetworks model to identify each input reference environment point cloud data and obtain the reference environment point cloud type prediction information; The environment loading determination module is used to load the simulation reconstruction environment of the robot on the simulation platform according to the prediction information of each reference environment point cloud type.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the point cloud-based robot environment simulation and reconstruction method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the point cloud-based robot environment simulation and reconstruction method according to any one of claims 1 to 7 when executed.

Images