Plant high-definition map construction method based on edge-fog-cloud cooperation

Abstract

The invention discloses a factory high-definition map construction method based on edge-fog-cloud cooperation, and the method comprises the steps: S1, collecting the environment information of a region in a factory scene, and transmitting the environment information to a fog calculation server in the region; s2, the fog computing server performs feature extraction, feature conversion, rasterization processing and map post-processing on the received environment information to obtain a high-definition map of a current area in the factory scene, and transmits the high-definition map to a cloud server through an ROS message mechanism; s3, repeatedly executing the steps S1 and S2 until all local occupied grid maps of the whole factory scene are generated; s4, the cloud server splices all the local occupation grid maps into an integral occupation grid map; and extracting high-level information of semantic prediction and direction prediction from the overall occupied grid map to obtain a factory high-definition map. According to the method, the efficiency of building the factory high-definition map is improved, and the precision of the factory high-definition map is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of driverless technology, and particularly to a method for constructing a high-definition map of an industrial park based on edge-fog-cloud collaboration. Background Art

[0002] A high-definition map, also known as a high-resolution map, is a map specifically for driverless services. Different from traditional navigation maps, it can not only provide road-level navigation information but also lane-level navigation information. In terms of both the richness and high precision of information, it far exceeds traditional navigation maps.

[0003] Industrial parks in closed scenarios have advantages such as fixed routes and simple operations, which contribute to the commercial implementation of driverless in this scenario. The work of driverless vehicles in this scenario mainly involves short-distance cargo transportation between various warehouses and workshops in the closed park, with the goal of approaching the transfer platform efficiently and accurately. However, since the quantity and location of goods stacked in the warehouse are constantly changing, this poses a high requirement for the map update frequency.

[0004] The existing single-vehicle SLAM mapping method based on multi-sensor fusion can only rely on loop detection to reduce cumulative errors, with low efficiency and accuracy. Summary of the Invention

[0005] Aiming at the problems existing in the prior art, the present invention provides a method for constructing a high-definition map of an industrial park based on edge-fog-cloud collaboration with high efficiency and accuracy.

[0006] To this end, the present invention adopts the following technical solutions:

[0007] A method for constructing a high-definition map of an industrial park based on edge-fog-cloud collaboration includes the following steps:

[0008] S1, collecting environmental information of an area in the industrial park scene and transmitting the environmental information of this area to the fog computing server in this area, where the environmental information includes the original image and original point cloud of this area;

[0009] S2, the fog computing server performs feature extraction, feature transformation, rasterization processing, and map post-processing on the received environmental information to obtain a high-definition map of the current area in the industrial park scene, and transmits it to the cloud server as a local occupancy grid map through the ROS message mechanism,

[0010] S3, repeating steps S1 and S2 until local occupancy grid maps of each area in the entire industrial park scene are generated;

[0011] S4. In the cloud server, splice the multiple local occupancy grid maps generated by multiple fog computing servers into an overall occupancy grid map; perform high-level information extraction such as semantic prediction and direction prediction on the overall occupancy grid map to obtain a high-definition map of the factory area.

[0012] The S2 includes the following sub-steps:

[0013] S21. Respectively perform feature extraction on the original image and the original point cloud in the environmental information to obtain all feature points O m,n,k of the original image and all feature points O h,w of the original point cloud. The specific steps are as follows:

[0014] S211. Perform feature extraction on the original image in the environmental information, including:

[0015] Normalize, scale, and crop the original image to adjust the original image to the scale required for convolution operations to obtain a processed image;

[0016] Perform feature extraction on the processed image through a pre-trained convolutional layer. The calculation formula of the convolutional layer is as follows:

[0017] O m,n,k =∑ m,n W m,n,k ·I i+m,j+n +b k

[0018] where O m,n,k is the feature point at (i + m, j + n) in the k-th layer of the convolutional layer, W m,n,k is the convolution kernel of the convolutional layer, b k is the bias of the convolutional layer, and I i+m,j+n represents the pixel value of the local area of the processed image centered on (i, j) with a bias of (m, n);

[0019] According to the feature points O m,n,k output by the k-th layer convolutional layer, form the feature O k of this layer. Summarize the features of all convolutional layers to form a corresponding feature map F. That is, the value at the corresponding position in the feature map F comes from the output O m,n,k of the convolutional layer. The feature map F is a three-dimensional tensor, and the dimensions of the feature map F are C×H×W. Among them, C is the number of channels, reflecting the number of convolution kernels, and H and W respectively represent the height and width of the feature map F. By stacking multiple convolutional layers, multi-scale representation of visual features can be achieved;

[0020] S212. Based on the PointPillar network, perform feature extraction on the original point cloud in the environmental information to obtain the feature points O in the original point cloudh,w ;

[0021] S22. Transform the coordinates of all feature points O m,n,k and all feature points O h,w to the world coordinate system;

[0022] S23. Generate an empty occupancy grid map occ map ; Map all feature points O m,n,k and all feature points O h,w in the world coordinate system to the empty occupancy grid map occ map to obtain the occupancy grid map occ map '; Update the occupancy probability of each grid in the occupancy grid map occ map ' according to the observed occupancy probabilities of the vision sensor and the Lidar sensor using the probability update method to obtain the occupancy grid map occ map '';

[0023] S24. Post-process the occupancy grid map occ map '' to obtain a high-definition map of the current area in the factory area. Preferably, the post-processing includes:

[0024] Noise filtering: Median filter each grid according to the median value of each grid in the occupancy grid map occ map '' to remove isolated noise points or small regions in the grid map, eliminate noise, and improve the clarity and reliability of the occupancy grid map;

[0025] Dilation operation: Expand each grid in the occupancy grid map to its neighborhood and fill the empty grids;

[0026] Erosion operation: Shrink each grid in the occupancy grid map to its neighborhood and remove isolated points in the grid;

[0027] Connectivity analysis: Use the connected component labeling algorithm to identify and process the connected regions in the occupancy grid map occ map '' to separate real obstacles and noise regions.

[0028] The S23 includes the following sub-steps:

[0029] S231. Initialize the grid:

[0030] Set the size M×N of the grid map and the resolution r of the grid map, and initialize the grid map to generate an empty occupancy grid map occ map :

[0031] occ map = zeros(M,N)

[0032] Among them, the zeros() function is used to create a matrix of size M×N with all elements being zero, and this matrix is used to represent the initialized empty occupancy grid map occ map ;

[0033] S232, Feature point mapping: Map all the feature points O m,n,k and O h,w in the world coordinate system obtained in S22 to the empty occupancy grid map occ map to obtain the occupancy grid map occ map ′. Among them, if the coordinate of any feature point in the world coordinate system is (x, y), then the mapping formula for the coordinate (u, v) of this feature point in the grid map occ map ′ is as follows:

[0034]

[0035] where x min and y min respectively represent the smallest abscissa and the smallest ordinate among all feature points in the world coordinates, and r is the resolution of the occupancy grid map occ map ′;

[0036] S233, Feature point fusion: According to the observed occupancy probability of the vision sensor and the observed occupancy probability of the Lidar sensor, use the method of probability update to update the occupancy probability of each grid in the occupancy grid map occ map ′ to obtain the occupancy grid map occ map ″. Among them, the value in each grid of the occupancy grid map occ map ″ is the occupancy probability of this grid, and the occupancy probability represents the possibility that this grid is occupied by feature points.

[0037] In the above-mentioned S12, between the edge node and the fog computing server, the environmental information is transmitted through the message mechanism of ROS based on Ubuntu20.04. The edge node publishes the environmental information to the fog computing server; the fog computing server subscribes to the environmental information published by the edge node and further processes the received environmental information.

[0038] The steps in the above-mentioned S212 are as follows:

[0039] Convert the original point cloud into multiple pillars of equal size, each of which stores local point cloud information. The original point cloud includes N laser points, and each laser point has a corresponding three-dimensional coordinate and a corresponding dimensional feature K. Use the PointNet network to extract local features for each pillar to obtain the feature vector corresponding to each pillar. Use the PointNet network to aggregate the feature vectors corresponding to each pillar, aggregate the feature vectors corresponding to all pillars into a two-dimensional columnar feature map, and extract high-level features from the two-dimensional columnar feature map through a two-dimensional convolutional neural network. Finally, the feature points in the features of the original point cloud are output as O h,w , where h represents the height index in the two-dimensional columnar feature map, and w represents the width index in the two-dimensional columnar feature map.

[0040] The specific steps of S233 are as follows:

[0041] (1) Initialize the occupancy probability P(OCC map[u,v] ′) = P0, where P0 represents the initial occupancy probability, set to the neutral probability (0.5);

[0042] (2) Normalize the sum of the observed occupancy probabilities of the visual sensor and the Lidar sensor;

[0043] (3) Judge each grid in the occupancy grid map occ map ′: If there is only one type of sensor's feature points in a grid, then the occupancy probability of this grid is the observed occupancy probability of this sensor; if there are two types of sensor's feature points in a grid, then update the occupancy probability of each grid through the Bayesian update formula:

[0044]

[0045] where P sensor represents the observed occupancy probability of the visual sensor, (1 - P sensor ) represents the observed occupancy probability of the Lidar sensor, P(occ map[u,v]sensor ) represents the occupancy probability of the current grid, and P(occ map[u,v] ′) represents the prior occupancy probability of the grid, that is, the initial occupancy probability or the updated occupancy probability.

[0046] S4 includes the following steps:

[0047] S41, stitching and merging of the local occupancy grid map: Perform coordinate transformation, stitching and merging on multiple local occupancy grid maps generated by multiple fog computing servers, and stitch multiple local occupancy grid maps into an overall occupancy grid map;

[0048] S42. Extract high-level information such as semantic prediction and direction prediction from the overall occupancy grid map to obtain a high-definition map of the factory area;

[0049] Preferably, the specific steps of S41 are as follows:

[0050] Coordinate transformation: Each local occupancy grid map is transformed to the global coordinate system through translation and rotation, and the pose of each local map in the global coordinate system is obtained through calculation. Among them, a point (u′, v′) in a local occupancy grid map is transformed to the global coordinate system, and the transformation formula is:

[0051]

[0052] where (x′, y′) is the coordinate of the point (u′, v′) in the global coordinate system, and (x k , y k ) represents the position of the origin of the local occupancy grid map in the local coordinate system in the global coordinate system, θ k represents the pose transformation angle, and the corresponding heading angle θ is obtained by measuring the angular velocity of the driverless vehicle in real time through the in-vehicle inertial measurement unit IMU k , and r is the resolution of the local occupancy grid map;

[0053] Stitching and merging: In the global coordinate system, align according to the common area between each local occupancy grid map, stitch each local occupancy grid map to obtain an overall occupancy grid map, and then use the method of probability update to merge and update each grid in the overall occupancy grid map, so as to realize the rapid update of the overall occupancy grid map.

[0054] The specific steps of the said S42 are as follows:

[0055] Semantic prediction: Classify each grid in the overall occupancy grid map through semantic category prediction to determine the semantic category to which each grid belongs. Among them, the semantic categories include roads, buildings, pedestrians, and vehicles. Semantic category prediction is to select the category with the highest probability in the grid as the semantic category of the grid. The specific formula is as follows:

[0056]

[0057] where s i″,v″ is the semantic category of the grid (u″, v″), c is the number of categories of s u′′,v"" , and represents the probability that the grid (u″, v″) belongs to category c;

[0058] Direction prediction: Predict the moving direction of objects with moving ability in each grid of the overall occupancy grid map, for example, the driving direction of a vehicle or the walking direction of a pedestrian.

[0059] The specific steps of S1 are as follows:

[0060] S11, Take the driverless vehicle in the factory area as an edge node, and collect the environmental information of an area in the factory area through the vision sensor and Lidar sensor carried by the driverless vehicle to obtain the environmental information of the current area;

[0061] S12, Transmit the environmental information of the current area from the edge node to the fog computing server corresponding to the current area.

[0062] The specific steps of S22 are as follows:

[0063] All feature points O m,n,k Are converted from the image coordinate system to the camera coordinate system, and the coordinate conversion formula is as follows:

[0064]

[0065] Among them, i+m, j+n are the coordinates of the feature point O m,n,k In the image coordinate system, k is the internal parameter matrix of the camera, and d is the depth value of the camera; All feature points O m,n,k After mapping, are converted from the camera coordinate system to the vehicle coordinate system; Then, through rotation and translation, all feature points O m,n,k Are converted from the vehicle coordinate system to the world coordinate system;

[0066] Perform coordinate conversion on all feature points O h,w Perform coordinate conversion on all feature points O h,w Are converted from the Lidar coordinate system to the vehicle coordinate system through rotation and translation, and then from the vehicle coordinate system to the world coordinate system through rotation and translation.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. The method for constructing a high-definition map of the factory area of the present invention adopts an edge-fog-cloud system architecture, processes the data collected by two sensors in the occupancy grid map, and completes the construction of the high-definition map of the factory area.

[0069] 2. In the method for constructing a high-definition map of the factory area of the present invention, the cloud (cloud server) can receive data from multiple fog ends (fog computing servers) and edge ends (edge nodes), and can collaboratively construct a high-definition map of the factory area, which not only improves the efficiency of constructing the high-definition map of the factory area, but also avoids the cumulative error of long-term SLAM and improves the accuracy of the high-definition map of the factory area.

[0070] 3. The present invention deploys Hadoop in the cloud server, realizing the convenient call and efficient management of high-definition maps of large-scale factory areas under the condition of frequent changes in the factory environment.

[0071] 4. The method of the present invention realizes the rapid update of the map through feature alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a flowchart of an embodiment of the method for constructing a high-definition map of a factory area of the present invention;

[0073] Figure 2 is a schematic diagram of the edge-cloud-fog scenario of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0074] The technical solutions of the present invention will be further described in detail below with reference to the drawings and embodiments.

[0075] Refer to Figure 1 and Figure 2 , a method for constructing a high-definition map of a factory area based on edge-fog-cloud collaboration of the present invention includes the following steps:

[0076] S1. Collect the environmental information of an area in the factory area scene and transmit the environmental information of this area to the fog computing server in this area. The specific steps are as follows:

[0077] S11. Use the driverless vehicle in the factory area scene as an edge node, and collect the environmental information of an area in the factory area scene through the vision sensor and Lidar sensor carried by the driverless vehicle to obtain the environmental information of the current area.

[0078] Among them, the environmental information includes the original image of the current area collected by the vision sensor and the original point cloud of the current area collected by the Lidar sensor.

[0079] S12. Transmit the environmental information of the current area from the edge node (driverless vehicle) to the fog computing server corresponding to the current area.

[0080] Among them, the environmental information is transmitted between the edge node and the fog computing server through the message mechanism of ROS based on Ubuntu20.04. The edge node publishes data (environmental information) to the fog computing server; the fog computing server subscribes to the data (environmental information) published by the edge node and further processes the received data (environmental information).

[0081] S2. The fog computing server performs feature extraction, feature transformation, rasterization processing, and map post-processing on the received environmental information to obtain a high-definition map of the current area in the factory area scene, and transmits it to the cloud server as a local occupancy grid map through the ROS message mechanism. The specific steps are as follows:

[0082] S21. Perform feature extraction on the original image and the original point cloud in the environmental information respectively, including:

[0083] S211. Perform feature extraction on the original image in the environmental information:

[0084] Normalize, scale, and crop the original image to adjust the original image to the scale required for convolution operations, and obtain the processed image. The normalization formula is:

[0085]

[0086] where, I norm represents the normalized image, I represents the original image, μ represents the mean of the original image, and σ is the standard deviation of the original image.

[0087] Perform feature extraction on the processed image through a pre-trained convolutional layer. Among them, the calculation formula of the convolutional layer is as follows:

[0088] o m,n,k =∑ m,n w m,n,k ·I i+m,j+n +b k

[0089] where, O m,n,k is the feature point at (i + m, j + n) in the k-th layer (k = 5) of the convolutional layer, W m,n,k is the convolution kernel of the convolutional layer, b k is the bias of the convolutional layer, I i+m,j+n represents the pixel value of the local area of the processed image centered on (i, j) with a bias of (m, n);

[0090] According to the feature points O m,n,k output by the k-th convolutional layer, form the feature O k of this layer. Summarize the features of all convolutional layers to form the corresponding feature map F. That is, the value at the corresponding position in the feature map F comes from the output O m,n,k of the convolutional layer. The feature map F is a three-dimensional tensor, and the dimension of the feature map F is C × H × W. Among them, C is the number of channels, which reflects the number of convolution kernels, and H and W respectively represent the height and width of the feature map F. By stacking multiple convolutional layers, multi-scale representation of visual features can be achieved.

[0091] S212. Based on the PointPillar network (Lang A H, Vora S, Caesar H, et al. Pointpillars: Fast encoders for object detection from point clouds [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 12697-12705.), extract features from the original point cloud in the environmental information to obtain the feature points O in the original point cloud. h,w , and the specific steps are as follows:

[0092] Convert the original point cloud into multiple pillars of equal size, and each pillar stores local point cloud information. Among them, the original point cloud includes N laser points, and each laser point has a corresponding three-dimensional coordinate and a corresponding dimensional feature K; use the PointNet network to extract local features for each pillar to obtain the feature vector corresponding to each pillar; use the PointNet network to perform feature aggregation on the feature vector corresponding to each pillar, aggregate the feature vectors corresponding to all pillars into a two-dimensional columnar feature map, and perform high-level feature extraction on the two-dimensional columnar feature map through a two-dimensional convolutional neural network, and finally output the feature points in the features of the original point cloud as O. h,w , where h represents the height index in the two-dimensional columnar feature map, and w represents the width index in the two-dimensional columnar feature map.

[0093] There is no sequential order for the above steps S211 and S212.

[0094] S22. For all the feature points O obtained in S211 m,n,k and all the feature points O obtained in S212 h,w perform coordinate transformation to the world coordinate system to facilitate subsequent feature fusion and the generation of the OCC map. The specific steps are as follows:

[0095] Perform coordinate transformation on all the feature points O in S211 m,n,k :

[0096] Convert all the feature points O in S211 m,n,k from the image coordinate system to the camera coordinate system. The coordinate transformation formula is as follows:

[0097]

[0098] where i + m, j + n are the coordinates of the feature point O in S211 m,n,k in the image coordinate system, K is the internal parameter matrix of the camera, and d is the depth value of the camera;

[0099] All the feature points O obtained in S211 m,n,k After mapping, convert from the camera coordinate system to the vehicle coordinate system; then through rotation and translation, all the feature points O in S211 m,n,k Are converted from the vehicle coordinate system to the world coordinate system.

[0100] For all the feature points O in S212 h,w Perform coordinate transformation:

[0101] All the feature points O obtained in S212 h,w Are converted from the Lidar coordinate system to the vehicle coordinate system through rotation and translation, and then from the vehicle coordinate system to the world coordinate system through rotation and translation;

[0102] S23. Generate an empty occupancy grid map occ map ; All the feature points O in S211 m,n,k And all the feature points O in S212 h,w Are mapped to the empty occupancy grid map occ map To obtain the occupancy grid map occ map '; According to the observed occupancy probabilities of the vision sensor and the Lidar sensor, use the method of probability update to update the occupancy probability of each grid in the occupancy grid map occ map ' to obtain the occupancy grid map occ map ''. The occupancy grid map occ map '' accurately reflects the environmental information after multi-sensor fusion. The specific steps are as follows:

[0103] S231. Initialize the grid:

[0104] Set the size M×N of the grid map and the resolution r of the grid map, and initialize the grid map to generate an empty occupancy grid map occ map :

[0105] occ map = zeros(M,N)

[0106] Among them, the zeros() function is used to create a matrix of size M×N with all elements being zero, and this matrix is used to represent the initialized empty occupancy grid map occ map ;

[0107] S232. Feature point mapping: Map all the feature points O in S22 in the world coordinate system m,n,k And O h,w To the empty occupancy grid map occ map To obtain the occupancy grid map occmap ′, where the coordinates of any feature point in the world coordinate system are (x, y), then the mapping formula for the coordinates (u, v) of this feature point in the occupancy grid map occ map ′ is as follows:

[0108]

[0109] where x min and y min respectively represent the minimum abscissa and minimum ordinate among all feature points in the world coordinates, and r is the resolution of the occupancy grid map occ map ′;

[0110] S233, Feature point fusion: According to the observed occupancy probability of the vision sensor and the observed occupancy probability of the Lidar sensor, the occupancy probability of each grid in the occupancy grid map occ map ′ is updated using the probability update method to obtain the occupancy grid map occ map ″.

[0111] where the value in each grid of the occupancy grid map occ map ″ is the occupancy probability of this grid, and the occupancy probability represents the possibility that this grid is occupied by feature points; the occupancy probability of each grid is updated by the probability update method to make the occupancy grid map occ map ″ more conform to the actual environment. The specific steps are as follows:

[0112] (1) Initialize the occupancy probability P(occ map[u,v] ′) = P0, where P0 represents the initial occupancy probability, set as the neutral probability (0.5);

[0113] (2) Normalize the sum of the observed occupancy probability of the vision sensor and the observed occupancy probability of the Lidar sensor;

[0114] (3) Judge each grid in the occupancy grid map occ map ′: If there is only one type of sensor's feature point in a grid, then the occupancy probability of this grid is the observed occupancy probability of this sensor; if there are two types of sensor's feature points in a grid, then update the occupancy probability of each grid through the Bayesian update formula:

[0115]

[0116] where P sensor represents the observed occupancy probability of the vision sensor, (1 - P sensor ) represents the observed occupancy probability of the Lidar sensor, and P(occ map[u,v]sensor) represents the occupancy probability of the current grid, P(occ map[u,v] ′) represents the prior occupancy probability of the grid, i.e., the initial occupancy probability or the updated occupancy probability.

[0117] S24, post-process the occupancy grid map occ map ″ to improve accuracy and practicality. Among them, the post-processing operations include noise filtering, dilation and erosion, and connectivity analysis. The specific steps of the post-processing operations are as follows:

[0118] Noise filtering: According to the median value of each grid in the occupancy grid map occ map ″, perform median filtering on each grid to remove isolated noise points or small areas (outliers) in the grid map, achieve noise elimination, and improve the clarity and reliability of the occupancy grid map.

[0119] Dilation operation: Expand each grid in the occupancy grid map to its neighborhood and fill the empty grids.

[0120] Erosion operation: Shrink each grid in the occupancy grid map to its neighborhood and remove isolated points in the grid.

[0121] The dilation operation and the erosion operation improve the coherence and continuity of the occupancy grid map occ map ″.

[0122] Connectivity analysis: Use the connected component labeling algorithm to identify and process the connected regions in the occupancy grid map occ map ″, and separate the real obstacles and noise regions.

[0123] S3, repeatedly execute steps S1 and S2 until local occupancy grid maps of each region in the entire factory area scene are generated: Edge nodes collect environmental information for all different regions in the factory area scene, and process the environmental information of each region in the fog computing server corresponding to each region to generate local occupancy grid maps of each region; Transmit the local occupancy grid maps generated for each region from the fog computing server to the cloud server through the ROS message mechanism.

[0124] S4, in the cloud server, splice the multiple local occupancy grid maps generated by multiple fog computing servers into an overall occupancy grid map; Perform high-level information extraction such as semantic prediction and direction prediction on the overall occupancy grid map to obtain a high-definition map of the factory area. Among them, extracting high-level information can make the overall occupancy grid map contain more comprehensive semantic information and improve the details and accuracy of the overall occupancy grid map; Store the data of the high-definition map of the factory area. The specific steps are as follows:

[0125] S41, Stitching and Merging of Locally Occupied Grid Maps: Coordinate transformation, stitching, and merging of multiple locally occupied grid maps generated by multiple fog computing servers are performed to stitch multiple locally occupied grid maps into an overall occupied grid map. The specific steps are as follows:

[0126] Coordinate Transformation: Each locally occupied grid map is transformed to the global coordinate system through translation and rotation, and the pose of each local map in the global coordinate system is obtained through calculation. Among them, when transforming a point (u′, v′) in the locally occupied grid map to the global coordinate system, the transformation formula is:

[0127]

[0128] where (x′, y′) is the coordinate of the point (u′, v′) in the global coordinate system, (x k , y k ) represents the position of the origin of the locally occupied grid map in the local coordinate system in the global coordinate system, θ k represents the pose transformation angle, and the corresponding heading angle θ is obtained by measuring the angular velocity of the driverless vehicle in real time through the in-vehicle inertial measurement unit IMU k , and r is the resolution of the locally occupied grid map.

[0129] Stitching and Merging: In the global coordinate system, alignment is performed according to the common area between each locally occupied grid map, and each locally occupied grid map is stitched to obtain an overall occupied grid map. Then, using the probability update method (step S233), each grid in the overall occupied grid map is merged and updated, thereby achieving rapid update of the overall occupied grid map.

[0130] S42, High-level Information Extraction of Semantic Prediction and Direction Prediction for the Overall Occupied Grid Map to Obtain a High-Definition Map of the Factory Area. The specific steps are as follows:

[0131] Semantic Prediction: Each grid in the overall occupied grid map is classified through semantic class prediction to determine the semantic class to which each grid belongs. Among them, the semantic classes include roads, buildings, pedestrians, and vehicles. Semantic class prediction is to select the class with the highest probability in the grid as the semantic class of the grid. The specific formula is as follows:

[0132]

[0133] where s u,v is the semantic class of the grid (u″, v″), c is the number of classes of s u″,v″ , represents the probability that the grid (u″, v″) belongs to class c.

[0134] Direction prediction: Predict the moving direction of an object with moving ability in each grid of the overall occupancy grid map. For example, the driving direction of a vehicle or the walking direction of a pedestrian.

[0135] S43, Management and storage of the high-definition map of the factory area: The obtained high-definition map of the factory area is transmitted to HDFS deployed based on CentOS7 through Kafka for management and storage. Among them, inside the cloud server, HDFS (Hadoop Distributed File System) deployed based on CentOS7 is used to efficiently manage and store the data (high-definition map of the factory area), solving the problem of low efficiency in managing a large number of high-definition maps of the factory area in the ROS environment based on Ubuntu20.04.

[0136] HDFS adopts a distributed storage method. Among them, the architecture of HDFS includes a main server Admin, a NameNode, a Second NameNode, and DataNodes. The main server Admin is used to receive the high-definition map of the factory area. Admin connects to the NameNode, the NameNode connects to the sub-node DataNode, and the high-definition map of the factory area is managed distributively through the sub-node DataNode to achieve the physical storage of the high-definition map of the factory area. The Second NameNode connects to the Admin to implement the backup of the high-definition map of the factory area. In addition, when expanding the storage space of HDFS, DataNodes are added as physical data nodes for horizontal expansion.

[0137] Among them, Kafka is used to realize the communication between the Ubuntu operating system of ROS and the CentOS7 operating system of HDFS. Kafka has the advantage that even if an edge (edge node) or fog (fog computing server) stops running due to a fault, it will not affect the message operation. In addition, Kafka allows connections between multiple nodes, facilitating system expansion.

Claims

1. A method for constructing a high-definition map of a factory area based on edge-fog-cloud collaboration, characterized in that, It includes the following steps: S1. Collect the environmental information of an area in the factory area scene, and transmit the environmental information of this area to the fog computing server in this area. Wherein, the environmental information includes the original image and the original point cloud of this area; S2. The fog computing server performs feature extraction, feature transformation, rasterization processing and map post-processing on the received environmental information to obtain a high-definition map of the current area in the factory area scene, and transmits it to the cloud server as a local occupancy grid map through the ROS message mechanism; S3. Repeat steps S1 and S2 until local occupancy grid maps of each area in the entire factory area scene are generated; S4. In the cloud server, splice multiple local occupancy grid maps generated by multiple fog computing servers into an overall occupancy grid map; perform high-level information extraction such as semantic prediction and direction prediction on the overall occupancy grid map to obtain a high-definition map of the factory area.

2. The method for constructing a high-definition map of the factory area according to claim 1, wherein, S2 includes the following sub-steps: S21. Feature extraction is performed on the original image and the original point cloud in the environmental information respectively to obtain all the feature points O of the original image m,n,k and all the feature points O of the original point cloud h,w . The specific steps are as follows: S211. Perform feature extraction on the original image in the environmental information, including: Normalize, scale and crop the original image to adjust the original image to the scale required for convolution operations to obtain a processed image; Perform feature extraction on the processed image through a pre-trained convolutional layer. Wherein, the calculation formula of the convolutional layer is as follows: O m,n,k = ∑ m,n W m,n,k · I i+m,j+n + b k Among them, O m,n,k is the feature point at (i + m, j + n) in the k-th layer of the convolutional layer, W m,n,k is the convolution kernel of the convolutional layer, b k is the bias of the convolutional layer, and I i+m,j+n represents the pixel values of the local region of the processed image centered at (i, j) with a bias of (m, n); According to the feature points O output by the k-th convolutional layer m,n,k to form the feature O of this layer k , the features of all convolutional layers are aggregated to form the corresponding feature map F, that is, the value at the corresponding position in the feature map F comes from the output O of the convolutional layer m,n,k , the feature map F is a three-dimensional tensor, and the dimension of the feature map F is C×H×W. Among them, C is the number of channels, which reflects the number of convolutional kernels, and H and W respectively represent the height and width of the feature map F. By stacking multiple convolutional layers, multi-scale representation of visual features can be achieved; S212, perform feature extraction on the original point cloud in the environmental information based on the PointPillar network to obtain the feature points O in the original point cloud h,w ; S22, transform the coordinates of all feature points O m,n,k and all feature points O h,w to the world coordinate system; S23, generate an empty occupancy grid map occ map ; Map all feature points O m,n,k and all feature points O h,w in the world coordinate system to the empty occupancy grid map occ map to obtain an occupancy grid map occ map '; According to the observed occupancy probabilities of the visual sensor and the Lidar sensor, use the method of probability update to update the occupancy probability of each grid in the occupancy grid map occ map ' to obtain an occupancy grid map occ map ''; S24, post-process the occupied grid map occ map ″ to obtain a high-definition map of the current area in the factory area scene. Preferably, the post-processing includes: Noise filtering: Based on the median value of each grid in the occupancy grid map occ map ″, perform median filtering on each grid to remove isolated noise points or small areas in the grid map, achieve noise elimination, and improve the clarity and reliability of the occupancy grid map; Dilation operation: Expand each grid in the occupancy grid map to its neighborhood and fill the empty grids; Erosion operation: Shrink each grid in the occupancy grid map to its neighborhood and remove the isolated points in the grid; Connectivity analysis: Use the connected component labeling algorithm to identify and process the connected regions in the occupancy grid map occ map ″, and separate the real obstacles and noise regions.

3. The method for constructing a high-definition map of the factory area according to claim 2, characterized in that S23 includes the following sub-steps: S231. Initialize the grid: Set the size M×N of the grid map and the resolution r of the grid map, and initialize the grid map to generate an empty occupancy grid map occ map : occ map = zeros(M,N) Among them, the zeros() function is used to create a matrix of size M×N with all elements being zero, and this matrix is used to represent the initialized empty occupancy grid map occ map ; S232, Feature Point Mapping: Map all the feature points O m,n,k and O h,w in the world coordinate system obtained in S22 to the empty occupancy grid map occ map to obtain the occupancy grid map occ map ′. Among them, if the coordinate of any feature point in the world coordinate system is (x, y), then the mapping formula for the coordinate (u, v) of this feature point in the grid map occ map ′ is as follows: Among them, x min , y min respectively represent the minimum abscissa and the minimum ordinate among all feature points in the world coordinates, and r is the resolution of the occupancy grid map occ map '; S233, Feature Point Fusion: According to the observed occupancy probability of the vision sensor and the observed occupancy probability of the Lidar sensor, the occupancy probability of each grid in the occupancy grid map occ map ′ is updated using a probability update method to obtain the occupancy grid map occ map ″, where the value in each grid of the occupancy grid map occ map ″ is the occupancy probability of the grid, and the occupancy probability represents the likelihood that the grid is occupied by feature points.

4. The method for constructing a high-definition map of the factory area according to claim 1, wherein: In S12, the environmental information is transmitted between the edge node and the fog computing server through the ROS message mechanism based on Ubuntu20.04, and the edge node publishes the environmental information to the fog computing server; The fog computing server subscribes to the environmental information published by the edge node and further processes the received environmental information.

5. The method for constructing a high-definition map of the factory area according to claim 1, wherein: In S212, the steps are as follows: Convert the original point cloud into multiple pillars of equal size, each of which stores local point cloud information. The original point cloud includes N laser points, and each laser point has a corresponding three-dimensional coordinate and a corresponding dimensional feature K. Use the PointNet network to extract local features for each pillar to obtain the feature vector corresponding to each pillar. Use the PointNet network to perform feature aggregation on the feature vector corresponding to each pillar, aggregate the feature vectors corresponding to all pillars into a two-dimensional columnar feature map, and extract high-level features from the two-dimensional columnar feature map through a two-dimensional convolutional neural network. Finally, the feature points in the features of the original point cloud are output as O h,w , where h represents the height index in the two-dimensional columnar feature map, and w represents the width index in the two-dimensional columnar feature map.

6. The method for constructing a high-definition map of a factory area according to claim 1, characterized in that The specific steps of S233 are as follows: (1) Initialize the occupancy probability P(occ map[u,v] ′) = P0, where P0 represents the initial occupancy probability, set as the neutral probability (0.5); (2) Normalize the sum of the observation occupancy probabilities of the visual sensor and the observation occupancy probabilities of the Lidar sensor; (3) Judge each grid in the occupied grid map occ map ′: If there is only one type of sensor feature point in a grid, then the occupancy probability of this grid is the observed occupancy probability of this sensor; if there are two types of sensor feature points in a grid, then update the occupancy probability of each grid through the Bayesian update formula: Among them, P sensor represents the occupancy probability of the observation of the visual sensor, and (1 - P sensor ) represents the occupancy probability of the observation of the Lidar sensor. P(occ map[u,v]sensor ) represents the occupancy probability of the current grid, and P(occ map[u,v] ′) represents the prior occupancy probability of the grid, that is, the initial occupancy probability or the updated occupancy probability.

7. The method for constructing a high-definition map of a factory area according to claim 1, characterized in that S4 includes the following steps: S41. Splicing and merging of local occupancy grid maps: Perform coordinate transformation, splicing and merging on multiple local occupancy grid maps generated by multiple fog computing servers, and splice multiple local occupancy grid maps into an overall occupancy grid map; S42. Perform high-level information extraction such as semantic prediction and direction prediction on the overall occupancy grid map to obtain a high-definition map of the factory area; Preferably, the specific steps of S41 are as follows: Coordinate transformation: Transform each local occupancy grid map to the global coordinate system through translation and rotation, and calculate the pose of each local map in the global coordinate system. Wherein, a point (u′, v′) in a local occupancy grid map is transformed to the global coordinate system, and the transformation formula is: where (x′, y′) are the coordinates of the point (u′, v′) in the global coordinate system, and (x k , y k ) represents the position of the origin of the local occupancy grid map in the local coordinate system in the global coordinate system, θ k represents the pose conversion angle, and the corresponding heading angle θ is obtained by measuring the angular velocity of the driverless vehicle in real time through the vehicle-mounted inertial measurement unit IMU k , and r is the resolution of the local occupancy grid map; Stitching and merging: In the global coordinate system, align each local occupancy grid map according to the common area between them, stitch each local occupancy grid map to obtain an overall occupancy grid map, and then use the method of probability update to merge and update each grid in the overall occupancy grid map, so as to achieve rapid update of the overall occupancy grid map.

8. The method for constructing a high-definition map of the factory area according to claim 7, characterized in that, The specific steps of S42 are as follows: Semantic prediction: Each grid in the overall occupancy grid map is classified through semantic category prediction to determine the semantic category to which each grid belongs. The semantic categories include roads, buildings, pedestrians, and vehicles. Semantic category prediction is to select the category with the highest probability as the semantic category of the grid. The specific formula is as follows: where s u″,v″ is the semantic category of the grid (u″, v″), c is the number of u″,v″ categories of s indicating the probability that the grid (u″, v″) belongs to category c; Direction prediction: Predict the movement direction of the objects with movement ability in each grid of the overall occupancy grid map. For example, the driving direction of a vehicle or the walking direction of a pedestrian.

9. The method for constructing a high-definition map of a factory area according to claim 1, characterized in that The specific steps of S1 are as follows: S11, Take the driverless vehicle in the factory area scene as an edge node, and collect the environmental information of an area in the factory area scene through the vision sensor and Lidar sensor carried by the driverless vehicle to obtain the environmental information of the current area; S12, Transmit the environmental information of the current area from the edge node to the fog computing server corresponding to the current area.

10. The method for constructing a high-definition map of a factory area according to claim 1, wherein The specific steps of S22 are as follows: Transfer all feature points O m,n,k from the image coordinate system to the camera coordinate system. The coordinate transformation formula is as follows: Among them, i + m and j + n are the coordinates of the feature point O m,n,k in the image coordinate system, K is the internal parameter matrix of the camera, and d is the depth value of the camera; all the feature points O m,n,k are mapped and transformed from the camera coordinate system to the vehicle coordinate system; then, through rotation and translation, all the feature points O m,n,k are transformed from the vehicle coordinate system to the world coordinate system; For all feature points O h,w perform coordinate transformation: Transform all feature points O h,w from the Lidar coordinate system to the vehicle coordinate system through rotation and translation, and then from the vehicle coordinate system to the world coordinate system through rotation and translation.

Images