A method for constructing high-definition factory maps based on edge-fog-cloud collaboration

Abstract

This invention discloses a method for constructing a high-definition map of a factory area based on edge-fog-cloud collaboration, comprising: S1, collecting environmental information of a region in a factory scene and transmitting the environmental information to a fog computing server within that region; S2, the fog computing server performing feature extraction, feature transformation, rasterization, and map post-processing on the received environmental information to obtain a high-definition map of the current region in the factory scene, and transmitting it to a cloud server via a ROS message mechanism; S3, repeating steps S1 and S2 until all locally occupied raster maps of the entire factory scene are generated; S4, the cloud server stitching all locally occupied raster maps into a unified occupied raster map; and extracting high-level information such as semantic prediction and direction prediction from the unified occupied raster map to obtain a high-definition map of the factory area. This invention improves the efficiency and accuracy of constructing high-definition factory maps.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method for constructing high-definition maps of factory areas based on edge-fog-cloud collaboration. Background Technology

[0002] High-definition maps, also known as high-resolution maps, are a type of map specifically designed for autonomous driving. Unlike traditional navigation maps, they not only provide road-level navigation information but also lane-level navigation information. In terms of both the richness and accuracy of information, they far surpass traditional navigation maps.

[0003] Industrial parks in enclosed environments offer advantages such as fixed routes and simple operations, facilitating the commercialization of autonomous driving in this context. In this scenario, autonomous vehicles primarily transport goods between warehouses and workshops within the enclosed area, aiming for efficient and accurate approach to loading docks. However, the constantly changing quantity and location of goods in warehouses necessitates a high frequency of map updates.

[0004] Existing single-vehicle SLAM mapping methods based on multi-sensor fusion can only rely on loop closure detection to reduce cumulative errors, resulting in low efficiency and accuracy. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a highly efficient and accurate method for constructing high-definition factory maps based on edge-fog-cloud collaboration.

[0006] Therefore, the present invention adopts the following technical solution:

[0007] A method for constructing a high-definition factory map based on edge-fog-cloud collaboration includes the following steps:

[0008] S1, Collect environmental information of a region in the factory scene and transmit the environmental information of that region to the fog computing server in that region, wherein the environmental information includes the original image and the original point cloud of that region;

[0009] S2, the fog computing server performs feature extraction, feature transformation, rasterization, and map post-processing on the received environmental information to obtain a high-definition map of the current area in the factory scene, and transmits it to the cloud server as a locally occupied raster map via the ROS message mechanism, including:

[0010] S21, feature extraction is performed on the original image and the original point cloud in the environmental information to obtain all feature points O of the original image. m,n,k And all feature points O of the original point cloud h,w The specific steps are as follows:

[0011] S211, extracting features from the original image in the environmental information, including:

[0012] The original image is normalized, scaled, and cropped to adjust it to the scale required for the convolution operation, resulting in the processed image.

[0013] Feature extraction is performed on the processed image using pre-trained convolutional layers, wherein the calculation formula for the convolutional layer is as follows:

[0014] O m,n,k =Σ m,n W m,n,k ·I i+m,j+n +b k

[0015] Among them, O m,n,k W is the feature point at (i+m,j+n) in the k-th layer of the convolutional layer. m,n,k It is the convolution kernel of the convolutional layer, b k It is the bias of the convolutional layer, I i+m,j+n This represents the pixel values ​​of a local region in the processed image centered at (i,j) and biased at (m,n).

[0016] Based on the feature points O output by the k-th convolutional layer m,n,k The features O that make up the k-th layer k The features from all convolutional layers are aggregated to form a corresponding feature map F. That is, the values ​​at corresponding positions in feature map F come from the O values ​​output by the convolutional layers. m,n,k The feature map F is a three-dimensional tensor with dimensions C×H×W, where C is the number of channels, reflecting the number of convolutional kernels, and H and W represent the height and width of the feature map F, respectively. By stacking multiple convolutional layers, a multi-scale representation of visual features is achieved.

[0017] S212, Based on the PointPillar network, feature extraction is performed on the original point cloud in the environmental information to obtain feature points O in the original point cloud. h,w ;

[0018] S22, for all feature points O m,n,k and all feature points O h,w Transform the coordinates to the world coordinate system;

[0019] S23, Generate an empty occupied grid map (occ) map ; All feature points O in the world coordinate system m,n,k and all feature points O h,w Mapped to the empty occupied grid map occ map In the middle, the occupied grid map occ is obtained. mapBased on the occupancy probabilities observed by the visual sensor and LiDAR sensor, a probability update method is used to update the occupancy grid map occupancy (occupancy rate). map The occupancy probability of each grid cell in the map is updated to obtain the occupied grid map (occupancy grid map occupancy). map ";

[0020] S24, for the occupied grid map occ map "After post-processing, a high-resolution map of the current area in the factory scene is obtained;"

[0021] S3, Repeat steps S1 and S2 until a local occupied raster map of each area in the entire factory scene is generated;

[0022] S4, in the cloud server, stitches together multiple local occupancy raster maps generated by multiple fog computing servers into a whole occupancy raster map; performs high-level information extraction such as semantic prediction and direction prediction on the whole occupancy raster map to obtain a high-definition map of the factory area.

[0023] The post-processing includes:

[0024] Noise filtering: based on the occupied grid map OCC map The median value of each grid cell is used to perform median filtering on each grid cell to remove isolated noise points or small areas in the grid map.

[0025] Inflation operation: Expands each cell in the grid map to its neighborhood and fills empty cells;

[0026] Erosion operation: Shrinks each cell in the raster map to its neighborhood, removing isolated points in the raster;

[0027] Connectivity analysis: Using a connected component labeling algorithm to identify occupied grid cells (OCCs) in the map. map The system identifies and processes connected regions to separate real obstacles from noisy areas.

[0028] S23 includes the following sub-steps:

[0029] S231, Initialize the grid:

[0030] Set the raster map size M×N and resolution r, initialize the raster map, and generate an empty raster map occ. map :

[0031] occ map =zeros(M,N)

[0032] The `zeros()` function creates an M×N matrix with all elements equal to zero. This matrix represents the initialized empty-occupy raster map (occ).map ;

[0033] S232, Feature Point Mapping: Mapping all feature points O in the world coordinate system obtained from S22. m,n,k and O h,w Mapped to empty occupied grid map OCC map In the middle, the occupied grid map occ is obtained. map ′, where the coordinates of any feature point in the world coordinate system are (x, y), then the feature point occupies the grid map occ. map The mapping formula for the coordinates (u,v) in the triangle is as follows:

[0034]

[0035] Where, x min y min Let x and y represent the minimum x-coordinate and minimum y-coordinate of all feature points in world coordinates, respectively, and r represent the area occupied by the raster map occupants. map The resolution of r′ is r′ = r;

[0036] S233, Feature Point Fusion: Based on the occupancy probability observed by the visual sensor and the occupancy probability observed by the LiDAR sensor, a probability update method is used to merge the occupancy grid map occupancy points (OCCs). map The occupancy probability of each grid cell in the map is updated to obtain the occupied grid map (occupancy grid map occupancy). map ",wherein, occupying the grid map occ" map The value in each grid cell is the occupancy probability of that cell, which represents the likelihood that the cell will be occupied by a feature point.

[0037] In step S12, environmental information is transmitted between the edge node and the fog computing server through a message mechanism based on ROS 20.04. The edge node publishes environmental information to the fog computing server; the fog computing server subscribes to the environmental information published by the edge node and further processes the received environmental information.

[0038] The steps in S212 are as follows:

[0039] The original point cloud is transformed into multiple pillars of equal size, each containing local point cloud information. The original point cloud includes N laser points, each with corresponding 3D coordinates and a corresponding dimensional feature K′. A PointNet network is used to extract local features from each pillar, resulting in a feature vector for each pillar. The PointNet network is then used to aggregate the feature vectors from all pillars into a 2D histogram feature map. A 2D convolutional neural network is then used to extract high-level features from this histogram feature map, ultimately outputting O features from the original point cloud. h,w Where h represents the height index of the two-dimensional histogram feature map, and w represents the width index of the two-dimensional histogram feature map.

[0040] The specific steps of S233 are as follows:

[0041] (1) Initialize the prior occupancy probability P(occ) of each grid cell. map[u,v] ′)=P0, where P0 represents the initial occupancy probability, set as a neutral probability (0.5);

[0042] (2) Normalize the sum of the observation occupancy probability of the visual sensor and the observation occupancy probability of the Lidar sensor.

[0043] (3) Occupied grid map map For each grid cell in the array, the following criteria are applied: If a grid cell contains feature points from only one sensor, then the occupancy probability of that grid cell is the observation occupancy probability of that sensor; if a grid cell contains feature points from two sensors, then the occupancy probability of each grid cell is updated using the Bayesian update formula.

[0044]

[0045] Among them, P sensor This represents the probability of observations being taken by the visual sensor, (1-P) sensor P(occ) represents the occupancy probability of an observation by a LiDAR sensor. map[u,v]sensor P(occ) represents the occupancy probability of the current grid cell. map[u,v] ′) represents the prior occupancy probability of the grid, that is, the initial occupancy probability or the updated occupancy probability.

[0046] S4 includes the following steps:

[0047] S41, Stitching and merging of partially occupied raster maps: Perform coordinate transformation, stitching and merging of multiple partially occupied raster maps generated by multiple fog computing servers, and stitch the multiple partially occupied raster maps into a whole occupied raster map.

[0048] S42 extracts high-level information such as semantic prediction and direction prediction from the overall occupied grid map to obtain a high-definition map of the factory area.

[0049] The specific steps of S41 are as follows:

[0050] Coordinate transformation: Each locally occupied raster map is transformed to the global coordinate system through translation and rotation. The pose of each local map in the global coordinate system is calculated. Specifically, the transformation formula for a point (u′, v′) in a locally occupied raster map to the global coordinate system is as follows:

[0051]

[0052] Where (x′, y′) are the coordinates of point (u′, v′) in the global coordinate system; (x k ,y k ) represents the position of the origin of the locally occupied raster map in the local coordinate system within the global coordinate system; θ k The pose transformation angle is obtained by measuring the angular velocity of the autonomous vehicle in real time using the onboard inertial measurement unit (IMU); r″ is the resolution of the locally occupied grid map, r″ = r;

[0053] Stitching and merging: In the global coordinate system, each local occupied grid map is aligned according to the common area between them, and then stitched together to obtain a whole occupied grid map. Then, a probabilistic update method is used to merge and update each grid in the whole occupied grid map, thereby achieving rapid update of the whole occupied grid map.

[0054] The specific steps of S42 are as follows:

[0055] Semantic prediction: This involves classifying each grid cell in the overall occupied grid map using semantic category prediction, determining the semantic category of each cell. Semantic categories include roads, buildings, pedestrians, and vehicles. Semantic category prediction selects cells from the grid... The category with the highest probability is taken as the semantic category of the raster, and the specific formula is as follows:

[0056]

[0057] Among them, s u′,v″ Let c be the semantic category of the raster (u″, v″), and s be the semantic category of the raster. u",v″ Number of categories This represents the probability that the grid (u″, v″) belongs to class c;

[0058] Direction prediction: Predicts the movement direction of movable objects in each grid cell of the overall occupied grid map.

[0059] The specific steps of S1 are as follows:

[0060] S11 uses the unmanned vehicles in the factory scene as edge nodes, and collects environmental information of a region in the factory scene through the visual sensors and LiDAR sensors carried by the unmanned vehicles to obtain the environmental information of the current region.

[0061] S12, the environmental information of the current area is transmitted from the edge node to the fog computing server corresponding to the current area.

[0062] The specific steps of S22 are as follows:

[0063] All feature points O m,n,k The coordinate transformation formula from the image coordinate system to the camera coordinate system is as follows:

[0064]

[0065] Where i+m and j+n are feature points O m,n,k In the image coordinate system, K is the camera's intrinsic parameter matrix, and d is the camera's depth value; all feature points O m,n,k After mapping, the coordinate system is transformed from the camera coordinate system to the vehicle coordinate system; then, all feature points O are transformed through rotation and translation. m,n,k Transform from the vehicle coordinate system to the world coordinate system;

[0066] For all feature points O h,w Perform coordinate transformation: transform all feature points O h,w The system is transformed from the Lidar coordinate system to the vehicle coordinate system by rotation and translation, and then transformed from the vehicle coordinate system to the world coordinate system by rotation and translation.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. The high-definition map construction method for the factory area of ​​the present invention adopts an edge-fog-cloud system architecture, and processes the data collected by the two types of sensors in the grid map to complete the construction of the high-definition map of the factory area.

[0069] 2. The high-definition map construction method for the factory area of ​​the present invention allows the cloud (cloud server) to receive data from multiple fog terminals (fog computing servers) and edge terminals (edge ​​nodes), enabling collaborative construction of a high-definition map of the factory area. This not only improves the efficiency of constructing a high-definition map of the factory area but also avoids the cumulative error of long-term SLAM, thereby improving the accuracy of the high-definition map of the factory area.

[0070] 3. This invention deploys Hadoop on a cloud server, enabling convenient access and efficient management of high-definition maps of large-area factory areas under conditions of frequent changes in the factory environment.

[0071] 4. The method of the present invention achieves rapid map updates through feature alignment. Attached Figure Description

[0072] Figure 1 This is a flowchart of an embodiment of the high-definition map construction method for factory areas according to the present invention;

[0073] Figure 2 This is a schematic diagram of the edge-cloud-fog scenario of the present invention. Detailed Implementation

[0074] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0075] See Figure 1 and Figure 2 The present invention provides a method for constructing a high-definition factory map based on edge-fog-cloud collaboration, comprising the following steps:

[0076] S1: Collect environmental information for a specific area within the factory environment and transmit this information to the fog computing server within that area. The specific steps are as follows:

[0077] S11 uses autonomous vehicles in the factory area as edge nodes, and collects environmental information of a region in the factory area through the visual sensors and LiDAR sensors carried by the autonomous vehicles to obtain the environmental information of the current region.

[0078] The environmental information includes the original image of the current area acquired by the visual sensor and the original point cloud of the current area acquired by the LiDAR sensor.

[0079] S12, the environmental information of the current area is transmitted from the edge node (autonomous vehicle) to the fog computing server corresponding to the current area.

[0080] In this system, environmental information is transmitted between edge nodes and the fog computing server via a message mechanism based on ROS (Ubuntu 20.04). Edge nodes publish data (environmental information) to the fog computing server; the fog computing server subscribes to the data (environmental information) published by the edge nodes and further processes the received data (environmental information).

[0081] S2, the fog computing server performs feature extraction, feature transformation, rasterization, and map post-processing on the received environmental information to obtain a high-definition map of the current area in the factory scene, and transmits it to the cloud server as a locally occupied raster map via the ROS message mechanism. The specific steps are as follows:

[0082] S21, feature extraction is performed on the original image and original point cloud in the environmental information, including:

[0083] S211, Feature extraction from the original image in the environmental information:

[0084] The original image is normalized, scaled, and cropped to adjust it to the scale required for the convolution operation, resulting in the processed image. The normalization formula is:

[0085]

[0086] Among them, I norm Let I represent the normalized image, μ represent the original image, and σ represent the standard deviation of the original image.

[0087] Feature extraction is performed on the processed image using pre-trained convolutional layers. The calculation formula for the convolutional layer is as follows:

[0088] O m,n,k =∑ m,n W m,n,k ·I i+m,j+n +b k

[0089] Among them, O m,n,k W is the feature point at (i+m,j+n) in the k-th layer (k=5) of the convolutional layer. m,n,k It is the convolution kernel of the convolutional layer, b k It is the bias of the convolutional layer, I i+m,j+n This represents the pixel values ​​of a local region in the processed image centered at (i,j) and biased at (m,n).

[0090] Based on the feature points O output by the k-th convolutional layer m,n,k The features O that make up the k-th layer k The features from all convolutional layers are aggregated to form a corresponding feature map F. That is, the values ​​at corresponding positions in feature map F come from the O values ​​output by the convolutional layers. m,n,k The feature map F is a three-dimensional tensor with dimensions C×H×W, where C is the number of channels, reflecting the number of convolutional kernels, and H and W represent the height and width of the feature map F, respectively. By stacking multiple convolutional layers, multi-scale representation of visual features is achieved.

[0091] S212, based on the PointPillar network (Lang AH, Vora S, Caesar H, et al. Pointpillars: Fast encoders for object detection from point clouds[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019:12697-12705.), feature extraction is performed on the original point cloud in the environmental information to obtain feature points O in the original point cloud. h,w The specific steps are as follows:

[0092] The original point cloud is transformed into multiple pillars of equal size, each containing local point cloud information. The original point cloud includes N laser points, each with corresponding 3D coordinates and a corresponding dimensional feature K′. A PointNet network is used to extract local features from each pillar, resulting in a feature vector for each pillar. The PointNet network is then used to aggregate the feature vectors from all pillars into a 2D histogram feature map. A 2D convolutional neural network is then used to extract high-level features from this histogram feature map, ultimately outputting O features from the original point cloud. h,w Where h represents the height index of the two-dimensional histogram feature map, and w represents the width index of the two-dimensional histogram feature map.

[0093] The steps S211 and S212 mentioned above are not in any particular order.

[0094] S22, for all feature points O obtained from S211 m,n,k All feature points O obtained from S212 h,w Perform coordinate transformation to the world coordinate system to facilitate subsequent feature fusion and OCC map generation. The specific steps are as follows:

[0095] For all feature points O in S211 m,n,k Perform coordinate transformation:

[0096] All feature points O in S211 m,n,k The coordinate transformation formula from the image coordinate system to the camera coordinate system is as follows:

[0097]

[0098] Where i+m and j+n are feature points O in S211. m,n,kThe coordinates in the image coordinate system, where K is the camera's intrinsic parameter matrix and d is the camera's depth value;

[0099] All feature points O obtained in S211 m,n,k After mapping, the coordinate system is transformed from the camera coordinate system to the vehicle coordinate system; then, all feature points O in S211 are transformed by rotation and translation. m,n,k Transform from the vehicle coordinate system to the world coordinate system.

[0100] For all feature points O in S212 h,w Perform coordinate transformation:

[0101] All feature points O obtained in S212 h,w Transform from LiDAR coordinate system to vehicle coordinate system by rotation and translation, and then transform from vehicle coordinate system to world coordinate system by rotation and translation.

[0102] S23, Generate an empty occupied grid map (occ) map ; All feature points O in S211 m,n,k and all feature points O in S212 h,w Mapped to empty occupied grid map OCC map In the middle, the occupied grid map occ is obtained. map Based on the occupancy probabilities observed by the visual sensor and LiDAR sensor, a probability update method is used to update the occupancy grid map occupancy (occupancy rate). map The occupancy probability of each grid cell in the map is updated to obtain the occupied grid map (occupancy grid map occupancy). map "。Occupy the grid map OCC" map "It accurately reflects the environmental information after multi-sensor fusion. The specific steps are as follows:"

[0103] S231, Initialize the grid:

[0104] Set the raster map size M×N and resolution r, initialize the raster map, and generate an empty raster map occ. map :

[0105] occ map =zeros(M,N)

[0106] The `zeros()` function creates an M×N matrix with all elements equal to zero. This matrix represents the initialized empty-occupy raster map (occ). map ;

[0107] S232, Feature Point Mapping: Mapping all feature points O in S22 under the world coordinate system. m,n,k and O h,w Mapped to empty occupied grid map OCCmap In the middle, the occupied grid map occ is obtained. map ′, where the coordinates of any feature point in the world coordinate system are (x, y), then the feature point occupies the grid map occ. map The mapping formula for the coordinates (u,v) in the triangle is as follows:

[0108]

[0109] Where, x min y min Let r' and r' represent the minimum x-coordinate and minimum y-coordinate of all feature points in world coordinates, respectively, and r' be the area of ​​the occupied raster map occupancy. map The resolution of r′ is r′ = r;

[0110] S233, Feature Point Fusion: Based on the occupancy probability observed by the visual sensor and the occupancy probability observed by the LiDAR sensor, a probability update method is used to merge the occupancy grid map occupancy points (OCCs). map The occupancy probability of each grid cell in the map is updated to obtain the occupied grid map (occupancy grid map occupancy). map ″.

[0111] Among them, the grid map occupancy map The value in each grid cell is the occupancy probability of that cell, representing the likelihood that the cell will be occupied by a feature point. The occupancy probability of each cell is updated using a probability update method to ensure that the occupied grid cell map is occupied (occupied). map "This is more in line with the actual environment. The specific steps are as follows:"

[0112] (1) Initialize the prior occupancy probability P(occ) of each grid cell. map[u,v] ′)=P0, where P0 represents the initial occupancy probability, set as a neutral probability (0.5);

[0113] (2) Normalize the sum of the observation occupancy probability of the visual sensor and the observation occupancy probability of the Lidar sensor.

[0114] (3) Occupied grid map map For each grid cell in the array, the following criteria are applied: If a grid cell contains feature points from only one sensor, then the occupancy probability of that grid cell is the observation occupancy probability of that sensor; if a grid cell contains feature points from two sensors, then the occupancy probability of each grid cell is updated using the Bayesian update formula.

[0115]

[0116] Among them, P sensor This represents the probability of observations being taken by the visual sensor, (1-P) sensorP(occ) represents the occupancy probability of an observation by a LiDAR sensor. map[u,v]sensor P(occ) represents the occupancy probability of the current grid cell. map[u,v] ′) represents the prior occupancy probability of the grid, that is, the initial occupancy probability or the updated occupancy probability.

[0117] S24, for occupying grid map occ map "Post-processing is performed to improve accuracy and usability. This post-processing includes noise filtering, dilation and erosion analysis, and connectivity analysis. The specific steps of the post-processing are as follows:"

[0118] Noise filtering: based on the occupied grid map OCC map The median of each grid cell is used to perform median filtering on each grid cell, removing isolated noise points or small areas (outliers) in the grid map, thereby eliminating noise and improving the clarity and reliability of the grid map.

[0119] Inflation operation: Expands each cell in the grid map to its neighborhood and fills empty cells.

[0120] Erosion operation: Shrinks each cell in the grid map to its neighborhood, removing isolated points in the grid.

[0121] Expansion and erosion operations improve the occupation of the raster map (OCC). map The coherence and continuity of "".

[0122] Connectivity analysis: Using a connected component labeling algorithm to identify occupied grid cells (OCCs) in the map. map The system identifies and processes connected regions to separate real obstacles from noisy areas.

[0123] S3, repeat steps S1 and S2 until a local occupancy raster map of each area in the entire factory scene is generated: the edge node collects environmental information for all different areas in the factory scene, and processes the environmental information of each area in the fog computing server corresponding to each area to generate a local occupancy raster map of each area; the local occupancy raster map generated for each area is transmitted to the cloud server by the fog computing server through the ROS message mechanism.

[0124] S4, within the cloud server, multiple local occupancy raster maps generated by multiple fog computing servers are stitched together into a unified occupancy raster map. High-level information extraction, including semantic and directional prediction, is performed on the unified occupancy raster map to obtain a high-definition map of the factory area. Extracting high-level information allows the unified occupancy raster map to contain more comprehensive semantic information, improving its detail and accuracy. The high-definition map of the factory area is then stored. The specific steps are as follows:

[0125] S41, Stitching and Merging of Partial Occupation Raster Maps: Multiple partial occupation raster maps generated by multiple fog computing servers are transformed, stitched, and merged to form a single overall occupation raster map. The specific steps are as follows:

[0126] Coordinate transformation: Each locally occupied raster map is transformed to the global coordinate system through translation and rotation. The pose of each locally occupied raster map in the global coordinate system is calculated. Specifically, the transformation of a point (u′, v′) in the locally occupied raster map to the global coordinate system is performed using the following formula:

[0127]

[0128] Where (x′, y′) are the coordinates of point (u′, v′) in the global coordinate system; (x k ,y k ) represents the position of the origin of the locally occupied raster map in the local coordinate system within the global coordinate system; θ k The heading angle θ represents the pose transition angle. It is obtained by measuring the angular velocity of the autonomous vehicle in real time using the onboard inertial measurement unit (IMU). k r′′ represents the resolution of the locally occupied raster map, and r′′=r.

[0129] Stitching and merging: In the global coordinate system, each local occupied grid map is aligned according to the common area between them, and then stitched together to obtain a whole occupied grid map. Then, a probabilistic update method (step S233) is used to merge and update each grid in the whole occupied grid map, thereby achieving a fast update of the whole occupied grid map.

[0130] S42, perform high-level information extraction, including semantic prediction and direction prediction, on the overall occupied grid map to obtain a high-definition map of the factory area. The specific steps are as follows: Semantic prediction: Classify each grid cell in the overall occupied grid map using semantic category prediction to determine the semantic category to which each grid cell belongs. Semantic categories include roads, buildings, pedestrians, and vehicles. Semantic category prediction involves selecting cells from the grid... The category with the highest probability is taken as the semantic category of the raster, and the specific formula is as follows:

[0131]

[0132] Among them, s v,v Let c be the semantic category of the raster (u″, v″), and s be the semantic category of the raster. u″,v″ Number of categories This represents the probability that the grid (u″, v″) belongs to class c.

[0133] Direction prediction: Predicts the direction of movement of movable objects in each grid cell of the overall occupied grid map, such as the direction of travel of vehicles or the direction of walking of pedestrians.

[0134] S43, Management and Storage of High-Definition Factory Maps: The obtained high-definition factory maps are transmitted via Kafka to HDFS deployed on CentOS 7 for management and storage. Within the cloud server, HDFS (Hadoop Distributed File System) based on CentOS 7 is used for efficient management and storage of the data (high-definition factory maps), solving the problem of low efficiency in managing large numbers of high-definition factory maps in a ROS environment based on Ubuntu 20.04.

[0135] HDFS employs a distributed storage approach. Its architecture includes a master server (Admin), a NameNode, a Second NameNode, and DataNodes. The master server (Admin) receives high-resolution maps of the factory area. Admin connects to the NameNode, which in turn connects to the DataNodes, enabling distributed management of the high-resolution maps and providing physical storage. The Second NameNode connects to Admin for map backup. Furthermore, when expanding HDFS storage, DataNodes are added as physical data nodes for horizontal scaling.

[0136] Specifically, Kafka is used to enable communication between the ROS Ubuntu operating system and the HDFS CentOS 7 operating system. Kafka has the advantage that even if a single edge node or fog server fails, message delivery will not be affected. Furthermore, Kafka allows for connections between multiple nodes, facilitating system scalability.

Claims

1. A method for constructing a high-definition map of a factory area based on edge-fog-cloud collaboration, characterized in that, Includes the following steps: S1, Collect environmental information of a region in the factory scene and transmit the environmental information of that region to the fog computing server in that region, wherein the environmental information includes the original image and the original point cloud of that region; S2, the fog computing server performs feature extraction, feature transformation, rasterization, and map post-processing on the received environmental information to obtain a high-definition map of the current area in the factory scene, and transmits it to the cloud server as a locally occupied raster map via the ROS message mechanism, including: S21, feature extraction is performed on the original image and the original point cloud in the environmental information to obtain all feature points O of the original image. m,n,k and all feature points O of the original point cloud h,w The specific steps are as follows: S211, extracting features from the original image in the environmental information, including: The original image is normalized, scaled, and cropped to adjust it to the scale required for the convolution operation, resulting in the processed image. Feature extraction is performed on the processed image using pre-trained convolutional layers, where the calculation formula for the convolutional layers is as follows: The m,n,k =S m,n W m,n,k ·I i+m,j+n +b k Among them, O m,n,k W is the feature point at (i+m,j+n) in the k-th layer of the convolutional layer. m,n,k It is the convolution kernel of the convolutional layer, b k It is the bias of the convolutional layer, I i+m,j+n This represents the pixel values ​​of a local region in the processed image centered at (i,j) and biased at (m,n). Based on the feature points O output by the k-th convolutional layer m,n,k The features O that make up the kth layer k The features from all convolutional layers are aggregated to form a corresponding feature map F. That is, the values ​​at corresponding positions in feature map F come from the O values ​​output by the convolutional layers. m,n,k The feature map F is a three-dimensional tensor with dimensions C×H×W, where C is the number of channels, reflecting the number of convolutional kernels, and H and W represent the height and width of the feature map F, respectively. By stacking multiple convolutional layers, a multi-scale representation of visual features is achieved. S212, Based on the PointPillar network, feature extraction is performed on the original point cloud in the environmental information to obtain feature points O in the original point cloud. h,w ; S22, for all feature points O m,n,k and all feature points O h,w Transform the coordinates to the world coordinate system; S23, Generate an empty occupied grid map (occ) map ; All feature points O in the world coordinate system m,n,k and all feature points O h,w Mapped to the empty occupied grid map occ map In the middle, the occupied grid map occ is obtained. map Based on the occupancy probabilities observed by the visual sensor and LiDAR sensor, a probability update method is used to update the occupancy grid map occupancy (occupancy rate). map The occupancy probability of each grid cell in the map is updated to obtain the occupied grid map (occupancy grid map occupancy). map "; S24, for the occupied grid map occ map "After post-processing, a high-resolution map of the current area in the factory scene is obtained;" S3, Repeat steps S1 and S2 until a local occupancy raster map of each area in the entire factory scene is generated; S4, in the cloud server, stitches together multiple local occupancy raster maps generated by multiple fog computing servers into a whole occupancy raster map; performs high-level information extraction such as semantic prediction and direction prediction on the whole occupancy raster map to obtain a high-definition map of the factory area.

2. The method for constructing a high-definition map of a factory area according to claim 1, characterized in that, The post-processing includes: Noise filtering: based on the occupied grid map OCC map The median value of each grid cell is used to perform median filtering on each grid cell to remove isolated noise points or small areas in the grid map. Inflation operation: Expands each cell in the grid map to its neighborhood and fills empty cells; Erosion operation: Shrinks each cell in the raster map to its neighborhood, removing isolated points in the raster; Connectivity analysis: Using a connected component labeling algorithm to identify occupied grid cells (OCCs) in the map. map The system identifies and processes connected regions to separate real obstacles from noisy areas.

3. The method for constructing a high-definition map of a factory area according to claim 1, characterized in that, S23 includes the following steps: S231, Initialize the grid: Set the raster map size M×N and resolution r, initialize the raster map, and generate an empty raster map occ. map : occ map =zeros(M,N) The `zeros()` function creates an M×N matrix with all elements equal to zero. This matrix represents the initialized empty-occupy raster map (occ). map ; S232, Feature Point Mapping: Mapping all feature points O in the world coordinate system obtained from S22. m,n,k and O h,w Mapped to empty occupied grid map OCC map In the middle, the occupied grid map occ is obtained. map ′, where the coordinates of any feature point in the world coordinate system are (x, y), then the feature point occupies the grid map occ. map The mapping formula for the coordinates (u,v) in the triangle is as follows: Where, x min y min Let r' and r' represent the minimum x-coordinate and minimum y-coordinate of all feature points in world coordinates, respectively, and r' be the area of ​​the occupied raster map occupancy. map The resolution of r′ is r′ = r; S233, Feature Point Fusion: Based on the occupancy probability observed by the visual sensor and the occupancy probability observed by the LiDAR sensor, a probability update method is used to merge the occupancy grid map occupancy points (OCCs). map The occupancy probability of each grid cell in the map is updated to obtain the occupied grid map (occupancy grid map occupancy). map ",wherein, occupying the grid map occ" map The value in each grid cell is the occupancy probability of that cell, which represents the likelihood that the cell will be occupied by a feature point.

4. The method for constructing a high-definition map of a factory area according to claim 1, characterized in that: In S12, environmental information is transmitted between edge nodes and fog computing servers through a message mechanism based on ROS 20.

04. Edge nodes publish environmental information to fog computing servers. The fog computing server subscribes to environmental information published by edge nodes and further processes the received environmental information.

5. The method for constructing a high-definition map of a factory area according to claim 1, characterized in that: The steps in S212 are as follows: The original point cloud is transformed into multiple pillars of equal size, each containing local point cloud information. The original point cloud includes N laser points, each with corresponding 3D coordinates and a corresponding dimensional feature K′. A PointNet network is used to extract local features from each pillar, resulting in a feature vector for each pillar. The PointNet network is then used to aggregate the feature vectors from all pillars into a 2D histogram feature map. A 2D convolutional neural network is then used to extract high-level features from this histogram feature map, ultimately outputting O features from the original point cloud. h,w Where h represents the height index of the two-dimensional histogram feature map, and w represents the width index of the two-dimensional histogram feature map.

6. The method for constructing a high-definition map of a factory area according to claim 1, characterized in that, The specific steps for S233 are as follows: (1) Initialize the prior occupancy probability P(occ) of each grid cell. map[u,v] ′)=P0, where P0 represents the initial occupancy probability, set as a neutral probability of 0.5; (2) Normalize the sum of the observation occupancy probability of the visual sensor and the observation occupancy probability of the Lidar sensor. (3) Occupied grid map map For each grid cell in the array, the following criteria are applied: If a grid cell contains feature points from only one type of sensor, then the occupancy probability of that grid cell is the observation occupancy probability of that sensor; if a grid cell contains feature points from two types of sensors, then the occupancy probability of each grid cell is updated using the Bayesian update formula. Among them, P sensor This represents the probability of observations being taken by the visual sensor, (1-P) sensor P(occ) represents the occupancy probability of an observation by a LiDAR sensor. map[u,v]sensor P(occ) represents the occupancy probability of the current grid cell. map[u,v] ′) represents the prior occupancy probability of the grid, that is, the initial occupancy probability or the updated occupancy probability.

7. The method for constructing a high-definition map of a factory area according to claim 1, characterized in that, S4 includes the following steps: S41, Stitching and merging of partially occupied raster maps: Perform coordinate transformation, stitching and merging of multiple partially occupied raster maps generated by multiple fog computing servers, and stitch the multiple partially occupied raster maps into a whole occupied raster map. S42 extracts high-level information such as semantic prediction and direction prediction from the overall occupied grid map to obtain a high-definition map of the factory area. The specific steps of S41 are as follows: Coordinate transformation: Each locally occupied raster map is transformed to the global coordinate system through translation and rotation. The pose of each locally occupied raster map in the global coordinate system is calculated. Specifically, the transformation formula for a point (u′, v′) in a locally occupied raster map to the global coordinate system is as follows: Where (x′, y′) are the coordinates of point (u′, v′) in the global coordinate system; (x k ,y k ) represents the position of the origin of the locally occupied raster map in the local coordinate system within the global coordinate system; θ k The pose transformation angle is obtained by measuring the angular velocity of the autonomous vehicle in real time using the onboard inertial measurement unit (IMU); r″ is the resolution of the locally occupied grid map, r″ = r; Stitching and merging: In the global coordinate system, each local occupied grid map is aligned according to the common area between them, and then stitched together to obtain a whole occupied grid map. Then, a probabilistic update method is used to merge and update each grid in the whole occupied grid map, thereby achieving rapid update of the whole occupied grid map.

8. The method for constructing a high-definition map of a factory area according to claim 7, characterized in that, The specific steps of S42 are as follows: Semantic prediction: This involves classifying each grid cell in the overall occupied grid map using semantic category prediction, determining the semantic category of each cell. Semantic categories include roads, buildings, pedestrians, and vehicles. Semantic category prediction selects cells from the grid... The category with the highest probability is taken as the semantic category of the raster, and the specific formula is as follows: Among them, s u″,v″ Let c be the semantic category of the raster (u″, v″), and s be the semantic category of the raster. u″,v" Number of categories This represents the probability that the grid (u″, v″) belongs to class c; Direction prediction: Predicts the movement direction of movable objects in each grid cell of the overall occupied grid map.

9. The method for constructing a high-definition map of a factory area according to claim 1, characterized in that, The specific steps of S1 are as follows: S11 uses the unmanned vehicles in the factory scene as edge nodes, and collects environmental information of a region in the factory scene through the visual sensors and LiDAR sensors carried by the unmanned vehicles to obtain the environmental information of the current region. S12, the environmental information of the current area is transmitted from the edge node to the fog computing server corresponding to the current area.

10. The method for constructing a high-definition map of a factory area according to claim 1, characterized in that, The specific steps of S22 are as follows: All feature points O m,n,k The coordinate transformation formula from the image coordinate system to the camera coordinate system is as follows: Where i+m and j+n are feature points O m,n,k The coordinates in the image coordinate system, where K is the camera's intrinsic parameter matrix and d is the camera's depth value; All feature points O m,n,k After mapping, the coordinate system is transformed from the camera coordinate system to the vehicle coordinate system; then, all feature points O are transformed through rotation and translation. m,n,k Transform from the vehicle coordinate system to the world coordinate system; For all feature points O h,w Perform coordinate transformation: transform all feature points O h,w The system is transformed from the Lidar coordinate system to the vehicle coordinate system by rotation and translation, and then transformed from the vehicle coordinate system to the world coordinate system by rotation and translation.

Images