Hierarchical method for driverless vehicle task operation passable area extraction

Abstract

The invention discloses a hierarchical method for driverless vehicle task operation passable area extraction. The method comprises the following steps: 1) obtaining an original high-resolution remote sensing image; 2) making the original high-resolution remote sensing image into a road network raster image through a generative adversarial network, and obtaining a road network raster vector result through vectorization; 3) carrying out lightweight feature coding on the road network grid vector result by using a MobileViT architecture to obtain road network information; 4) analyzing the geographic information system vector data to obtain multi-source geographic space vector data association; 5) performing region segmentation on the road network information by applying a bilateral semantic segmentation network Bisenet, and obtaining a historical passable region through small-size object detection; 6) performing real-time obstacle avoidance and path planning by using various sensors on the unmanned vehicle to obtain a real-time path plan; and 7) obtaining a dynamic passable path in combination with the historical passable area and the real-time path planning. The method can be widely applied to autonomous navigation under partial non-road network information conditions when a ground unmanned aerial vehicle mobile platform performs task operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned vehicle geographic information analysis, and particularly relates to a hierarchical method for extracting passable areas for unmanned vehicle task operation. BACKGROUND

[0002] An unmanned vehicle is a self-driving vehicle that realizes autonomous driving without human intervention through artificial intelligence, sensors and advanced algorithms. Its core goal is to improve road safety, optimize traffic efficiency and reshape travel methods. Unmanned vehicles are not only a technological breakthrough, but also a systematic revolution in transportation, cities and society.

[0003] Traditional unmanned vehicles generally only base on road network information or calibrated spatial data to plan paths and judge environments, and then transport goods or people from one place to another. However, with the increasing mobility and load capacity of unmanned platforms, they can also independently undertake specific tasks such as rescue, search, detection, security and obstacle removal, especially in rapidly changing urban environments and unknown outdoor geographic environments. This requires unmanned vehicles to quickly extract the latest road grid from original or outdated road network information, analyze the ground material and slope based on geographic data, and rely on mainstream sensors for real-time local obstacle avoidance, so as to quickly extract the passable area of the unmanned vehicle. On this basis, efficient and automatic path planning algorithms are used to quickly reach the operation area and carry out specific tasks, thus getting rid of the inefficient work mode of real-time judgment and remote control by the last 1 kilometer of the operator. SUMMARY

[0004] The purpose of the present application is to provide a hierarchical method for extracting passable areas for unmanned vehicle task operation, comprising the following steps:

[0005] 1) Obtain a plurality of original remote sensing images, and select images with a spatial resolution greater than a preset value as original high-resolution remote sensing images.

[0006] 2) Generate a road network grid image from the original high-resolution remote sensing image through a generative adversarial network, and vectorize the road network grid image to obtain a road network grid vector result.

[0007] 3) Use the MobileViT architecture to perform lightweight feature coding on the road network grid vector result to obtain road network information.

[0008] 4) Analyze geographic information system vector data to obtain multi-source geographic spatial vector data association.

[0009] 5) On the basis of the multi-source geographic spatial vector data association, apply a bilateral semantic segmentation network Bisenet to regionally segment the road network information, and obtain historical passable areas through small-size object detection.

[0010] 6) Real-time obstacle avoidance and path planning are performed using various sensors on the unmanned vehicle to obtain real-time path planning.

[0011] 7) Dynamic passable paths are obtained by combining historical passable areas and real-time path planning.

[0012] Further, the network structure of the generative adversarial network includes a generator and a discriminator.

[0013] The generator is used to learn and generate virtual road networks with different road attributes, traffic flow data and environmental characteristics.

[0014] The discriminator evaluates the influence of road network features on the generation result by comparing the similarity between the generated virtual road network and the real road network.

[0015] Further, the objective function of the generative adversarial network is as follows:

[0016] G * =min G max D LcGAN(G,D)+βL L1 (G) (1)

[0017]

[0018] L L1 (G)=E x,z,y [ω‖G(x,z)-y‖1] (3)

[0019]

[0020] In the formula, G * represents the optimal generator. G and D represent the generator and the discriminator, respectively. LcGAN(G,D) represents the objective function of the conditional generative adversarial network cGAN. β represents the weight coefficient. L L1 (G) represents the objective function of the pixel pair. x represents the input real image. y represents the real target image corresponding to x. z represents noise. D(x,y) represents the output of the discriminator for the real image pair (x,y). G(x,z) represents the target image generated by the generator with the real image x and the noise z as input. D(x,G(x,z)) represents the output of the discriminator for the real image x and the generated target image G(x,z). E x,y 、E x,z 、 E x,z,y all represent the expectation function. λ represents the penalty coefficient. represents a random interpolation point between the real image x and the generated target image G(x,z). Representation generator to randomly interpolate points Generated image. Representation of gradient function.‖·‖1,‖·‖2 represent norm. ω represents space penalty term. α is a constant greater than 1.

[0021] Further, the MobileViT architecture is stacked by a plurality of MV2 modules and a plurality of MobileViT modules.

[0022] The MV2 module is composed of an inverse residual structure and a linear bottleneck structure, for mapping feature information from a high-dimensional space to a low-dimensional space through a nonlinear function.

[0023] The calculation formula of the MV2 module is as follows:

[0024]

[0025] In the formula, Y represents the output feature of the MV2 module. X represents the input feature. r is the step of deep convolution. Φ 1,p represents the first layer 1×1 convolution. Φ d represents the deep convolution. Φ 2,p represents the second layer 1×1 convolution.

[0026] The MobileViT module captures the input and output relationship at different levels through a multi-head attention mechanism.

[0027] The calculation formula of the MobileViT module is as follows:

[0028]

[0029] In the formula, F out is the output feature of the MobileViT module. f cat represents the fusion function. i represents the self-attention head index. h represents the total number of self-attention heads. represents the output feature of the i-th self-attention head. W O represents the weight matrix. f s represents the softmax function. Q represents the query matrix. represents the weight of the i-th self-attention head query matrix. K represents the index matrix. represents the weight of the i-th self-attention head index matrix. d k represents the dimension of the index matrix K. V represents the content matrix. represents the weight of the i-th self-attention head content matrix.

[0030] Further, the output feature of the MobileViT architecture also combines a pyramid decoder to realize the extraction and fusion of multi-scale features, and obtain road network information.

[0031] Further, the multi-source geospatial vector data association is obtained through adaptive quadtree-based spatial coding.

[0032] The adaptive quadtree-based spatial coding is shown as follows:

[0033] a1 spatially encodes the point entity.

[0034] a11 takes the current spatial region as the region to be encoded and initializes the encoding of the point entity.

[0035] a12 quadtree-grid divides the region to be encoded to obtain four grids and encodes in the counterclockwise direction.

[0036] a13 judges the position of the point entity in the four grids and adds the encoding of the grid where the point entity is located in front of the encoding of the point entity.

[0037] a14 judges whether the ending condition is met, if yes, the spatial encoding of the point entity is obtained, if not, the grid where the point entity is located is taken as the region to be encoded and step a12 is returned.

[0038] The ending condition includes reaching the preset division times.

[0039] a2 spatially encodes the line entity and the surface entity.

[0040] a21 adaptively grid divides the current spatial region, calculates the row and column numbers of the line entity or the surface entity in the grid, and determines the encoding length.

[0041] a22 determines the encoding bit number based on the row and column numbers of the line entity or the surface entity in the grid, as shown below:

[0042] 2 m <min(r,c)<2 m+1 (7)

[0043]

[0044] In the formula, r and c are respectively the row number and the column number of the line entity or the surface entity in the grid. m is the encoding bit number. m0 is the encoding length.

[0045] a23 spatially encodes the point entity of the left lower corner of the line entity or the surface entity to obtain m-bit encoding.

[0046] a24 calculates the constraint encoding and adds the constraint encoding in front of the m-bit encoding to obtain m+1-bit encoding.

[0047] The constraint encoding is encoded according to the distribution of the line entity or the surface entity in the grid.

[0048] a25 fill 0 in front of m+1th encoding until m0th encoding, get spatial encoding of line entity or surface entity.

[0049] a3 get multi-source geospatial vector data association by comparing valid encodings in spatial encodings of two different entities.

[0050] The valid encodings do not include constraint encodings.

[0051] If all the valid encodings are equal, the two different entities are related.

[0052] If only the first valid encoding is not equal, the two different entities are adjacent.

[0053] If there are other encodings in the valid encodings that are not equal except the first encoding, the two different entities are apart.

[0054] Further, the application of the Bisenet bilateral semantic segmentation network to the road network information region segmentation, and through small size object detection, the steps of obtaining the historical passable area are as follows:

[0055] 5.1) Use three convolutional layers with a step size of 2 and down-sampling to reduce the resolution of the road network information, and extract shallow features.

[0056] The shallow features include the edge lines and textures of the outdoor roads.

[0057] 5.2) Use the restnet18 residual network as the backbone to down-sample the road network information by different multiples, and use the global average pooling module and the attention mechanism module to obtain the context information of the road network information.

[0058] 5.3) Through the multi-feature fusion module, the shallow features and the context information are fused to realize the initial segmentation of the passable area.

[0059] 5.4) Use clustering method to detect small size objects in the initial segmented passable area, realize secondary segmentation of the passable area, and obtain the historical passable area.

[0060] Further, the clustering method includes a C-means clustering algorithm.

[0061] The steps of using the C-means clustering algorithm to detect small size objects in the initial segmented passable area are as follows:

[0062] 5.4.1) Set the number of clusters and the iteration convergence condition, and initialize each cluster center.

[0063] 5.4.2) Calculate the membership value using the current cluster center.

[0064] 5.4.3) judging whether the membership value of each data point to all cluster centers reaches a stable state, if yes, entering step 5.4.4), if not, returning to step 5.4.2).

[0065] 5.4.4) updating the cluster centers by using the membership value.

[0066] 5.4.5) judging whether the iteration convergence condition is reached, if yes, obtaining each cluster center and the membership value, if not, returning to step 5.4.2).

[0067] Further, the sensors on the unmanned vehicle include a laser radar, a millimeter wave radar, a GPS, and a camera.

[0068] The laser radar is configured to receive laser radar point cloud data.

[0069] The millimeter wave radar is configured to receive millimeter wave radar data.

[0070] The GPS is configured to receive positioning navigation data.

[0071] The camera is configured to receive video image data.

[0072] The step of obtaining the real-time path planning is as follows:

[0073] 6.1) receiving road environment information by each type of sensor on the unmanned vehicle.

[0074] 6.2) performing feature extraction on the received road environment information, and performing fusion optimization by an IMU pre-integration factor, a GPS factor, and a loop closing factor to obtain multi-sensor fusion data.

[0075] 6.3) constructing a three-dimensional point cloud map and a two-dimensional grid map by using the multi-sensor fusion data.

[0076] The three-dimensional point cloud map is configured to display surrounding environment information.

[0077] 6.4) performing global path planning on the two-dimensional grid map by using an improved A* algorithm with dynamic weights.

[0078] The improved A* algorithm with dynamic weights is as follows:

[0079] f(n) = g(n) + ω(n)h(n) (9)

[0080] In the formula, ω(n) is a weight coefficient, f(n) represents an estimated total cost from the starting point to the end point through node n, g(n) represents an estimated cost from the starting point to node n, and h(n) represents an estimated cost from node n to the end point.

[0081] 6.5) The planned path is segmented by turning nodes and the trajectory is smoothed by using Bezier curve.

[0082] The calculation formula for smoothing the trajectory by using Bezier curve is as follows:

[0083]

[0084] In the formula, u is a free variable. P(u) represents the smoothed trajectory. P(i0) is the motion control point of the unmanned vehicle. i0 represents the motion control point index. n0 represents the total number of motion control points. is the Bernstein polynomial expression.

[0085] 6.6) The segmented and smoothed path is fused and spliced to obtain the final real-time path planning.

[0086] Further, the determination formula of the passable area in the dynamic passable path is as follows:

[0087]

[0088]

[0089] In the formula, represents the passable area determination result. i1 represents the non-overlapping area pixel category index. j1 represents the passable area source index. represents the non-overlapping area pixel category, and c1 and c2 represent the passable area and the impassable area, respectively. represents the passable area source, and sc1, sc2 and sc3 represent the historical passable area source, the visual passable area source and the radar passable area source, respectively. represents the passable area source sc j recognizes the non-overlapping area pixel p as c i represents the weight of the non-overlapping area pixel category. Region h , Region c , Region l represent the areas of the historical passable area, the visual passable area and the radar passable area, respectively. p represents the non-overlapping area pixel. represents the passable area source sc j recognizes the non-overlapping area pixel p as c i represents the probability of the non-overlapping area pixel category. represents the binary classification probability of the non-overlapping area pixel p.

[0090] The technical effect of the present application is self-evident, and the present application can enable an unmanned vehicle to generate a passable area for dynamic and accurate path planning based on a road network, geographic data and its own sensor information when performing special operations such as rescue, search, detection, security and barrier breaking. The present application can be widely applied to the autonomous navigation of a ground unmanned mobile platform under partial non-road network information conditions when performing a task.

[0091] The present application solves the technical problem of passable area extraction of an unmanned vehicle in a complex unstructured off-road scene, has outstanding real-time and accuracy, and the hierarchical design can effectively improve the robustness of passable area detection and significantly improve the consistency with the actual road configuration, and has certain engineering application value. BRIEF DESCRIPTION OF DRAWINGS

[0092] Figure 1 The flowchart of the present application is shown in the figure;

[0093] Figure 2 The road network grid vector extraction schematic diagram is shown in the figure;

[0094] Figure 3 The road network extraction model RoadViT structure schematic diagram is shown in the figure;

[0095] Figure 4 The line (surface) entity space encoding flowchart is shown in the figure;

[0096] Figure 5 The Bisenet semantic segmentation framework diagram is shown in the figure;

[0097] Figure 6 The overall block diagram of the autonomous navigation system of the unmanned vehicle is shown in the figure. DETAILED DESCRIPTION

[0098] The present application will be further described below in conjunction with the embodiments, but should not be understood as limiting the above-mentioned subject matter of the present application to the following embodiments. Various substitutions and modifications can be made according to ordinary technical knowledge and conventional means in the art without departing from the above-mentioned technical idea of the present application, and all should be included in the protection scope of the present application.

[0099] Example 1

[0100] Referring to Figures 1 to 6 A hierarchical method for passable area extraction for unmanned vehicle task operation includes the following steps:

[0101] 1) Obtain a plurality of original remote sensing images, and select images with a spatial resolution greater than a preset value from the original high-resolution remote sensing images.

[0102] 2) The original high-resolution remote sensing image is made into a road network grid image by a generative adversarial network, and the road network grid image is vectorized to obtain a road network grid vector result.

[0103] 3) Utilize MobileViT architecture to encode the road network grid vector results with lightweight features, obtaining road network information.

[0104] 4) Obtain multi-source geographic spatial vector data association by analyzing geographic information system vector data.

[0105] 5) On the basis of multi-source geographic spatial vector data association, apply the bilateral semantic segmentation network Bisenet to regional segmentation of road network information, and obtain historical passable areas through small-size object detection.

[0106] 6) Use various sensors on the unmanned vehicle for real-time obstacle avoidance and path planning, obtaining real-time path planning.

[0107] 7) Combine historical passable areas and real-time path planning to obtain dynamic passable paths.

[0108] Embodiment 2:

[0109] A hierarchical method for extracting passable areas for unmanned vehicle task operation, the main technical content is seen in embodiment 1, further, the network structure of the generative adversarial network includes a generator and a discriminator.

[0110] The generator is used to learn and generate virtual road networks with different road attributes, traffic flow data and environmental characteristics.

[0111] The discriminator compares the similarity between the generated virtual road network and the real road network to evaluate the influence of road network features on the generation result.

[0112] Embodiment 3:

[0113] A hierarchical method for extracting passable areas for unmanned vehicle task operation, the main technical content is seen in any one of embodiments 1 to 2, further, the objective function of the generative adversarial network is as follows:

[0114] G * = min G max D LcGAN(G,D)+βL L1 (G) (1)

[0115]

[0116] L L1 (G)=E x,z,y [ω‖G(x,z)-y‖1] (3)

[0117]

[0118] In the formula, G* represents the optimal generator. G and D represent the generator and the discriminator, respectively. LcGAN(G, D) represents the objective function of the conditional generative adversarial network cGAN. β represents the weight coefficient. L L1 (G) represents the objective function of the pixel pair. x represents the input real image. y represents the real target image corresponding to x. z represents the noise. D(x, y) represents the output of the discriminator for the real image pair (x, y). G(x, z) represents the target image generated by the generator with the real image x and the noise z as input. D(x, G(x, z)) represents the output of the discriminator for the real image x and the generated target image G(x, z). E x,y , E x,z , E x,z,y represents the expected function. λ represents the penalty coefficient. represents the random interpolation point between the real image x and the generated target image G(x, z). represents the generated image by the generator with the random interpolation point . represents the gradient function. ‖·‖1 and ‖·‖2 represent the norm. ω represents the spatial penalty term. α is a constant greater than 1.

[0119] Embodiment 4:

[0120] A hierarchical method for unmanned vehicle task operation passable area extraction, the main technical content is any one of embodiments 1 to 3, further, the MobileViT architecture is stacked by a plurality of MV2 modules and a plurality of MobileViT modules.

[0121] The MV2 module is composed of an inverse residual structure and a linear bottleneck structure, which is used to map the feature information from a high-dimensional space to a low-dimensional space through a nonlinear function.

[0122] The calculation formula of the MV2 module is as follows:

[0123]

[0124] In the formula, Y represents the output feature of the MV2 module. X represents the input feature. r is the step of deep convolution. Φ 1,p represents the first layer 1×1 convolution. Φ d represents the deep convolution. Φ 2,p represents the second layer 1×1 convolution.

[0125] The MobileViT module captures the input and output relationship of different levels through a multi-head attention mechanism.

[0126] The calculation formula of the MobileViT module is as follows:

[0127]

[0128] In the formula, F out is the output feature of the MobileViT module. f cat represents a fusion function. i represents a self-attention head index. h represents the total number of self-attention heads. represents the output feature of the i-th self-attention head. W O represents a weight matrix. f s represents a softmax function. Q represents a query matrix. represents the weight of the i-th self-attention head query matrix. K represents an index matrix. represents the weight of the i-th self-attention head index matrix. d k represents the dimension of the index matrix K. V represents a content matrix. represents the weight of the i-th self-attention head content matrix.

[0129] Embodiment 5:

[0130] A hierarchical method for extracting passable areas for unmanned vehicle task operation, the main technical content of which is any one of embodiments 1 to 4, further, the output feature of the MobileViT architecture also combines a pyramid decoder to realize multi-scale feature extraction and fusion, and obtain road network information.

[0131] Embodiment 6:

[0132] A hierarchical method for extracting passable areas for unmanned vehicle task operation, the main technical content of which is any one of embodiments 1 to 5, further, the multi-source geospatial vector data association is obtained through adaptive quadtree-based spatial coding.

[0133] The steps of the adaptive quadtree-based spatial coding are as follows:

[0134] a1 encodes the point entity in space.

[0135] a11 takes the current spatial area as the to-be-encoded area and initializes the encoding of the point entity.

[0136] a12 performs quadtree grid division on the to-be-encoded area to obtain four grids, and encodes along the counterclockwise direction.

[0137] a13 judges the position of the point entity in the four grids, and adds the encoding of the grid where the point entity is located in front of the encoding of the point entity.

[0138] a14 judges whether the end condition is met, if yes, the spatial coding of the point entity is obtained, if not, the grid where the point entity is located is taken as the to-be-encoded area, and step a12 is returned.

[0139] The end condition comprises reaching a preset number of divisions.

[0140] a2 encodes a line entity or a surface entity in space.

[0141] a21 performs adaptive grid division on a current space region, calculates the number of rows and columns of the line entity or the surface entity in the grid, and determines the encoding length.

[0142] a22 determines the number of encoding bits based on the number of rows and columns of the line entity or the surface entity in the grid, as shown below:

[0143] 2 m <min(r,c)<2 m+1 (7)

[0144]

[0145] In the formula, r and c are respectively the number of rows and columns of the line entity or the surface entity in the grid. m is the number of encoding bits. m0 is the encoding length.

[0146] a23 encodes a point entity of a point at the lower left corner of the line entity or the surface entity in space to obtain m-bit encoding.

[0147] a24 calculates constraint encoding and adds the constraint encoding in front of the m-bit encoding to obtain m+1-bit encoding.

[0148] The constraint encoding is encoded according to the distribution of the line entity or the surface entity in the grid.

[0149] a25 fills 0 in front of the m+1-bit encoding until m0-bit encoding is obtained to obtain the space encoding of the line entity or the surface entity.

[0150] a3 obtains the correlation of multi-source geographic spatial vector data by comparing the effective encoding in the space encoding of two different entities.

[0151] The effective encoding does not include the constraint encoding.

[0152] If the effective encoding is all equal, the two different entities are related.

[0153] If only the first encoding in the effective encoding is not equal, the two different entities are adjacent.

[0154] If, in the effective encoding, in addition to the first encoding, there is other encoding that is not equal, the two different entities are apart.

[0155] Embodiment 7:

[0156] A hierarchical method for extracting passable area for unmanned vehicle task operation, the main technical content is seen in any one of embodiments 1 to 6, further, the step of using the bilateral semantic segmentation network Bisenet to segment the road network information and obtaining the historical passable area through small size object detection is as follows:

[0157] 5.1) Use three convolutional layers with a step size of 2 and down-sampling to reduce the resolution of the road network information and extract shallow features.

[0158] The shallow features include the edge lines and textures of the outdoor roads.

[0159] 5.2) Use the restnet18 residual network as the backbone to down-sample the road network information by different multiples, and use the global average pooling module and the attention mechanism module to obtain the context information of the road network information.

[0160] 5.3) Fuse the shallow features and the context information through a multi-feature fusion module to realize the initial segmentation of the passable area.

[0161] 5.4) Use a clustering method to detect small size objects in the initially segmented passable area to realize the secondary segmentation of the passable area and obtain the historical passable area.

[0162] The small size here depends on the range of entity sizes of the specific scene and is defined according to the actual situation.

[0163] Embodiment 8:

[0164] A hierarchical method for extracting passable area for unmanned vehicle task operation, the main technical content is seen in any one of embodiments 1 to 7, further, the clustering method includes a C-means clustering algorithm.

[0165] The step of using the C-means clustering algorithm to detect small size objects in the initially segmented passable area is as follows:

[0166] 5.4.1) Set the number of clusters and the iteration convergence condition, and initialize each cluster center.

[0167] 5.4.2) Calculate the membership value using the current cluster center.

[0168] 5.4.3) Determine whether the membership value of each data point to all cluster centers reaches a stable state, if yes, go to step 5.4.4), if no, return to step 5.4.2).

[0169] 5.4.4) Update the cluster center using the membership value.

[0170] 5.4.5) judge whether the iteration convergence condition is reached, if yes, obtain each cluster center and membership value, if no, return to step 5.4.2).

[0171] Embodiment 9:

[0172] A hierarchical method for unmanned vehicle task operation passable area extraction, the main technical content is seen in any one of embodiments 1 to 8, further, the sensors on the unmanned vehicle include lidar, millimeter wave radar, GPS, camera.

[0173] The lidar is used for receiving lidar point cloud data.

[0174] The millimeter wave radar is used for receiving millimeter wave radar data.

[0175] The GPS is used for receiving positioning navigation data.

[0176] The camera is used for receiving video image data.

[0177] The step of obtaining real-time path planning is as follows:

[0178] 6.1) receive road environment information by various sensors on the unmanned vehicle.

[0179] 6.2) extract features from the received road environment information, and fuse and optimize through IMU pre-integration factor, GPS factor, loop closing factor to obtain multi-sensor fusion data.

[0180] 6.3) construct a three-dimensional point cloud map and a two-dimensional grid map using the multi-sensor fusion data.

[0181] The three-dimensional point cloud map is used for displaying surrounding environment information.

[0182] 6.4) use an improved A* algorithm with dynamic weight to perform global path planning on the two-dimensional grid map.

[0183] The improved A* algorithm with dynamic weight is as follows:

[0184] f(n)=g(n)+ω(n)h(n) (9)

[0185] In the formula, ω(n) is a weight coefficient. f(n) represents the estimated total cost from the starting point to the end point through node n. g(n) represents the estimated cost from the starting point to node n. h(n) represents the estimated cost from node n to the end point.

[0186] 6.5) segment trajectory smoothing optimization according to the turning nodes of the planned path using a Bezier curve.

[0187] The calculation formula for smoothing and optimizing the trajectory using the Bezier curve is as follows:

[0188]

[0189] In the formula, u is a free variable. P(u) represents the smoothed trajectory. P(i0) is the motion control point of the unmanned vehicle. i0 represents the motion control point index. n0 represents the total number of motion control points. is the Bernstein polynomial expression.

[0190] 6.6) Fuse and splice the segmented and smoothed path to obtain the final real-time path planning.

[0191] Embodiment 10:

[0192] A hierarchical method for extracting the passable area of the unmanned vehicle task operation, the main technical content is any one of embodiments 1 to 9, further, the determination formula of the passable area in the dynamic passable path is as follows:

[0193]

[0194] In the formula, represents the passable area determination result. i1 represents the non-overlapping area pixel category index. j1 represents the passable area source index. represents the non-overlapping area pixel category, c1 and c2 respectively represent the passable area and the impassable area. represents the passable area source, sc1, sc2 and sc3 respectively represent the historical passable area source, the visual passable area source and the radar passable area source. represents the passable area source sc j recognize the non-overlapping area pixel p as c i The weight of the non-overlapping area pixel category. Region h , Region c , Region l respectively represent the area of the historical passable area, the visual passable area and the radar passable area. p represents the non-overlapping area pixel. represents the passable area source sc j recognize the non-overlapping area pixel p as c i The probability of the non-overlapping area pixel category. represents the binary classification probability of the non-overlapping area pixel p.

[0195] Embodiment 11:

[0196] Referring to Figures 1 to 6 , a hierarchical method for extracting the passable area of the unmanned vehicle task operation, comprising the following steps:

[0197] 1) Obtain a plurality of original remote sensing images, and select images with a spatial resolution greater than a preset value from the original remote sensing images as original high-resolution remote sensing images.

[0198] 2) A generative adversarial network (GAN) is used to generate a road network raster image from the original high-resolution remote sensing images, and the road network raster image is vectorized to obtain a road network raster vector result.

[0199] 3) A MobileViT architecture combining a convolutional neural network (CNN) and a Transformer is used to perform lightweight feature coding on the road network raster vector result to obtain road network information.

[0200] 4) By analyzing geographic information system vector data, the correlation and interaction between discrete, multi-source, and heterogeneous spatial data resources are mined, and multi-source geographic spatial vector data correlation is obtained based on adaptive quadtree spatial coding. The spatial position, spatial form, semantics, attributes, and other characteristics of geographic entities are analyzed to mine the relationship and mutual influence between geographic data, describe the association, constraint, and action relationship between surface features, and provide deep technical support for the drivable area data of the unmanned vehicle.

[0201] The data sources of the geographic information system are diversified, including satellite remote sensing monitoring data, government and scientific research institution open data, and GPS measurement data.

[0202] 5) Based on the multi-source geographic spatial vector data correlation, a bilateral semantic segmentation network (Bisenet) is applied to segment the road network information into four types of regions: drivable area, difficult-to-drive area, non-drivable area, and sky, and small-size object detection is performed to obtain historical drivable areas.

[0203] 6) Real-time obstacle avoidance and path planning are performed using various sensors on the unmanned vehicle to obtain real-time path planning.

[0204] In the drivable area, the unmanned vehicle uses monocular / dual-camera, 3D laser radar, millimeter wave radar, GPS / IMU, and other sensors to perceive dynamic information in the environment, uses a multi-sensor fusion LIO-SAM algorithm to realize real-time positioning of the unmanned vehicle and accurate mapping of the surrounding environment, then in the autonomous navigation stage, a global path planning using an improved A* algorithm and a local path planning using DWA are combined, and a Bézier curve is used to optimize the motion trajectory, and a laser radar and millimeter wave radar fusion obstacle avoidance strategy based on Kalman filtering is used to realize real-time obstacle avoidance.

[0205] 7) The dynamic drivable path is obtained by combining the historical drivable area and the real-time path planning.

[0206] To ensure the robustness of the passable area detection in unknown unstructured environments and the consistency with the actual road form, a multi-modal information fusion passable area verification method based on strong constraints of road topology is used, historical passable areas, visual passable area detection and radar-based passable area detection are used as three main information sources, and the Bayesian posterior estimation principle is used to realize multi-source information fusion, which can maintain consistency with the road form and has reliability, and is beneficial to the efficient autonomous navigation of unmanned vehicles in unknown unstructured scenes.

[0207] Embodiment 12:

[0208] A hierarchical method for unmanned vehicle task operation passable area extraction, the main technical content is seen in embodiment 11, further, the network structure of the generative adversarial network includes a generator and a discriminator.

[0209] The generator is used to learn and generate virtual road networks with different road attributes, traffic flow data and environmental characteristics.

[0210] The discriminator compares the similarity between the generated virtual road network and the real road network to evaluate the influence of road network features on the generation result.

[0211] The road network topology is input to the GAN generator. The generator can learn and generate virtual road networks similar to the real road topology, and the road attributes and road surrounding environment characteristics are input to the GAN generator as conditional inputs to generate virtual road networks with different road attributes, traffic flow data and environmental characteristics. In the training process of the GAN model, a suitable discriminator network structure is designed to effectively distinguish the differences between the generated virtual road network and the real road network. By comparing the similarity between the generated virtual road network and the real road network, the influence of road network features on the generation result is evaluated. The features with higher importance are selected for training of the GAN model.

[0212] Embodiment 13:

[0213] A hierarchical method for unmanned vehicle task operation passable area extraction, the main technical content is seen in any one of embodiments 11 to 12, further, the objective function of the generative adversarial network is as follows:

[0214] G * =min G max D LcGAN(G,D)+βL L1 (G) (1)

[0215]

[0216] L L1(G) = E x,z,y [ω‖G(x,z)-y‖1] (3)

[0217]

[0218] where G * denotes the optimal generator. G, D denote the generator, discriminator, respectively. LcGAN(G, D) denotes the objective function of the conditional generative adversarial network cGAN. β denotes the weight coefficient. L L1 (G) denotes the objective function of the pixel pair. x denotes the input real image. y denotes the real target image corresponding to x. z denotes the noise. D(x, y) denotes the output of the discriminator on the real image pair (x, y). G(x, z) denotes the target image generated by the generator with the real image x and the noise z as input. D(x, G(x, z)) denotes the output of the discriminator on the real image x and the generated target image G(x, z). E x,y , E x,z , E x,z,y denote the expectation function. λ denotes the penalty coefficient. denotes the random interpolation point between the real image x and the generated target image G(x, z). denotes the generated image by the generator with the random interpolation point as input. denotes the gradient function. ‖·‖1, ‖·‖2 both denote the norm. ω denotes the space penalty term. α is a constant greater than 1.

[0219] Embodiment 14:

[0220] A hierarchical method for unmanned vehicle task operation passable area extraction, the main technical content is seen in any one of embodiments 11 to 13, further, the MobileViT architecture is stacked by a plurality of MV2 modules and a plurality of MobileViT modules.

[0221] The MV2 module is composed of an inverse residual structure and a linear bottleneck structure, used for mapping feature information from a high-dimensional space to a low-dimensional space through a nonlinear function.

[0222] The calculation formula of the MV2 module is as follows:

[0223]

[0224] where Y denotes the output feature of the MV2 module. X denotes the input feature. r is the step of the depth convolution. Φ 1,p denotes the first layer 1×1 convolution. Φ d denotes the depth convolution. Φ 2,p denotes the second layer 1×1 convolution.

[0225] The MobileViT module captures the input and output relationships at different levels through a multi-head attention mechanism.

[0226] The calculation formula of the MobileViT module is as follows:

[0227]

[0228] In the formula, F out is the output feature of the MobileViT module. f cat represents a fusion function. i represents the index of the self-attention head. h represents the total number of self-attention heads. represents the output feature of the i-th self-attention head. W O represents a weight matrix. f s represents a softmax function. Q represents a query matrix. represents the weight of the i-th self-attention head query matrix. K represents an index matrix. represents the weight of the i-th self-attention head index matrix. d k represents the dimension of the index matrix K. V represents a content matrix. represents the weight of the i-th self-attention head content matrix.

[0229] Embodiment 15:

[0230] A hierarchical method for extracting passable areas for unmanned vehicle task operation, the main technical content of which is seen in any one of embodiments 11 to 14, further, the output feature of the MobileViT architecture also combines a pyramid decoder to realize multi-scale feature extraction and fusion, and obtain road network information.

[0231] Embodiment 16:

[0232] A hierarchical method for extracting passable areas for unmanned vehicle task operation, the main technical content of which is seen in any one of embodiments 11 to 15, further, the multi-source geospatial vector data association is obtained through spatial coding based on an adaptive quadtree.

[0233] The steps of the spatial coding based on the adaptive quadtree are as follows:

[0234] a1 performs spatial coding on point entities.

[0235] a11 takes the current spatial area as the to-be-coded area and initializes the coding of the point entity.

[0236] a12 performs quadtree grid division on the to-be-coded area to obtain four grids, and encodes along the counterclockwise direction.

[0237] a13 judges the position of the point entity in the four grids and adds the code of the grid where the point entity is located in front of the code of the point entity.

[0238] a14 judges whether the end condition is met, if yes, the spatial code of the point entity is obtained, if not, the grid where the point entity is located is taken as the to-be-coded area, and step a12 is returned.

[0239] The end condition includes reaching a preset division times.

[0240] a2 performs spatial coding on the line entity and the surface entity.

[0241] a21 performs adaptive grid division on the current spatial area, calculates the number of rows and columns of the line entity or the surface entity in the grid, and determines the coding length.

[0242] a22 determines the number of coding bits based on the number of rows and columns of the line entity or the surface entity in the grid, as follows:

[0243] 2 m <min(r,c)<2 m+1 (7)

[0244]

[0245] In the formula, r and c are the number of rows and columns of the line entity or the surface entity in the grid respectively. m is the number of coding bits. m0 is the coding length.

[0246] a23 performs spatial coding on the point entity of the left lower corner of the line entity or the surface entity to obtain m-bit coding.

[0247] a24 calculates the constraint coding and adds the constraint coding in front of the m-bit coding to obtain m+1-bit coding.

[0248] The constraint coding is coded according to the distribution of the line entity or the surface entity in the grid.

[0249] a25 fills 0 in front of the m+1-bit coding until m0-bit coding is reached to obtain the spatial coding of the line entity or the surface entity.

[0250] a3 obtains the correlation of multi-source geospatial vector data by comparing the effective coding in the spatial coding of two different entities.

[0251] The effective coding does not include the constraint coding.

[0252] If the effective coding is all equal, the two different entities are related.

[0253] If only the first coding of the effective coding is not equal, the two different entities are adjacent.

[0254] If there are other encodings unequal to the foremost encoding in the valid encoding, then the two different entities are apart.

[0255] Embodiment 17:

[0256] A hierarchical method for unmanned vehicle task operation passable area extraction, the main technical content of which is seen in any one of embodiments 11 to 16, further, the steps of applying the Bisenet bilateral semantic segmentation network to road network information for area segmentation and obtaining the historical passable area through small-size object detection are as follows:

[0257] 5.1) Reduce the resolution of the road network information by using three convolutional layers with a step size of 2 and downsampling, and extract the shallow features.

[0258] The shallow features include the edge lines and textures of the outdoor roads.

[0259] 5.2) Use the restnet18 residual network as the backbone to downsample the road network information by different multiples, and use the global average pooling module and the attention mechanism module to obtain the context information of the road network information.

[0260] 5.3) Fuse the shallow features and the context information through a multi-feature fusion module to realize the initial segmentation of the passable area.

[0261] 5.4) Use a clustering method to detect small-size objects in the initially segmented passable area to realize the secondary segmentation of the passable area and obtain the historical passable area.

[0262] Embodiment 18:

[0263] A hierarchical method for unmanned vehicle task operation passable area extraction, the main technical content of which is seen in any one of embodiments 11 to 17, further, the clustering method includes a C-means clustering algorithm.

[0264] The steps of using the C-means clustering algorithm to detect small-size objects in the initially segmented passable area are as follows:

[0265] 5.4.1) Set the number of clusters and the iteration convergence condition, and initialize each cluster center.

[0266] 5.4.2) Calculate the membership value using the current cluster center.

[0267] 5.4.3) Determine whether the membership value of each data point to all cluster centers reaches a stable state, if yes, go to step 5.4.4), if no, return to step 5.4.2).

[0268] 5.4.4) Update the cluster center using the membership value.

[0269] 5.4.5) judge whether the iteration convergence condition is reached, if yes, obtain each cluster center and membership value, if no, return to step 5.4.2).

[0270] Embodiment 19:

[0271] A hierarchical method for unmanned vehicle task operation passable area extraction, the main technical content is seen in any one of embodiments 11 to 18, further, the sensors on the unmanned vehicle include a laser radar, a millimeter wave radar, a GPS, and a camera.

[0272] The laser radar is used for receiving laser radar point cloud data.

[0273] The millimeter wave radar is used for receiving millimeter wave radar data.

[0274] The GPS is used for receiving positioning navigation data.

[0275] The camera is used for receiving video image data.

[0276] The step of obtaining real-time path planning is as follows:

[0277] 6.1) receive road environment information by each type of sensor on the unmanned vehicle.

[0278] 6.2) perform feature extraction on the received road environment information, and perform fusion optimization through an IMU pre-integration factor, a GPS factor, and a loop closing factor to obtain multi-sensor fusion data.

[0279] 6.3) construct a three-dimensional point cloud map and a two-dimensional grid map using the multi-sensor fusion data.

[0280] The three-dimensional point cloud map is used for displaying surrounding environment information.

[0281] 6.4) perform global path planning on the two-dimensional grid map using an improved A* algorithm with dynamic weights.

[0282] The improved A* algorithm with dynamic weights is as follows:

[0283] f(n) = g(n) + ω(n)h(n) (9)

[0284] In the formula, ω(n) is a weight coefficient. f(n) represents an estimated total cost from the starting point to the end point through node n. g(n) represents an estimated cost from the starting point to node n. h(n) represents an estimated cost from node n to the end point.

[0285] 6.5) segment trajectory smoothing optimization according to turning nodes using a Bezier curve.

[0286] The calculation formula for smoothing and optimizing the trajectory using the Bezier curve is as follows:

[0287]

[0288] In the formula, u is a free variable. P(u) represents the smoothed trajectory. P(i0) is the motion control point of the unmanned vehicle. i0 represents the motion control point index. n0 represents the total number of motion control points. is a Bernstein polynomial expression.

[0289] 6.6) Fuse and splice the segmented and smoothed path to obtain the final real-time path planning.

[0290] Embodiment 20:

[0291] A hierarchical method for extracting a passable area for unmanned vehicle task operation, the main technical content of which is seen in any one of embodiments 11 to 19, further, the determination formula of the passable area in the dynamic passable path is as follows:

[0292]

[0293]

[0294] In the formula, represents the passable area determination result. i1 represents the non-overlapping area pixel category index. j1 represents the passable area source index. represents the non-overlapping area pixel category, c1 and c2 respectively represent the passable area and the impassable area. represents the passable area source, sc1, sc2 and sc3 respectively represent the historical passable area source, the visual passable area source and the radar passable area source. represents the passable area source sc j identifies the non-overlapping area pixel p as c i the weight of the non-overlapping area pixel category. Region h , Region c , Region l respectively represent the area of the historical passable area, the visual passable area and the radar passable area. p represents the non-overlapping area pixel. represents the passable area source sc j identifies the non-overlapping area pixel p as c i the probability of the non-overlapping area pixel category. represents the binary classification probability of the non-overlapping area pixel p.

[0295] Embodiment 21:

[0296] See Figures 1 to 6A hierarchical method for unmanned vehicle task operation passable area extraction, comprising the following steps:

[0297] 1) The original high-resolution remote sensing image is processed into a binary image that can be used for model training by a generative adversarial network (GAN), which realizes the generation of realistic road network raster images from a macro and micro perspective. Road network vector extraction is based on the extraction of road network raster images, and a fast vectorization method is used to realize an end-to-end extraction process from remote sensing images to road network vectors;

[0298] 2) The road network raster vector is encoded by a MobileViT architecture combining convolutional neural network (CNN) and Transformer, which effectively extracts high-level context information and combines a pyramid decoder to realize multi-scale feature extraction and fusion, and generates a road network probability distribution of pixel categories for efficient recognition by unmanned vehicle edge computing devices;

[0299] 3) By analyzing geographic information system vector data, the correlation and interaction between discrete, multi-source and heterogeneous spatial data resources are mined, and multi-source geographic spatial vector data correlation is obtained based on adaptive quadtree spatial coding. The spatial position, spatial form, semantics, attributes and other characteristics of geographic entities are analyzed, the relationship and mutual influence between geographic data are mined, and the correlation, constraint and action relationship between surface features are described, providing deep technical support for unmanned vehicle passable area data;

[0300] 4) Based on the correlation of geographic spatial vector data, a bilateral semantic segmentation network (Bisenet) is applied to segment four types of passable areas, difficult passable areas, impassable areas and sky. Then, the C-means clustering algorithm is applied for coarse extraction and secondary segmentation of the passable area part, and further realization of more detailed passable area detection and detection of small-sized objects in the passable area;

[0301] 5) In the passable area, the unmanned vehicle uses single / multi cameras, 3D laser radar, millimeter wave radar and GPS / IMU and other sensors to perceive dynamic information in the environment, uses the LIO-SAM algorithm of multi-sensor fusion to realize real-time positioning of the unmanned vehicle and accurate mapping of the surrounding environment, then in the autonomous navigation stage, the global path planning of the improved A* algorithm and the local path planning of DWA are combined, and the Bezier curve is used to realize the optimization of the motion trajectory, and the laser radar and millimeter wave radar fusion obstacle avoidance strategy based on Kalman filtering is used to realize real-time obstacle avoidance;

[0302] 6) Finally, in order to ensure the robustness of the passable area detection in unknown unstructured environment and the consistency with the actual road form, a multi-modal information fusion passable area verification method based on strong constraints of road topology is used. The historical passable area, visual passable area detection and radar-based passable area detection are used as three main information sources. The multi-source information fusion is realized by using the Bayesian posterior estimation principle, which can maintain the consistency with the road form and has reliability, and is beneficial to the efficient autonomous navigation of the unmanned vehicle in unknown unstructured scene.

[0303] Embodiment 22:

[0304] A hierarchical method for passable area extraction for unmanned vehicle task operation, the main technical content is shown in embodiment 21, further, the network structure of the generator and the discriminator of the generative adversarial network model parameters is generated, and the road network topology is used as the input of the GAN generator. The generator can learn and generate a virtual road network similar to the real road topology, and the road attributes and the road surrounding environment features are used as the conditional input of the GAN generator, so as to generate a virtual road network with different road attributes, traffic flow data and environment features. In the training process of the GAN model, a suitable discriminator network structure is designed, which can effectively distinguish the difference between the generated virtual road network and the real road network. By comparing the similarity between the generated virtual road network and the real road network, the influence of the road network features on the generation result is evaluated. The features with higher importance are selected for the training of the GAN model.

[0305] (1) Set the objective function. The overall objective function Ltotal(G,D) used in the pix2pix model is:

[0306] LcGAN(G,D)+βL n (G)

[0307] In the formula, LcGAN(G,D) is the objective function of cGAN; L L1 (G) is the objective function of the pixel pair; β is the weight coefficient. L L1 (G) formula is:

[0308] L L1 (G)=E x,z,y [ω‖G(x,z)-y‖1]

[0309] Because the road network is sparse in high-resolution images, leading to the problem of unbalanced positive and negative samples, a spatial penalty term ω is added to the traditional L L1 (G) formula:

[0310]

[0311] If pixel j is not a road, ω is set to 1; if pixel j is a road, ω is set to α (α > 1). The formula of LcGAN(G, D) is:

[0312] LcGAN(G, D) = E x,y [logD(x, y)] + E x,z [log(1-D(x, G(x, z)))]

[0313] The traditional LcGAN(G, D) uses J-S divergence to evaluate the gap between real samples and generated samples, which may cause problems such as "gradient disappearance" and "mode collapse". Therefore, the Wasserstein distance is used instead of the traditional J-S divergence, and the modified LcGAN(G, D) formula is:

[0314]

[0315] In the formula, λ is the penalty coefficient; The gradient of .

[0316] (2) Objective function optimization. The training sample is input into the generator and discriminator of the GAN network, and the objective function Ltotal(G, D) is calculated. The objective function value is optimized, and the network parameters of the generator and the discriminator are updated. That is, first fix G, maximize D, update the parameters of D; then fix D, minimize G, update the parameters of G, the formula is:

[0317]

[0318] (3) Iterative training. Repeat the training process until the optimization result approaches Nash equilibrium or the loss function gradient of the generator and the discriminator hardly changes any more, and get the optimal generator G * .

[0319] The map image to be extracted is cropped into a 512x512 pixel image slice. In the obtained generator G*, the road network raster binary image is extracted and vectorized, and finally the road network vector result is output. The effect is shown in Figure 2 .

[0320] ​In the task of extracting lightweight and accurate urban road network, the encoder needs to extract high-level context information from the input image, which requires the encoder to have rich feature extraction capability and keep the spatial occupation small. The current advanced MobileViT is selected as the encoder, which can effectively utilize the spatial bias characteristics of convolutional neural network and the global information processing capability of Transformer, effectively enhancing the feature extraction performance. In structure, MobileViT is stacked by multiple MV2 modules and MobileViT modules, MV2 module is a lightweight reverse residual bottleneck unit proposed by MobileNetV2, and MobileViT module is a lightweight and efficient visual Transformer, see Figure 3 . The MV2 module is composed of a reverse residual structure and a linear bottleneck structure, which decomposes the standard convolution into a depth convolution and a 1×1 convolution, and the calculation of the MV2 module is as follows:

[0321]

[0322] In the formula: X and Y represent input features and output features respectively, and r is the step length of the depth convolution. The reverse residual structure is an improvement of the residual structure, which first expands the channel dimension through 1×1 convolution Φ 1,p , then encodes the spatial information in the high-dimensional space through depth convolution Φ d , and finally realizes information fusion and channel dimension reduction through 1×1 convolution Φ 2,p . When the feature information is mapped from a high-dimensional space to a low-dimensional space through a nonlinear function, there is a problem of information collapse. When performing channel dimension reduction, a linear bottleneck structure is used to reduce information loss, i.e. no nonlinear activation function is used. The skip connection is used only when the step length is 1. The MobileViT module effectively captures global information through the Transformer mechanism, and its core is the multi-head attention mechanism (MHA). MHA introduces multiple attention heads based on the self-attention mechanism, which can capture different levels of input and output relationships. The calculation of the self-attention mechanism is as follows:

[0323]

[0324] The input feature first passes through 3 linear transformations to obtain the query matrix Q, the index matrix K, and the content matrix V. The matrix multiplication is performed on Q and K based on the dimension d k of the matrix K to realize weighting and obtain the attention matrix. The attention matrix is adjusted through the softmax function f s , and then the output feature F a is obtained through matrix multiplication with V.

[0325] In MHA, the input features will be assigned to different self-attention heads, each of which learns different information relationships and then performs weighted fusion. The calculation process is as follows:

[0326]

[0327] The multi-source geospatial vector data is logically or physically concentrated in the adaptive quadtree coding spatial relationship calculation implementation logic or physical organic concentration, thereby realizing comprehensive and comprehensive expression of various types of spatial information, so as to realize the interconnection, interoperation and interoperation of geographic resource information for unmanned vehicles.

[0328] The adaptive quadtree-based spatial coding is based on the minimum bounding rectangle (MBR) of the spatial entity. According to whether the MBR of the spatial entity is completely located in the quadtree grid, iterative coding is performed until the MBR is not completely located in the sub-grid. Finally, the constraint coding is set according to the distribution of the MBR in the sub-grid of the belonging bottom layer grid, and the first bit of the existing coding is added, and the coding ends. The spatial coding of point entities and line entities is divided into point entities and line entities. Since the point entity has no MBR, the grid division times can be set according to actual needs. The number of coding bits of this method is 14.

[0329] (1) Point entity spatial coding. Point entity coding is relatively simple, that is, the spatial region is divided into quadtree grids, and the upper left corner of the quadtree is coded as "1, 2, 3, 4" in a counterclockwise manner. The position of the point in the quadtree grid is judged according to the longitude and latitude of the point, and the coding of the grid where the point is located is taken as the first coding of the point entity. Then recursively divide and judge the grid to which the entity belongs, and add the new coding to the previous bit of the existing coding in turn, until the division is 14 times, and the 14-bit coding of the point entity is obtained.

[0330] (2) Spatial coding of line and surface entities. The spatial coding of line and surface entities is based on adaptive quadtree coding of MBR. The coding method is similar to that of point entities, but additional constraint coding is added. In addition, the effective number of codes is different due to the different sizes of entities, and the process is as shown in Figure 4 The calculation of constraint coding is based on the distribution of the MBR of the entity in the grid. The MBR in the grid may be located in the same grid, cross or vertical across two grids, and cross four grids, and the constraint coding is set to "1, 2, 3, 4" respectively.

[0331] (3) Grid topology correlation calculation. According to the similarities and differences between the two entities in the grid, the grid topology correlation is defined as apart, adjacent and related. The calculation of the grid topology correlation is realized by comparing the spatial codes of the two entities. First, the effective bits of the code except for "0" are extracted. Among them, if it is a line or surface entity, the first effective digit is the constraint code, so the actual effective bits should be the extracted bits minus 1. Then take the smaller value of the effective bits as the comparison times t, from the last bit of the two entity codes, compare in turn whether they are equal, there are three cases:

[0332] 1. The code values are not equal in i(i<t) times of comparison, the relationship is "apart";

[0333] 2. Only the tth comparison is not equal, which is "adjacent";

[0334] 3. All are equal, which is "related".

[0335] At the same time, the grid distance correlation is divided into very close, close, general, far and very far 5 levels, and the calculation steps are: compare from the last bit of the two entity codes, get the number of continuous equal bits n, the longitude difference δx of the two entities is the total longitude Δx / 2n, the latitude difference δy is the total latitude difference Δy / 2n, the larger n is, the closer the two entities are.

[0336] Compared with general image recognition problems, the detection of passable areas of wild roads has higher requirements for recognition accuracy. Semantic segmentation mainly identifies from two aspects: spatial information mainly refers to visible features such as boundaries and textures; semantic information represents a rich receptive field, mainly referring to high-level features in pixel points. Unlike the general encoder-decoder mode network, the Bisenet network decouples spatial information and semantic information by adopting a double-side segmentation structure, as shown in Figure 5 It consists of two branches of spatial path and context path, attention refinement module (ARM), and feature fusion module (FFM).

[0337] The spatial path is only responsible for solving the problem of spatial information loss. Rich spatial information is obtained through a relatively shallow network and a wider channel number, mainly realized by 3 convolutional layers with a step of 2, and the output features extracted by this branch are reduced to 1 / 8 of the original features through downsampling. This branch is more inclined to extract the edge lines, textures and other shallow features of the wild road.

[0338] The context path is mainly responsible for providing semantic information and sufficient receptive field. The branch is based on a residual network as the main body to perform 4 times, 8 times, 16 times and 32 times fast down-sampling, so as to obtain a larger receptive field. A global average pooling module is added at the tail of the 32 times down-sampling, so that each channel is allocated a corresponding weight without increasing the amount of calculation, and the channel providing more semantic information is focused on.

[0339] Although the semantic segmentation algorithm detects the passable area, it does not further detect small-size objects in the passable area. A clustering method is used to perform secondary segmentation on the passable area part to realize high-precision passable area detection. In image processing, clustering is to gather images with certain similar features together, and to exclude other data. Fuzzy clustering analysis is one of the main techniques of unsupervised machine learning. This method applies fuzzy theory to analyze and model important data, and has been widely used in various fields such as image processing and pattern recognition, and has practical application value.

[0340] Among them, the fuzzy C-means clustering algorithm (Fuzzy C-Means, FCM) is the most widely used in fuzzy algorithms. By optimizing the objective function, the membership degree of each sample point to all class centers is obtained, and the data in the sample is automatically classified. The specific process of the fuzzy C-means clustering algorithm includes the following steps.

[0341] (1) The number of clusters in clustering is given, and the iteration convergence condition is set, and the cluster centers are initialized;

[0342] (2) Calculate the membership value using the current cluster center;

[0343] (3) Repeat the above operation until the membership value in each sample reaches a stable state;

[0344] (4) Recalculate the cluster center using the current membership value;

[0345] (5) Until the algorithm converges, the cluster center and membership value of each class are obtained, and the fuzzy clustering is completed.

[0346] The self-built multiple sensors are used to perceive the surrounding environment and the position of the unmanned vehicle, and the mathematical analysis method is used to determine the moving path of the unmanned vehicle. The laser radar and millimeter wave radar are fused to improve the distance and angle estimation of the target and improve the active obstacle avoidance capability, so as to realize the autonomous navigation of the unmanned vehicle from the starting point to the destination. The overall block diagram is shown in Figure 6 .

[0347] The LIO-SAM algorithm framework mainly includes four modules: image projection, feature extraction, IMU pre-integration and factor graph optimization. Through the fusion of multiple sensors such as LIDAR, IMU and GPS, the point cloud data obtained by LIDAR scanning is processed by removing distortion and extracting features, and then optimized by IMU pre-integration factor, GPS factor and loop closure factor fusion, to realize the 3D SLAM algorithm of multi-sensor fusion. Laser radar can construct three-dimensional point cloud map and two-dimensional grid map at the same time. The three-dimensional point cloud map can display the surrounding environment information more intuitively, and the two-dimensional grid map is used for path planning in the autonomous navigation stage.

[0348] On the basis of the original A* algorithm formula, a coefficient is added before the heuristic function, and the formula is as follows:

[0349] f(n)=g(n)+ω(n)h(n)

[0350] The weight coefficient ω(n) is designed in the heuristic function. When the distance to the target point is far, the weight coefficient is increased to improve the search efficiency; when approaching the target point, the weight coefficient is reduced to reduce the importance of the heuristic function, and the optimal path to the target point is found.

[0351] The trajectory is smoothed by using the Bezier curve:

[0352]

[0353] The free variable u is any point between the starting point 0 and the ending point 1, and P(u) is the control point of the mobile robot motion control. i,n (u) is the Bernstein polynomial expression. The steps of combining the Bezier curve and the improved A* algorithm are as follows:

[0354] (1) Use the improved A* algorithm with dynamic weight for global path planning;

[0355] (2) According to the turning nodes, the planned path is segmented and smoothed by using the Bezier curve for trajectory optimization;

[0356] (3) Fuse and splice the segmented and smoothed paths to output the final path.

[0357] On the basis of the global path planning algorithm using the improved A* algorithm, the local path planning algorithm DWA dynamic window method is added to deal with the interference of dynamic obstacles.

[0358] LiDAR boasts extremely high sensitivity and resolution, enabling it to detect changes in minute objects. However, lasers have weak penetrating power and are susceptible to extreme weather conditions such as high dust levels, rain, snow, and fog. Millimeter-wave radar, on the other hand, not only offers high resolution and sensitivity but also strong penetrating power, allowing it to penetrate media like fog for detection. Therefore, combining millimeter-wave radar and LiDAR for obstacle avoidance can make autonomous vehicles more adaptable to complex environments, improving their obstacle avoidance performance.

[0359] To further improve the reliability of real-time detection of passable areas, historical passable areas (Regions) were identified. h Visually accessible region c Radar-accessible region l As three important information input sources, Bayesian decision theory is used to perform post-fusion. The historical traversable area is mainly obtained by coordinate transformation based on the inter-frame pose relationship of the autonomous vehicle, and this area is unified with the other two information sources in the same coordinate system. Therefore, let Perh(c i |p,sc j ) is the source of detection information for passable areas. j The probability of identifying a pixel p in the raster map as a passable area is calculated, where c1 and c2 represent passable and impassable areas, respectively; sc1, sc2, and sc3 represent the three sources of passable areas: historical passable areas, visual passable areas, and radar passable areas, respectively. Therefore, the binary classification probability decision matrix for pixel p is constructed as follows:

[0360]

[0361] Considering that during the dynamic driving process of autonomous vehicles, due to factors such as turning and dynamic obstacle avoidance, there may be significant deviations between the vehicle's heading and the road's extension direction, which may lead to difficulties in growing the radar's passable area or even its failure, the fused category probability representation adopts a dynamic weighting form, specifically expressed as:

[0362]

[0363] Where, ω ij Indicates the passable area source sc j Pixel p is identified as c i Class weight, and However, in the actual experiment, ω ij The weights are dynamically changing, especially when radar passable areas are difficult to grow. For overlapping areas among the three passable areas, this paper considers them to have high confidence, and their final category is classified as passable. For non-overlapping areas, the weights ω of each information source are... ij, the specific calculation method is as follows:

[0364]

[0365] Wherein Region h , Region c and Region l respectively represent the area of the historical passable region, the visual passable region and the radar passable region. Finally, the final category determination formula of the non-overlapping region pixel p is:

[0366]

[0367] Wherein, represents the non-overlapping region pixel category, that is, whether it belongs to the passable region.

Claims

1. A hierarchical method for extracting passable areas for unmanned vehicle tasks, characterized in that, Includes the following steps: 1) Acquire several original remote sensing images and select those with a spatial resolution greater than a preset value as original high-resolution remote sensing images. 2) Generative adversarial networks are used to convert the original high-resolution remote sensing images into road network raster images, and the road network raster images are vectorized to obtain road network raster vector results; 3) Lightweight feature encoding of the road network raster vector results is performed using the MobileViT architecture to obtain road network information; 4) By analyzing geographic information system vector data, the correlation of multi-source geospatial vector data is obtained; 5) Based on the association of multi-source geospatial vector data, the Bisenet bilateral semantic segmentation network is applied to segment the road network information into regions, and historical passable areas are obtained through small-sized object detection; 6) Real-time obstacle avoidance and path planning are performed using various sensors on the unmanned vehicle to obtain real-time path planning; 7) By combining historical passable areas and real-time route planning, dynamic passable routes are obtained.

2. The hierarchical method for extracting passable areas for unmanned vehicle tasks according to claim 1, characterized in that, The network structure of the generative adversarial network includes a generator and a discriminator; The generator is used to learn and generate virtual road networks with different road attributes, traffic flow data and environmental characteristics; The discriminator assesses the impact of road network features on the generation result by comparing the similarity between the generated virtual road network and the real road network.

3. The hierarchical method for extracting passable areas for unmanned vehicle tasks according to claim 1, characterized in that, The objective function of the generative adversarial network is as follows: G * =min G max D LcGAN(G,D)+βL L1 (G) (1) L L1 (G)=E x,z,y [ω‖G(x,z)-y‖1] (3) In the formula, G * LcGAN(G,D) represents the optimal generator; G and D represent the generator and discriminator, respectively; LcGAN(G,D) represents the objective function of the conditional generative adversarial network (cGAN); β represents the weight coefficients; L L1 (G) represents the objective function for pixel pairs; x represents the input real image; y represents the real target image corresponding to x; z represents noise; D(x,y) represents the discriminator's output for the real image pair (x,y); G(x,z) represents the target image generated by the generator with the real image x and noise z as input; D(x,G(x,z)) represents the discriminator's output for the real image x and the generated target image G(x,z); E x,y E x,z , E x,z,y Both represent the expected function; λ represents the penalty coefficient; Represents the random interpolation points between the real image x and the generated target image G(x,z); The generator represents random interpolation points. The generated image; Let denote the gradient function; ‖·‖1 and ‖·‖2 both denote norms; ω denotes the space penalty term; α is a constant greater than 1.

4. The hierarchical method for extracting passable areas for unmanned vehicle tasks according to claim 1, characterized in that, The MobileViT architecture is composed of multiple MV2 modules and multiple MobileViT modules stacked together. The MV2 module consists of an inverted residual structure and a linear bottleneck structure, which is used to map feature information from a high-dimensional space to a low-dimensional space through a nonlinear function. The calculation formula for the MV2 module is as follows: In the formula, Y represents the output feature of the MV2 module; X represents the input feature; r is the stride of the depthwise convolution; Φ 1,p Φ represents the first layer of 1×1 convolution; d Represents depthwise convolution; Φ 2,p This represents the second 1×1 convolution; The MobileViT module captures input and output relationships at different levels through a multi-head attention mechanism; The calculation formula for the MobileViT module is as follows: In the formula, F out For the output characteristics of the MobileViT module; f cat The function represents the fusion function; i represents the index of the self-attention head; h represents the total number of self-attention heads. W represents the output feature of the i-th self-attention head; O f represents the weight matrix; s This represents the softmax function; Q represents the query matrix. represents the weight of the i-th self-attention head query matrix; K represents the index matrix; d represents the weight of the i-th self-attention head index matrix; k V represents the dimension of the index matrix K; V represents the content matrix. This represents the weight of the i-th self-attention head content matrix.

5. The hierarchical method for extracting passable areas for unmanned vehicle tasks according to claim 4, characterized in that, The output features of the MobileViT architecture are also combined with a pyramid decoder to extract and fuse multi-scale features to obtain road network information.

6. The hierarchical method for extracting passable areas for unmanned vehicle tasks according to claim 1, characterized in that, The association of the multi-source geospatial vector data is obtained through spatial coding based on an adaptive quadtree; The steps of the space coding based on the adaptive quadtree are as follows: a1 performs spatial encoding on point entities; a11 sets the current spatial region as the region to be encoded and initializes the encoding of point entities; a12 divides the region to be encoded into a quadtree grid, resulting in four grids, and encodes them in a counterclockwise direction; a13 determines the position of a point entity in the four grids and adds the code of the grid where the point entity is located to the front of the point entity's code; a14 determines whether the termination condition is met. If yes, the spatial code of the point entity is obtained. If not, the grid where the point entity is located is taken as the region to be encoded, and the process returns to step a12. The termination condition includes reaching a preset number of divisions; a2 performs spatial encoding on line entities and surface entities; a21 performs adaptive meshing of the current spatial region, calculates the number of rows and columns of line entities or surface entities in the mesh, and determines the encoding length; a22 determines the encoding bit length based on the number of rows and columns of line or polygon entities in the mesh, as shown below: 2 m <min(r,c)<2 m+1 (7) In the formula, r and c are the number of rows and columns of a line entity or surface entity in the mesh, respectively; m is the number of bits used for encoding; and m0 is the encoding length. a23 performs spatial encoding on the point at the lower left corner of a line entity or surface entity to obtain an m-bit code; a24 calculates the constraint code and adds the constraint code before the m-bit code to obtain the m+1-bit code; The constraint encoding is based on the distribution of line entities or surface entities in the mesh; a25 is padded with 0s before the m+1 bit code until the m0 bit code is reached, to obtain the spatial code of the line entity or the surface entity. a3 obtains the association of multi-source geospatial vector data by comparing the valid codes in the spatial codes of two different entities; The valid encoding does not include constraint encoding; If all valid codes are equal, then the two different entities are related; If the effective codes are only different at the very beginning, then the two different entities are adjacent. If, apart from the first valid code, there are other codes that are not equal, then the two different entities are separated.

7. The hierarchical method for extracting passable areas for unmanned vehicle tasks according to claim 1, characterized in that, The steps for using the bilateral semantic segmentation network BiseNet to segment road network information into regions and obtaining historically passable areas through small-sized object detection are as follows: 5.1) Use three convolutional layers with a stride of 2 and downsampling to reduce the resolution of road network information and extract shallow features; The shallow features include the edge lines and textures of roads in the field; 5.2) Using the Restnet18 residual network as the backbone, the road network information is downsampled by different multiples, and the context information of the road network information is obtained by using the global average pooling module and the attention mechanism module. 5.3) The shallow features and contextual information are fused through the multi-feature fusion module to achieve the initial segmentation of the passable region; 5.4) Clustering method is used to detect small objects in the initially segmented passable region to achieve secondary segmentation of the passable region and obtain the historical passable region.

8. The hierarchical method for extracting passable areas for unmanned vehicle tasks according to claim 7, characterized in that, The clustering method includes the C-means clustering algorithm; The steps for detecting small objects in the initially segmented passable region using the C-means clustering algorithm are as follows: 5.4.1) Set the number of clusters and the iteration convergence condition, and initialize each cluster center; 5.4.2) Calculate the membership value using the current cluster centers; 5.4.3) Determine whether the membership value of each data point to all cluster centers has reached a stable state. If yes, proceed to step 5.4.4); otherwise, return to step 5.4.

2. 5.4.4) Update the cluster centers using membership values; 5.4.5) Determine whether the iterative convergence condition has been met. If yes, obtain the cluster centers and membership values. If not, return to step 5.4.

2.

9. The hierarchical method for extracting passable areas for unmanned vehicle tasks according to claim 1, characterized in that, The sensors on the unmanned vehicle include lidar, millimeter-wave radar, GPS, and cameras; The lidar is used to receive lidar point cloud data; The millimeter-wave radar is used to receive millimeter-wave radar data; The GPS is used to receive positioning and navigation data; The camera is used to receive video image data; The steps to obtain the real-time path plan are as follows: 6.1) Various sensors on the autonomous vehicle receive road environment information; 6.2) Feature extraction is performed on the received road environment information, and the data is fused and optimized using IMU pre-integration factor, GPS factor, and loop closure factor to obtain multi-sensor fused data; 6.3) Construct 3D point cloud maps and 2D raster maps using multi-sensor fusion data; The three-dimensional point cloud map is used to display information about the surrounding environment; 6.4) Perform global path planning on a two-dimensional raster map using an improved A* algorithm with added dynamic weights; The improved A* algorithm with added dynamic weights is as follows: f(n)=g(n)+ω(n)h(n) (9) In the formula, ω(n) is the weight coefficient; f(n) represents the estimated total cost from the starting point through node n to the ending point; g(n) represents the estimated cost from the starting point to node n. h(n) represents the estimated cost of reaching the destination from node n; 6.5) Optimize the planned path by using Bézier curves to smooth the segmented trajectory according to the turning nodes; The calculation formula for smoothing and optimizing the trajectory using Bézier curves is shown below: In the formula, u is a free variable; P(u) represents the smoothed trajectory; P(i0) is the motion control point of the unmanned vehicle; i0 represents the index of the motion control point; n0 represents the total number of motion control points; For Bernstein polynomial expressions; 6.6) The smoothed path segments are merged and spliced ​​to obtain the final real-time path plan.

10. A hierarchical method for extracting passable areas for unmanned vehicle tasks according to claim 1, characterized in that, The formula for determining the passable area in the dynamically passable path is as follows: In the formula, This indicates the result of determining the passable area; i1 represents the pixel category index of the non-overlapping area; j1 represents the source index of the passable area. The pixel categories in the non-overlapping areas are represented by c1 and c2, which represent passable and impassable areas, respectively. The sc1, sc2, and sc3 represent the historical, visual, and radar accessibility sources, respectively. Indicates the source of the passable area (sc). j Pixel p in the non-overlapping region is identified as c. i Weights of pixel categories in non-overlapping regions; Region h Region c Region l represents the area of ​​the historical passable area, the visual passable area, and the radar passable area, respectively; p represents the non-overlapping pixel area. Indicates the source of the passable area (sc). j Pixel p in the non-overlapping region is identified as c. i The probability of pixel category in non-overlapping regions; This represents the binary classification probability of pixel p in the non-overlapping region.

Images