Crossable cost model training method, path generation method and device

Abstract

This application relates to the field of computer vision technology and discloses a method for training a traversable cost model, a path generation method, and an apparatus. The method includes: training a normalized vegetation network based on a pre-acquired first image and a corresponding labeled image to obtain a normalized vegetation model; the labeled image is an image labeled according to the normalized vegetation index of the first image; processing a second image using the normalized vegetation model to obtain vegetation pixels in the second image; training a traversable cost network based on the vegetation pixels and corresponding depth values ​​to obtain a traversable cost model; the depth values ​​are obtained by pre-acquired depth maps of the second image. The method in this embodiment can input the acquired image into the traversable cost model in real time, predict the traversable cost of vegetation in real time using the traversable cost model, and plan a feasible path based on the predicted traversable cost, thus ensuring driving safety for automobiles.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a method for training a traversable cost model, a path generation method, and an apparatus. Background Technology

[0002] Vision-based navigation is the foundation for autonomous control. Currently, autonomous driving technology is approaching maturity, but a prerequisite is the ability to perceive the surrounding environment. Real-world scenarios are generally divided into structured and unstructured roads. Structured scenarios typically refer to well-structured roads such as highways and urban arterial roads. These roads have clear road markings, a relatively simple background environment, and readily apparent geometric features. Therefore, the detection problem for these roads can be simplified to lane line or road boundary detection. Research on lane lines has reached a mature stage. Unstructured scenarios, on the other hand, generally refer to less structured scenarios such as urban non-arterial roads and rural streets. These scenarios lack lane lines and clear road boundaries, leading to poor performance. Furthermore, in rural and other rural conditions, environmental factors such as weather, road materials, shadows, and water accumulation can affect performance. Given the complex and varied road conditions in unstructured scenarios, autonomous vehicles cannot accurately perceive their surroundings, making it difficult to guarantee driving safety.

[0003] In addition, obstacle prediction is crucial for autonomous driving, especially in complex wilderness environments. For driverless cars, large rocks, fallen tree trunks, and small clumps of vegetation can all serve as obstacles. Depth cameras and LiDAR can accurately and effectively measure the distance between the vehicle and these obstacles. However, in wilderness environments, factors such as weather and lighting conditions can significantly impact performance. Light-resistant depth cameras are generally expensive, and LiDAR is also costly. Summary of the Invention

[0004] This application provides a method for training a traversable cost model to address the problem in the prior art that, given the complex and ever-changing road conditions in unstructured scenarios, intelligent driving vehicles cannot accurately perceive their surroundings, making it difficult to ensure driving safety.

[0005] Accordingly, embodiments of this application also provide a path generation method, a traversable cost model training device, and a path generation device to ensure the implementation and application of the above methods.

[0006] To address the aforementioned technical problems, this application discloses a method for training a traversable cost model, the method comprising:

[0007] A normalized vegetation network is trained based on a pre-collected first image and its corresponding labeled image to obtain a normalized vegetation model; the labeled image is an image labeled with the normalized vegetation index of the first image.

[0008] The second image is processed using the normalized vegetation model to obtain the vegetation pixels of the second image;

[0009] Based on the vegetation pixels and their corresponding depth values, a traversable cost network is trained to obtain a traversable cost model; the depth values ​​are obtained by pre-collecting the depth map of the second image.

[0010] This application also discloses a path generation method, the method comprising:

[0011] The acquired target image is input into the traversable cost model to obtain the traversable cost; wherein, the traversable cost model is trained according to the traversable cost model training method described in this application;

[0012] A topology graph is constructed using the traversable cost;

[0013] The optimal path is obtained by processing the topology graph using an optimal path algorithm.

[0014] The topology diagram includes one or more regional nodes and an original node; the regional nodes are obtained by segmenting the target image; adjacent regional nodes are bidirectionally connected, and the original node is unidirectionally connected to any one or more regional nodes.

[0015] This application also discloses a training device for a traversable cost model, the device comprising:

[0016] The normalized vegetation model training module trains a normalized vegetation network based on a pre-collected first image and the corresponding labeled image to obtain a normalized vegetation model; the labeled image is an image labeled according to the normalized vegetation index of the first image.

[0017] The image processing module processes the second image using the normalized vegetation model to obtain the vegetation pixels of the second image;

[0018] The traversable cost model training module trains a traversable cost network based on the vegetation pixels and their corresponding depth values ​​to obtain a traversable cost model; the depth values ​​are obtained by pre-collecting the depth map of the second image.

[0019] This application also discloses a path generation apparatus, the apparatus comprising:

[0020] The image processing module inputs the acquired target image into the traversable cost model to obtain the traversable cost; wherein, the traversable cost model is trained according to the traversable cost model training method described in this application;

[0021] The topology graph generation module constructs a topology graph using the traversable cost.

[0022] The data processing module processes the topology graph using an optimal path algorithm to obtain the optimal path;

[0023] The topology diagram includes one or more regional nodes and an original node; the regional nodes are obtained by segmenting the target image; adjacent regional nodes are bidirectionally connected, and the original node is unidirectionally connected to any one or more regional nodes.

[0024] This application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements one or more of the methods described in this application.

[0025] This application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements one or more of the methods described in this application.

[0026] In this embodiment, a normalized vegetation network is trained based on a pre-acquired first image and its corresponding labeled image to obtain a normalized vegetation model. The labeled image is an image labeled according to the normalized vegetation index of the first image. The normalized vegetation model is used to process a second image to obtain vegetation pixels in the second image. Based on the vegetation pixels and their corresponding depth values, a traversability cost network is trained to obtain a traversability cost model. The depth values ​​are obtained by pre-acquired depth maps of the second image. The method in this embodiment can input the acquired image into the traversability cost model in real time, predict the traversability cost of vegetation in real time using the traversability cost model, and plan feasible paths based on the predicted traversability cost, thus ensuring driving safety.

[0027] Additional aspects and advantages of the embodiments of this application will be set forth in the following description, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description

[0028] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0029] Figure 1 A flowchart of the traversable cost model training method provided in the embodiments of this application;

[0030] Figure 2 A schematic diagram of a training normalized vegetation network provided in an embodiment of this application;

[0031] Figure 3 A schematic diagram illustrating the training of a traversable cost network provided in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of the traversal cost network structure provided in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram of the output head provided in an embodiment of this application;

[0034] Figure 6 A schematic diagram illustrating the generation of a traversable cost model provided in an embodiment of this application;

[0035] Figure 7 A flowchart of the path generation method provided in the embodiments of this application;

[0036] Figure 8 A schematic diagram illustrating the generation of the optimal path provided in an embodiment of this application;

[0037] Figure 9 Topology diagram provided for embodiments of this application;

[0038] Figure 10 This is a schematic diagram of the structure of the traversable cost model training device provided in the embodiments of this application;

[0039] Figure 11 This is a schematic diagram of the path generation device provided in the embodiments of this application;

[0040] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0042] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0043] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0044] The solutions provided in this application can be executed by any electronic device, such as a terminal device or a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. The traversable cost model training method, path generation method, and apparatus provided in this application aim to solve at least one of the technical problems existing in the prior art.

[0045] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0046] This application provides a possible implementation method, such as... Figure 1The diagram shows a flowchart of a method for training a traversable cost model. This method can be executed by any electronic device, optionally on a server or a terminal device. For ease of description, the method provided in this application embodiment will be described below with the server as the execution subject.

[0047] like Figure 1 As shown, the method may include the following steps:

[0048] Step 101: Train a normalized vegetation network based on the pre-collected first image and the corresponding labeled image to obtain a normalized vegetation model.

[0049] The labeled image is an image labeled according to the normalized vegetation index of the first image.

[0050] Most outdoor environments are natural environments. Within these environments, the Normalized Difference Vegetation Index (NDVI) can be used to distinguish between vegetated areas and passable surfaces. It's a spectral transform used to monitor vegetation characteristics in remote sensing. Because near-infrared light is not absorbed by plants and may even be strongly reflected, while red light is absorbed, NDVI measures the difference between near-infrared and red light to quantify vegetated areas. Its range is between -1 and 1. A negative value indicates water within the area. Conversely, a value close to 1 indicates green leaves or grassland. A value close to zero indicates the absence of vegetation and may represent roads or urbanized areas.

[0051] As a first example, such as Figure 2 As shown, a first image and a corresponding labeled image can be pre-acquired. Optionally, the acquired first image can be an RGB image. The RGB image can be pre-divided into multiple regions, and then the normalized vegetation index (NVI) corresponding to each region can be obtained. The NVI of each region can then be used to generate a labeled image. Specifically, obtaining the labeled image corresponding to the first image is as follows:

[0052] A visible light camera is used to acquire the red band corresponding to each region in the first image, and a near-infrared camera is used to acquire the near-infrared band corresponding to each region in the first image.

[0053] The Normalized Difference Vegetation Index (NDVI) is obtained using the formula:

[0054]

[0055] NDVI is the Normalized Difference Vegetation Index. In pixels Reflectivity in the near-infrared band; For in pixels The red band reflectance at that location.

[0056] The first image is labeled using the normalized vegetation index calculated according to the above steps to generate the labeled image. It should be noted that in this embodiment, there can be multiple first images and multiple labeled images.

[0057] like Figure 2 The first image and its corresponding labeled image can be input into the normalized vegetation network for training. Optionally, the normalized vegetation network uses the stochastic gradient descent (SGD, or SGDRegressor) model from the scikit-learn library as the base model. The SGDRegressor model mainly uses stochastic gradient descent to perform linear regression to predict vegetation pixels.

[0058] The first image obtained by the above method and its corresponding labeled image can be input into the SGDRegressor model for training, and finally a normalized vegetation model can be obtained.

[0059] Step 102: Process the second image using the normalized vegetation model to obtain the vegetation pixels of the second image.

[0060] As a second example, the acquired second image can be input into the normalized vegetation model trained above. The normalized vegetation model outputs the vegetation pixels corresponding to each region in the second image. Optionally, the second image can be divided into multiple regions, such as 8×8 rectangular regions, and then the second image can be input into the normalized vegetation model to output the vegetation pixels corresponding to each region. Optionally, the second image can be the same as the first image or different from the first image.

[0061] Step 103: Train a traversable cost network based on the vegetation pixels and their corresponding depth values ​​to obtain a traversable cost model.

[0062] The depth value is obtained by pre-acquiring a depth map of the second image.

[0063] As a third example, the second image can be segmented into 8×8 rectangular regions. In addition, this embodiment can also use a depth camera to acquire a depth map corresponding to the second image, the depth map including depth values ​​corresponding to each rectangular region of the second image. Optionally, the traversable cost network uses the MobileNetV3 network as the base model, and can be trained using vegetation pixels and depth values ​​to generate a traversable cost model that can directly predict the traversable cost of vegetation.

[0064] In this embodiment, a normalized vegetation network is trained based on a pre-acquired first image and its corresponding labeled image to obtain a normalized vegetation model. The labeled image is an image labeled according to the normalized vegetation index of the first image. The normalized vegetation model is used to process a second image to obtain vegetation pixels in the second image. Based on the vegetation pixels and their corresponding depth values, a traversability cost network is trained to obtain a traversability cost model. The depth values ​​are obtained by pre-acquired depth maps of the second image. The method in this embodiment can input the acquired image into the traversability cost model in real time, predict the traversability cost of vegetation in real time using the traversability cost model, and plan feasible paths based on the predicted traversability cost, thus ensuring driving safety.

[0065] In an optional embodiment, training a traversable cost network based on the vegetation pixels and corresponding depth values ​​to obtain a traversable cost model includes:

[0066] Using the vegetation pixels and the corresponding depth values, a traversable cost matrix map of the second image is obtained.

[0067] In conjunction with the third example, such as Figure 3 As shown, optionally, the traversable cost corresponding to each region of the second image is calculated using the vegetation pixels and the corresponding depth values ​​of each region, and a traversable cost matrix is ​​generated using the traversable cost corresponding to each region.

[0068] The traversable cost network is trained using the second image and the traversable cost matrix graph to obtain the traversable cost model.

[0069] Optionally, the second image and its corresponding traversable cost matrix can be input into the MobileNetV3 network for training to obtain a traversable cost model. In this embodiment, there can be multiple second images and their corresponding traversable cost matrices.

[0070] In an optional embodiment, obtaining the traversable cost matrix of the second image using the vegetation pixels and the corresponding depth values ​​includes:

[0071] The Normalized Difference Vegetation Index (NDV) calculated using the formula in the first example ranges from -1 to 1. A negative value indicates water within the area; conversely, a value close to 1 indicates foliage or grassland; and a value close to zero indicates the absence of vegetation, suggesting a road or urbanized area. For the various vegetation types in a forest scene—shrubs, grasses, trees, etc.—the traversability of each type of vegetation varies for vehicles. For example, grasslands can be traversed, while shrubs require avoidance.

[0072] The vegetation pixels are converted into visible light atmospheric impedance index, and the vegetation cost is obtained using the visible light atmospheric impedance index:

[0073]

[0074] in, For in pixels The visible light atmospheric impedance index at that location.

[0075] Optionally, the visible light atmospheric impedance index of the vegetated area is calculated using the following formula:

[0076]

[0077] in, In pixels Reflectivity in the green band; For in pixels Reflectivity in the blue band; For in pixels Reflectivity in the red band.

[0078] Visible light cameras can be used to acquire the green, blue, and red bands corresponding to the vegetation area. Then, the visible light atmospheric impedance index of the vegetation area can be calculated according to the above formula.

[0079] In this embodiment, since there are various types of vegetation in the forest scene, such as shrubs, grasses, and trees, each with different traversability, the normalized vegetation model generated above can be used to predict vegetation and non-vegetation in the second image. Generally, the higher the chlorophyll content of organic matter, the easier it is to traverse. At the same scale, leaves have relatively high chlorophyll concentration, shrubs are less abundant, brambles are even less common, and bark is the least abundant. The visible light atmospheric impedance index (VARI) is a good representative of chlorophyll content, measuring vegetation by its "green" appearance. Therefore, the vegetation areas predicted by the normalized vegetation model are transformed using VARI to calculate the traversability cost of each area, and this cost is used as the basis for determining whether an obstacle can be traversed.

[0080] In addition, the depth cost is obtained by comparing the depth value with the maximum depth value in the depth map:

[0081]

[0082] in, This is the maximum depth value in the depth map. The depth value is given.

[0083] The ability to perceive depth is crucial for driving a car to avoid obstacles. Knowing the relative position of obstacles is essential for navigation and maneuvering around them. Therefore, obstacle avoidance also requires understanding the distance between the obstacle and the vehicle. Thus, depth maps are captured using depth cameras, and these maps are used to calculate depth costs.

[0084] The traversable cost matrix is ​​obtained using the vegetation cost and the depth cost.

[0085] Optionally, the traversable cost corresponding to the vegetation region of the second image is calculated using the vegetation pixels and the corresponding depth values ​​of the vegetation region in the second image, and a traversable cost matrix is ​​generated using the traversable cost corresponding to the vegetation region.

[0086] Using only depth cost to determine traversability cost is inappropriate because depth information is three-dimensional and only effective in the near field. Depth cameras limit the inference of maximum depth from depth maps. Therefore, it is necessary to combine vegetation cost and depth cost, and use far-field information from vegetation indices to estimate costs beyond the depth range.

[0087] The traversable cost network is trained using the second image and the traversable cost matrix graph to obtain the traversable cost model.

[0088] This embodiment segments the target image into multiple regions, defining the traversable cost of each region as its fully common cost (fc) value. The target image is then input into a traversable cost model trained using the above method for prediction, yielding the fc value for each region, which is the traversable cost. Optionally, the traversable cost model uses the MobileNetV3 network as its base model. MobileNetV3 is a typical lightweight network, suitable for deployment, and capable of quickly predicting the traversable cost from the target image captured by the camera.

[0089] Alternatively, the structure of a traversable cost network is as follows: Figure 4As shown, the system includes a MobileNetV3 network and a global average pooling layer. The input second image is processed by the MobileNetV3 network to extract features, and then global average pooling is performed. Finally, the data is input to 64 output heads (assuming the second image is divided into 64 8×8 rectangular regions). Each output head represents the traversability cost of a rectangular region in the second image. Optionally, as... Figure 5 As shown, each output head consists of four fully-connected layers (FC layers). The first three fully-connected layers are followed by ReLU activation functions, and the last fully-connected layer is followed by a linear layer to output the final result. The 64 output heads output an 8×8 cost matrix.

[0090] like Figure 6 As shown, the target image is divided into 64 equal 8×8 rectangular regions. The fc value of each region is obtained using the target image and depth map, and a traversable cost matrix map is generated. Figure 6 P0, P1, ..., P7, P8, P9, ..., P 15 , ..., P 56 P 57 , ..., P 64 These are all regions corresponding to the traversable cost matrix diagram. The traversable cost model can be trained using the traversable cost matrix diagram and the second image.

[0091] In an optional embodiment, obtaining the traversable cost matrix using the vegetation cost and the depth cost includes:

[0092] Using the vegetation cost and the depth cost, the far-field cost is obtained:

[0093]

[0094] And, near-field cost:

[0095]

[0096] The traversal cost is obtained using the far-field cost and the near-field cost:

[0097]

[0098] Based on the traversable costs of all regions in the second image, generate the traversable cost matrix.

[0099] in, (0, 10) -6 α and β are constants between (0, 1); α and β are constants between (0, 1).

[0100] In distant scenes, smaller obstacles only become visible when a car approaches, so knowing they are far away is sufficient; precise values ​​are not required. In close-up scenes, depth cost can be weighted by vegetation cost to obtain near-field cost. Furthermore, flat ground is not considered an obstacle for a car, therefore, it should not be assigned any cost. Combining far-field cost, approach cost, and ground cost to obtain traversable cost improves the reliability of the traversable cost.

[0101] like Figure 7 As shown in the embodiments of this application, a path generation method is also provided, the method comprising:

[0102] Step 701: Input the acquired target image into the traversable cost model to obtain the traversable cost; wherein the traversable cost model is trained according to the above-described traversable cost model training method.

[0103] This embodiment segments the target image into multiple regions, defining the traversable cost of each region as its fully common cost (fc) value. The target image is then input into a traversable cost model trained using the above method for prediction, yielding the fc value for each region, which is the traversable cost. Optionally, the traversable cost model uses the MobileNetV3 network as its base model. MobileNetV3 is a typical lightweight network, suitable for deployment, and capable of quickly predicting the traversable cost from the target image captured by the camera.

[0104] Step 702: Construct a topology graph using the traversable cost.

[0105] The topology diagram includes one or more region nodes and an original node; the region nodes refer to regions obtained by segmenting the target image; adjacent region nodes are bidirectionally connected; and the original node is unidirectionally connected to any one or more region nodes.

[0106] Assuming the target image is segmented into 8×8 rectangular regions, there are 64 regions in total. These 64 regions are designated as 64 region nodes. Each node is bidirectionally connected to its adjacent nodes, indicating that a car can travel between two region nodes. For example, if region node 1 and region node 2 are connected, the car can travel from region node 1 to region node 2, or vice versa. The original node represents the current position of the car. The car can travel from the original node to its connected region nodes, but cannot travel from a region node to the original node, to avoid the original node interfering with subsequent path planning.

[0107] In this embodiment, the traversable cost corresponding to the regional node is used as the weight of the regional node.

[0108] Step 703: Process the topology graph using the optimal path algorithm to obtain the optimal path.

[0109] The optimal path algorithm compares the weights of the nodes in the topology graph and selects the node with the lowest weight as the target node, adding it to the optimal path. Finally, the optimal path is generated based on the target node.

[0110] In this embodiment, the acquired target image is input into a traversable cost model to obtain the traversable cost; a topology graph is constructed using the traversable cost; the topology graph is then processed using an optimal path algorithm to obtain the optimal path. The traversable cost model uses the MobileNetV3 network as its base network, which is a lightweight model that can be deployed on various small vehicles and can input target images and plan optimal paths in real time. Furthermore, this embodiment converts the target image into a topology graph, uses the predicted traversable cost as weights, and processes the topology graph using an optimal path algorithm, reducing the computational load on the driving vehicle and further improving the efficiency of generating the optimal path.

[0111] In an optional embodiment, constructing the topology graph using the traversable cost includes:

[0112] The traversable cost is smoothed by Gaussian filtering and normalized to obtain the weights corresponding to the nodes in the region.

[0113] The weights are associated with the regional nodes to establish the topology graph.

[0114] The traversable cost is then smoothed and normalized using Gaussian filtering, so that the traversable cost of each region is normalized to between 1 and 255. The Gaussian filtering smoothing is implemented using a Gaussian filter function, which is as follows:

[0115]

[0116] in, y is the standard deviation; x is the x-coordinate of the corresponding region in the target image, and y is the y-coordinate of the corresponding region in the target image.

[0117] like Figure 8As shown, in this embodiment, the overall process of obtaining the optimal path using the traversable cost model is as follows: the traversable cost model is used to predict the acquired target image to obtain the traversable cost matrix; then, the traversable cost matrix is ​​smoothed and normalized using Gaussian filtering, and the results are used as weights to construct a topology graph; finally, the optimal path algorithm is used to process the topology graph to obtain the optimal path and generate the optimal path graph.

[0118] In an optional embodiment, processing the topology graph using an optimal path algorithm to obtain the optimal path includes:

[0119] The optimal node is obtained from the original node or the adjacent regional nodes of the regional node using the optimal path algorithm;

[0120] Connect several of the optimal nodes to generate the optimal path.

[0121] like Figure 9 As shown, the topology graph has 65 nodes, with nodes numbered 0-64 being region nodes and node number 65 being the original node. Assuming the original node connects to original nodes numbered 59 and 60, if the weight of region node number 60 is lower than that of region node number 59, then region nodes numbered 64 and 60 are included in the optimal path. The next step is to calculate the node with the shortest distance to node number 60, i.e., to calculate and compare the weights of the 5 region nodes surrounding node number 60; and so on, calculating the weights of subsequent region nodes from their 5 surrounding region nodes to find the shortest distance region node. Finally, connecting all the calculated region nodes yields the optimal path with the lowest traversable cost. Figure 9 The optimal path shown includes the original node numbered 64 and the regional nodes numbered 60, 52, 44, and 35.

[0122] Optionally, the target image and the topology map can be divided into upper and lower parts. In the topology map, the region nodes in the upper part are designated as distant nodes, and the region nodes in the lower part are designated as near nodes. For example, ... Figure 9 Nodes numbered 0-31 are designated as distant nodes; nodes numbered 32-63 are designated as near nodes. Since most distant nodes in the topology map corresponding to the target image captured by the RGB camera mounted on the car are distant targets such as the sky, this embodiment only needs to predict the optimal path from the near nodes.

[0123] Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application also provide a traversable cost model training device, such as... Figure 10 As shown, the device includes:

[0124] The normalized vegetation model training module 1001 trains a normalized vegetation network based on the pre-collected first image and the corresponding labeled image to obtain a normalized vegetation model.

[0125] The labeled image is an image labeled according to the normalized vegetation index of the first image.

[0126] Most outdoor environments are natural environments. Within these environments, the Normalized Difference Vegetation Index (NDVI) can be used to distinguish between vegetated areas and passable surfaces. It's a spectral transform used to monitor vegetation characteristics in remote sensing. Because near-infrared light is not absorbed by plants and may even be strongly reflected, while red light is absorbed, NDVI measures the difference between near-infrared and red light to quantify vegetated areas. Its range is between -1 and 1. A negative value indicates water within the area. Conversely, a value close to 1 indicates green leaves or grassland. A value close to zero indicates the absence of vegetation and may represent roads or urbanized areas.

[0127] As a first example, such as Figure 2 As shown, a first image and a corresponding labeled image can be pre-acquired. Optionally, the acquired first image can be an RGB image. The RGB image can be pre-divided into multiple regions, and then the normalized vegetation index (NVI) corresponding to each region can be obtained. The NVI of each region can then be used to generate a labeled image. Specifically, obtaining the labeled image corresponding to the first image is as follows:

[0128] A visible light camera is used to acquire the red band corresponding to each region in the first image, and a near-infrared camera is used to acquire the near-infrared band corresponding to each region in the first image.

[0129] The Normalized Difference Vegetation Index (NDVI) is obtained using the formula:

[0130]

[0131] NDVI is the Normalized Difference Vegetation Index. For in pixels Reflectivity in the near-infrared band; For in pixels Reflectivity in the red band.

[0132] The first image is labeled using the normalized vegetation index calculated according to the above steps to generate the labeled image. It should be noted that in this embodiment, there can be multiple first images and multiple labeled images.

[0133] like Figure 2 The first image and its corresponding labeled image can be input into the normalized vegetation network for training. Optionally, the normalized vegetation network uses the SGDRegressor model from the scikit-learn library as the base model. The SGDRegressor model mainly uses stochastic gradient descent to perform linear regression to predict vegetation pixels.

[0134] The first image obtained by the above method and its corresponding labeled image can be input into the SGDRegressor model for training, and finally a normalized vegetation model can be obtained.

[0135] The image processing module 1002 processes the second image using the normalized vegetation model to obtain the vegetation pixels of the second image.

[0136] As a second example, the acquired second image can be input into the normalized vegetation model trained above. The normalized vegetation model outputs the vegetation pixels corresponding to each region in the second image. Optionally, the second image can be divided into multiple regions, such as 8×8 rectangular regions, and then the second image can be input into the normalized vegetation model to output the vegetation pixels corresponding to each region. Optionally, the second image can be the same as the first image or different from the first image.

[0137] The traversable cost model training module 1003 trains a traversable cost network based on the vegetation pixels and the corresponding depth values ​​to obtain a traversable cost model; the depth values ​​are obtained by pre-collecting the depth map of the second image.

[0138] The depth value is obtained by pre-acquiring a depth map of the second image.

[0139] As a third example, the second image can be segmented into 8×8 rectangular regions. In addition, this embodiment can also use a depth camera to acquire a depth map corresponding to the second image, the depth map including depth values ​​corresponding to each rectangular region of the second image. Optionally, the traversable cost network uses the MobileNetV3 network as the base model, and can be trained using vegetation pixels and depth values ​​to generate a traversable cost model that can directly predict the traversable cost of vegetation.

[0140] In this embodiment, a normalized vegetation network is trained based on a pre-acquired first image and its corresponding labeled image to obtain a normalized vegetation model. The labeled image is an image labeled according to the normalized vegetation index of the first image. The normalized vegetation model is used to process a second image to obtain vegetation pixels in the second image. Based on the vegetation pixels and their corresponding depth values, a traversability cost network is trained to obtain a traversability cost model. The depth values ​​are obtained by pre-acquired depth maps of the second image. The method in this embodiment can input the acquired image into the traversability cost model in real time, predict the traversability cost of vegetation in real time using the traversability cost model, and plan feasible paths based on the predicted traversability cost, thus ensuring driving safety.

[0141] In one optional embodiment of this application, the traversable cost model training module 1003 includes:

[0142] The first submodule uses the vegetation pixels and the corresponding depth values ​​to obtain the traversable cost matrix of the second image;

[0143] The second submodule trains the traversable cost network using the second image and the traversable cost matrix graph to obtain the traversable cost model.

[0144] In one optional embodiment of this application, the first submodule includes:

[0145] The first unit converts the vegetation pixels into visible light atmospheric impedance index, and uses the visible light atmospheric impedance index to obtain the vegetation cost:

[0146]

[0147] in, For in pixels Visible light atmospheric impedance index at that location;

[0148] The second unit uses the depth value and the maximum depth value in the depth map to obtain the depth cost:

[0149]

[0150] in, This is the maximum depth value in the depth map. For in pixels The depth value at that location.

[0151] The third unit uses the vegetation cost and the depth cost to obtain the traversable cost matrix.

[0152] In one optional embodiment of this application, the third unit includes:

[0153] The first sub-unit uses the vegetation cost and the depth cost to obtain the far-field cost:

[0154]

[0155] And, near-field cost:

[0156]

[0157] The second sub-unit obtains the traversability cost using the far-field cost and the near-field cost:

[0158]

[0159] The third subunit generates the traversable cost matrix based on the traversable cost of all regions in the second image;

[0160] in, (0, 10) -6 α and β are constants between (0, 1); α and β are constants between (0, 1).

[0161] The traversable cost model training device provided in this application embodiment can achieve... Figures 1 to 9 The various processes implemented in the method embodiments are not described in detail here to avoid repetition.

[0162] The traversable cost model training device of this application embodiment can execute the traversable cost model training method provided in this application embodiment. The implementation principle is similar. The actions performed by each module and unit in the traversable cost model training device in each embodiment of this application are corresponding to the steps in the traversable cost model training method in each embodiment of this application. For detailed functional descriptions of each module of the traversable cost model training device, please refer to the descriptions in the corresponding traversable cost model training methods shown above. They will not be repeated here.

[0163] Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application also provide a path generation device, such as... Figure 11 As shown, the device includes:

[0164] The image processing module 1101 inputs the acquired target image into the traversable cost model to obtain the traversable cost; wherein the traversable cost model is trained according to the traversable cost model training method described in this application.

[0165] This embodiment segments the target image into multiple regions, defining the traversable cost of each region as its fully common cost (fc) value. The target image is then input into a traversable cost model trained using the above method for prediction, yielding the fc value for each region, which is the traversable cost. Optionally, the traversable cost model uses the MobileNetV3 network as its base model. MobileNetV3 is a typical lightweight network, suitable for deployment, and capable of quickly predicting the traversable cost from the target image captured by the camera.

[0166] The topology graph generation module 1102 constructs a topology graph using the traversable cost.

[0167] The topology diagram includes one or more region nodes and an original node; the region nodes refer to regions obtained by segmenting the target image; adjacent region nodes are bidirectionally connected; and the original node is unidirectionally connected to any one or more region nodes.

[0168] Assuming the target image is segmented into 8×8 rectangular regions, there are 64 regions in total. These 64 regions are designated as 64 region nodes. Each node is bidirectionally connected to its adjacent nodes, indicating that a car can travel between two region nodes. For example, if region node 1 and region node 2 are connected, the car can travel from region node 1 to region node 2, or vice versa. The original node represents the current position of the car. The car can travel from the original node to its connected region nodes, but cannot travel from a region node to the original node, to avoid the original node interfering with subsequent path planning.

[0169] In this embodiment, the traversable cost corresponding to the regional node is used as the weight of the regional node.

[0170] Data processing module 1103 processes the topology graph using an optimal path algorithm to obtain the optimal path;

[0171] The optimal path algorithm compares the weights of the nodes in the topology graph and selects the node with the lowest weight as the target node, adding it to the optimal path. Finally, the optimal path is generated based on the target node.

[0172] In this embodiment, the acquired target image is input into a traversable cost model to obtain the traversable cost; a topology graph is constructed using the traversable cost; the topology graph is then processed using an optimal path algorithm to obtain the optimal path. The traversable cost model uses the MobileNetV3 network as its base network, which is a lightweight model that can be deployed on various small vehicles and can input target images and plan optimal paths in real time. Furthermore, this embodiment converts the target image into a topology graph, uses the predicted traversable cost as weights, and processes the topology graph using an optimal path algorithm, reducing the computational load on the driving vehicle and further improving the efficiency of generating the optimal path.

[0173] In one optional embodiment of this application, the topology diagram generation module 1102 includes:

[0174] The first generation submodule performs Gaussian filtering smoothing and normalization on the traversable cost to obtain the weights corresponding to the region nodes.

[0175] The second generation submodule associates the weights with the region nodes to establish the topology graph.

[0176] In one optional embodiment of this application, the data processing module 1103 includes:

[0177] The first processing submodule first uses an optimal path algorithm to obtain the optimal node from the original node or the adjacent regional nodes of the regional node;

[0178] The second processing submodule connects several of the optimal nodes to generate the optimal path.

[0179] The path generation device provided in this application embodiment can achieve... Figures 1 to 9 The various processes implemented in the method embodiments are not described in detail here to avoid repetition.

[0180] The path generation device of this application embodiment can execute the path generation method provided in this application embodiment. The implementation principle is similar. The actions performed by each module and unit in the path generation device in each embodiment of this application are corresponding to the steps in the path generation method in each embodiment of this application. For detailed functional descriptions of each module of the path generation device, please refer to the descriptions in the corresponding path generation methods shown above. They will not be repeated here.

[0181] Based on the same principles as the methods shown in the embodiments of this application, this application also provides an electronic device, which may include, but is not limited to, a processor and a memory; the memory is used to store computer programs; the processor is used to execute the traversable cost model training method shown in any optional embodiment of this application by calling the computer program. Compared with the prior art, the traversable cost model training method provided in this application trains a normalized vegetation network based on a pre-acquired first image and a corresponding labeled image to obtain a normalized vegetation model; the labeled image is an image labeled according to the normalized vegetation index of the first image; the normalized vegetation model is used to process a second image to obtain vegetation pixels of the second image; a traversable cost network is trained based on the vegetation pixels and the corresponding depth values ​​to obtain a traversable cost model; the depth values ​​are obtained by pre-acquired depth maps of the second image. The method in this embodiment can input the acquired image into the traversable cost model in real time, predict the traversable cost of vegetation in real time through the traversable cost model, and plan a feasible path based on the predicted traversable cost, which can ensure driving safety of vehicles.

[0182] In an alternative embodiment, an electronic device, such as Figure 12 As shown, Figure 12 The illustrated electronic device 1200 can be a server, including a processor 1201 and a memory 1203. The processor 1201 and the memory 1203 are connected, for example, via a bus 1202. Optionally, the electronic device 1200 may also include a transceiver 1204. It should be noted that in practical applications, the transceiver 1204 is not limited to one unit, and the structure of this electronic device 1200 does not constitute a limitation on the embodiments of this application.

[0183] Processor 1201 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1201 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0184] Bus 1202 may include a pathway for transmitting information between the aforementioned components. Bus 1202 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1202 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 12 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0185] The memory 1203 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0186] The memory 1203 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 1201. The processor 1201 is used to execute the application code stored in the memory 1203 to implement the content shown in the foregoing method embodiments.

[0187] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 12 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0188] The server provided in this application can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0189] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0190] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0191] It should be noted that the computer-readable storage medium described above in this application can also be a computer-readable signal medium or a combination of computer-readable storage media and computer-readable storage media. Computer-readable storage media can be, for example,—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0192] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0193] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0194] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the traversable cost model training method and path generation method provided in the various optional implementations described above.

[0195] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0196] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0197] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules are not necessarily limiting in certain circumstances; for example, a topology graph generation module can also be described as "a topology graph generation module that constructs a topology graph using the traversable cost".

[0198] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A traversable cost model training method, characterized in that, The method includes: A normalized vegetation network is trained based on a pre-collected first image and its corresponding labeled image to obtain a normalized vegetation model; the labeled image is an image labeled with the normalized vegetation index of the first image. The second image is processed using the normalized vegetation model to obtain the vegetation pixels of the second image; Based on the vegetation pixels and their corresponding depth values, a traversable cost network is trained to obtain a traversable cost model; the depth values ​​are obtained by pre-collecting the depth map of the second image. The step of training a traversable cost network based on the vegetation pixels and their corresponding depth values ​​to obtain a traversable cost model includes: The vegetation pixels are converted into visible light atmospheric impedance index, and the vegetation cost is obtained using the visible light atmospheric impedance index: , For in pixels Visible light atmospheric impedance index at that location; The depth cost is obtained by comparing the depth value with the maximum depth value in the depth map. , This is the maximum depth value in the depth map. The depth value; The far-field cost is obtained by using the vegetation cost and the depth cost. and near-field cost ， ； The traversable cost is obtained using the far-field cost and the near-field cost: , For (0, α and β are constants between (0, 1); Based on the traversable costs of all regions in the second image, generate the traversable cost matrix. The traversable cost network is trained using the second image and the traversable cost matrix graph to obtain the traversable cost model.

2. A path generation method, characterized in that, The method includes: The acquired target image is input into the traversable cost model to obtain the traversable cost; wherein, the traversable cost model is trained by the traversable cost model training method according to claim 1; A topology graph is constructed using the traversable cost; The optimal path is obtained by processing the topology graph using an optimal path algorithm. The topology diagram includes one or more regional nodes and an original node; the regional nodes are obtained by segmenting the target image; adjacent regional nodes are bidirectionally connected, and the original node is unidirectionally connected to any one or more regional nodes.

3. The path generation method according to claim 2, characterized in that, The construction of the topology graph using the traversable cost includes: The traversable cost is smoothed by Gaussian filtering and normalized to obtain the weights corresponding to the nodes in the region. The weights are associated with the regional nodes to establish the topology graph.

4. The path generation method according to claim 2, characterized in that, The process of processing the topology graph using an optimal path algorithm to obtain the optimal path includes: The optimal node is obtained from the original node or the adjacent regional nodes of the regional node using the optimal path algorithm; Connect several of the optimal nodes to generate the optimal path.

5. A traversable cost model training device implemented according to the traversable cost model training method according to claim 1, characterized in that, The device includes: The normalized vegetation model training module trains a normalized vegetation network based on a pre-collected first image and the corresponding labeled image to obtain a normalized vegetation model; the labeled image is an image labeled according to the normalized vegetation index of the first image. The image processing module processes the second image using the normalized vegetation model to obtain the vegetation pixels of the second image; The traversable cost model training module trains a traversable cost network based on the vegetation pixels and their corresponding depth values ​​to obtain a traversable cost model; the depth values ​​are obtained by pre-collecting the depth map of the second image.

6. A path generation device, characterized in that, The device includes: The image processing module inputs the acquired target image into the traversable cost model to obtain the traversable cost; wherein, the traversable cost model is trained by the traversable cost model training method according to claim 1; The topology graph generation module constructs a topology graph using the traversable cost. The data processing module processes the topology graph using an optimal path algorithm to obtain the optimal path; The topology diagram includes one or more regional nodes and an original node; the regional nodes are obtained by segmenting the target image; adjacent regional nodes are bidirectionally connected, and the original node is unidirectionally connected to any one or more regional nodes.

7. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method of claim 1.

Images