A path generation method, a neural network training method, and a data generation method.

Abstract

This application provides a path generation method, a neural network training method, and a data generation method. The path generation method includes: acquiring multiple frames of images and / or point clouds of a site; processing the multiple frames of images and / or point clouds using a neural network to obtain a risk field and a first travel path within the site, to instruct vehicles to travel within the site. The neural network includes a first network, a second network, and a third network. The output of the first network is the input to the second and third networks. The head of the second network is used for data classification, and the head of the third network is used for data segmentation. The output of the second network is the risk field, and the output of the third network is the first travel path. The risk field represents the terrain changes of the site in the form of a field. The risk field represents a joint relationship between terrain obstacles and the road surface, enabling the accurate perception of the terrain obstacles' positions on the road surface, which is beneficial for improving obstacle avoidance success rates.

Description

Technical Field

[0001] This application relates to the field of intelligent driving, and in particular to a path generation method, a neural network training method, and a data generation method. Background Technology

[0002] Autonomous driving technology is a comprehensive technology based on artificial intelligence. With its development, autonomous driving technology has been applied in many scenarios, with open-pit mines being one of the earliest examples. In open-pit mines, vehicles carry heavy loads, and the overburden consists mainly of mudstone and siltstone, resulting in significant dust and the potential for rockfalls and spillage on the roads, creating potholes and ruts. Vehicles traversing these sections are prone to overturning, tire blowouts, and other malfunctions.

[0003] Currently, obstacles and road surfaces are represented using detection frames. In this method, obstacles and road surfaces exist independently. In open-pit mine autonomous driving scenarios, it is difficult to detect the location of terrain obstacles such as fallen rocks, spills, and tire tracks on the road surface, which can easily lead to missed or false detections of terrain obstacles, affecting the vehicle's obstacle avoidance success rate. Summary of the Invention

[0004] This application provides a path generation method, a neural network training method, and a data generation method. It abandons the method of representing obstacles with detection boxes and adopts a risk field to represent obstacles. Since terrain obstacles and the road surface coexist in the risk field, the location of terrain obstacles on the road surface can be perceived through the risk field, which helps to improve the detection rate and detection accuracy of terrain obstacles, thereby improving the obstacle avoidance success rate.

[0005] In a first aspect, this application provides a path generation method, comprising: acquiring multiple frames of images of a site; each frame of the multiple frames is used to record at least a portion of the site; processing the multiple frames of images through a neural network to obtain a risk field of the site and a first passage path in the site, so as to indicate the driving of vehicles in the site; wherein the neural network includes a first network, a second network and a third network, the output of the first network is the input of the second network and the third network, the head of the second network is used to classify data, the head of the third network is used to segment data, the output of the second network is the risk field, the output of the third network is the first passage path, and the risk field represents the terrain changes of the site in the form of a field.

[0006] In the path generation method provided in this application, terrain obstacles and the road surface coexist in the risk field, enabling the perception of the terrain obstacle's position on the road surface. This helps improve the detection rate and accuracy of terrain obstacles, thereby increasing the obstacle avoidance success rate. Furthermore, the neural network outputs both the first passage path and the risk field, enriching the reference information used for obstacle avoidance and further contributing to the improved obstacle avoidance success rate.

[0007] As one possible implementation, the terrain changes represented by a risk location are related to changes in elevation, gradient, and the degree of bumpiness experienced by vehicles. Changes in elevation and gradient indicate how terrain obstacles change within the site, while the degree of bumpiness reflects vehicle traffic conditions. Characterizing the risk field based on these changes helps in accurately identifying the location of terrain obstacles on the road surface. Of course, in other possible implementations, the terrain changes represented by a risk location can also be related to changes in elevation and the degree of bumpiness experienced by vehicles, or to changes in both elevation and gradient.

[0008] As one possible implementation, multi-frame images are images taken at multiple points in time from the same viewpoint, or multi-frame images are images taken from multiple viewpoints.

[0009] As one possible implementation, the loss functions used by the second network include Focal Loss, Cross-Entropy Loss, or Mean Squared Error Loss, while the loss functions used by the third network include Dice Loss, Cross-Entropy Loss, or Mean Squared Error Loss.

[0010] As one possible implementation, the risk field and / or the first passage path can be displayed.

[0011] After the neural network generates the risk field and the first pass path, both can be displayed simultaneously on the vehicle's visual interface. This allows users to optimize the first pass path by referring to the risk field, thereby improving the obstacle avoidance success rate. This method is suitable for scenarios where terrain obstacles have a significant impact on the pass path.

[0012] After the neural network generates the risk field and the first passable path, only the risk field can be displayed on the vehicle's visual interface. This allows users to refer to the risk field to plan their passage and improve the obstacle avoidance success rate. This method is suitable for scenarios where terrain obstacles have a significant impact on the passage path.

[0013] After the neural network generates the risk field and the first travel path, only the first travel path can be displayed to the user on the vehicle's visual interface, allowing the user to drive according to the first travel path. This method is suitable for scenarios where terrain obstacles have little impact on the travel path.

[0014] As one possible approach, a second access path is displayed on a site map based on topographical changes characterizing the risky location.

[0015] After the neural network generates the risk field and the first travel path, a second travel path is generated based on the terrain changes represented by the risk field. Only the second travel path can be displayed on the vehicle's visual interface, allowing the user to navigate according to it and improving obstacle avoidance success rate. This method is suitable for scenarios where terrain obstacles have a significant impact on the travel path.

[0016] After generating the second travel path, both the first and second travel paths can be simultaneously displayed on the vehicle's visual interface. This allows users to plan their travel path based on the first and second paths, further improving obstacle avoidance success rates. This implementation is suitable for scenarios where terrain obstacles significantly impact the travel path. Users can optimize the first travel path by referring to the second travel path, or vice versa, to generate a suitable travel path.

[0017] After generating the second travel path, the risk field, the first travel path, and the second travel path are simultaneously displayed on the vehicle's visual interface. This allows users to plan their travel path based on the risk field, the first travel path, and the second travel path, further improving obstacle avoidance success rate. This implementation method is suitable for scenarios where terrain obstacles have a significant impact on the travel path. Users can perform comprehensive planning using the risk field, the first travel path, and the second travel path to generate a travel path for driving.

[0018] The display of the risk field, the first passage path, and the second passage path is also applicable to the client side, making it convenient for users on the client side to view one or more of the risk field, the first passage path, and the second passage path, thereby making it convenient for clients on the client side to control the vehicle based on one or more of the displayed risk field, the first passage path, and the second passage path.

[0019] One possible implementation involves using a heuristic algorithm to perform path search in a risk field to obtain a second viable path. This heuristic algorithm includes parameters representing one or more costs among distance cost, angle cost, time cost, and collision cost. Path search is performed in the risk field based on one or more cost terms to find the second viable path with the lowest risk, thereby improving obstacle avoidance success rate.

[0020] Secondly, this application provides a method for training a neural network, comprising: acquiring multiple frames of sample images of a site and / or sample point clouds of the site; each frame of the multiple frames of sample images is used to record at least a portion of the site; processing the multiple frames of sample images and / or sample point clouds through a neural network to obtain a risk field of the site and a first passage path in the site; wherein the neural network includes a first network, a second network, and a third network, the output of the first network is the input of the second network and the third network, the head of the second network is used for classifying data, the head of the third network is used for segmenting data, the output of the second network is the risk field, the output of the third network is the first passage path, and the risk field represents the terrain changes of the site in the form of a field; supervising the training of the neural network through the sample risk field and sample passage paths to obtain the trained neural network; the sample risk field and sample passage paths are labeled data of the multiple frames of sample images and / or sample point clouds of the site.

[0021] In the neural network training method provided in this application, the method of representing obstacles using bounding boxes is abandoned. Instead, a risk field is used to represent terrain obstacles. Since terrain obstacles and the road surface coexist in the risk field, the location of terrain obstacles on the road surface can be perceived through the risk field, which helps to improve the detection rate and accuracy of terrain obstacles, thereby improving the obstacle avoidance success rate. Furthermore, the neural network outputs both the first passage path and the risk field, enriching the reference information used for obstacle avoidance and further contributing to improving the obstacle avoidance success rate.

[0022] As one possible implementation, the terrain changes represented by a risk location are related to changes in elevation, gradient, and the degree of bumpiness experienced by vehicles. Changes in elevation and gradient indicate how terrain obstacles change within the site, while the degree of bumpiness reflects vehicle traffic conditions. Characterizing the risk field based on these changes helps in accurately identifying the location of terrain obstacles on the road surface. Of course, in other possible implementations, the terrain changes represented by a risk location can also be related to changes in elevation and the degree of bumpiness experienced by vehicles, or to changes in both elevation and gradient.

[0023] As one possible implementation, multi-frame images are images taken at multiple points in time from the same viewpoint, or multi-frame images are images taken from multiple viewpoints.

[0024] As one possible implementation, the loss function used by the second network includes Focal Loss, and the loss function used by the third network includes Dice Loss.

[0025] Thirdly, this application provides a data generation method, comprising: acquiring multiple first sample point clouds of a site; processing the multiple first sample point clouds to obtain a first elevation map of each first sample point cloud; fusing the multiple first elevation maps to generate a second elevation map; generating a risk field of the site based on the second elevation map; the risk field characterizing the terrain changes of the site in the form of a field; and performing path search in the risk field to obtain the access paths of the site.

[0026] In the data generation method provided in this application, the method of representing obstacles by detection boxes is abandoned, and a risk field is used to represent terrain obstacles. Since terrain obstacles and the road surface exist together in the risk field, the position of terrain obstacles on the road surface can be perceived through the risk field, which helps to improve the detection rate and detection accuracy of terrain obstacles, thereby improving the obstacle avoidance success rate.

[0027] As one possible implementation, before fusing multiple first elevation maps to generate a second elevation map, the process includes: filtering out interfering regions in the first elevation maps. These interfering regions include interfering points, which are points in the first elevation maps whose weighted variance and gradient values ​​are greater than a threshold. This method filters out interfering regions, increasing the sensing distance for terrain obstacles even in the presence of interference such as sand and dust, thereby further improving the detection rate and accuracy of terrain obstacles.

[0028] One possible implementation involves generating a risk field based on a second elevation map, including: calculating the height changes, gradient changes, and bump levels at multiple points on the second elevation map; calculating the risk values ​​at these multiple points based on the height changes, gradient changes, and bump levels; and generating the risk field based on the risk values ​​at these multiple points. Height and gradient changes represent the changes in terrain obstacles within the site, while the bump level during vehicle movement represents the vehicle's passage conditions within the site. Characterizing the risk field based on the changes in terrain obstacles and the vehicle's passage conditions helps in perceiving the accurate location of terrain obstacles on the road surface. Of course, in other possible implementations, the terrain changes represented by the risk field can also be related to the height changes and bump levels during vehicle movement, or to both height and gradient changes.

[0029] As one possible implementation, path search is performed in the risk field to determine the passageways to the site, specifically including:

[0030] A heuristic algorithm is used to perform path search in a risky environment to determine the passable path. The heuristic algorithm includes parameters representing one or more costs among distance cost, angle cost, time cost, and collision cost. Path search is performed in the risky environment based on one or more cost terms to find the second passable path with the lowest risk, thereby improving the obstacle avoidance success rate.

[0031] Fourthly, this application provides a path generation apparatus for executing the method in the first aspect or any possible implementation of the first aspect. Specifically, the apparatus includes modules for executing the method in the first aspect or any possible implementation of the first aspect.

[0032] Fifthly, this application provides a neural network training apparatus for executing the method in the second aspect or any possible implementation thereof. Specifically, the apparatus includes modules for executing the method in the second aspect or any possible implementation thereof.

[0033] Sixthly, this application provides a data generation apparatus for performing the method in the third aspect or any possible implementation thereof. Specifically, the apparatus includes modules for performing the method in the third aspect or any possible implementation thereof.

[0034] In a seventh aspect, this application provides a vehicle that includes the device of the sixth aspect.

[0035] Eighthly, this application provides an electronic device, comprising: at least one memory for storing a program; at least one processor for executing the program stored in the memory; wherein, when the program stored in the memory is executed, the processor is configured to execute a method of the first aspect, the second aspect, the third aspect, any possible implementation of the first aspect, a possible implementation of the second aspect, or a possible implementation of the third aspect.

[0036] Ninthly, this application provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it performs the method of the first aspect, the second aspect, the third aspect, any possible implementation of the first aspect, a possible implementation of the second aspect, and a possible implementation of the third aspect.

[0037] In a tenth aspect, this application provides a computer program product having a computer program stored thereon, wherein when the computer program is executed by a processor, it performs the first aspect, the second aspect, the third aspect, any possible implementation of the first aspect, a possible implementation of the second aspect, and a possible implementation of the third aspect. Attached Figure Description

[0038] Figure 1 This is an example diagram of a path planning system in the prior art;

[0039] Figure 2 This is a system architecture example diagram provided in this application;

[0040] Figure 3This is an example diagram of the system modules provided in this application;

[0041] Figure 4 This is an example diagram of the system framework provided in this application;

[0042] Figure 5 This is a flowchart of the path generation method provided in this application;

[0043] Figure 6 This is a flowchart of a processing method for a first network provided in this application;

[0044] Figure 7 This is a flowchart of the neural network training method provided in this application;

[0045] Figure 8 This is a flowchart of another first network processing method provided in this application;

[0046] Figure 9 This is a flowchart of the data generation method provided in this application;

[0047] Figure 10 This is a flowchart of the training method for the sparse Gaussian model provided in this application;

[0048] Figure 11 This is a functional block diagram of the path generation device provided in this application;

[0049] Figure 12 This is a functional block diagram of the neural network training device provided in this application;

[0050] Figure 13 This is a functional block diagram of the data generation device provided in this application;

[0051] Figure 14 This is an example diagram of the electronic device provided in this application. Detailed Implementation

[0052] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0053] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to show the relevant concepts in a specific manner.

[0054] First, some of the terms used in the embodiments of this application will be introduced to facilitate understanding by those skilled in the art.

[0055] (1) Elevation map

[0056] An elevation map, also known as a topographic map or landform map, is a type of map. By simulating the elevation information of the Earth's surface, an elevation map displays the three-dimensional features of the terrain and can show the undulations of the Earth's surface.

[0057] (2) Sparse Gaussian process

[0058] A sparse Gaussian process is a variant of the standard Gaussian process designed to handle large-scale datasets, improving computational efficiency by reducing computational complexity. The standard Gaussian process has high computational complexity, especially when dealing with large datasets, while the sparse Gaussian process significantly reduces computational complexity by approximating the data using a finite number of "inducible points." The sparse Gaussian model in this application is a sparse Gaussian process.

[0059] (3) Obstacles

[0060] An obstacle is an object that occupies a drivable road surface and affects the movement of an autonomous vehicle. Since any type of object (not certain specific or common types) can be an obstacle, obstacles can also be called general obstacles. In the embodiments of this application, terrain obstacles refer to objects such as fallen rocks, debris, and ruts that can cause changes in the terrain of the drivable road surface and affect the movement of parked vehicles.

[0061] (4) Risk Field

[0062] In physics, a field is a physical quantity that varies with spacetime. A risk field is a field used to characterize the degree of influence of various elements in the vehicle-driving environment on driving risks.

[0063] In this embodiment, the risk field represents the terrain changes of the site in the form of a field. It quantifies the risk level by simulating factors such as the relative position and speed between the vehicle and terrain obstacles, so as to help the intelligent driving system make safe decisions.

[0064] (5) Multiscale features

[0065] Multi-scale features refer to features extracted at different spatial scales. In image processing, scale is usually associated with the size of the observed object. By extracting features at different scales, information at different levels and granularities can be captured.

[0066] The following sections introduce the solutions related to this application from different perspectives.

[0067] See Figure 1 This diagram illustrates an example of a path planning system in existing technology. Current path planning typically involves global path planning based on a global map and target points, calculating an initial path, performing collision modeling based on a perception module, generating a local map based on the collision modeling results, optimizing the path in corresponding areas of the initial path based on the local map, generating an optimized local path, and then sending the local path to the vehicle. The vehicle can then drive according to the local path to traverse areas where obstacles exist. The perception module can contain data observed by sensors, vehicle pose, etc., and the local map can be a 3D grid or a symbolic distance field, etc. Methods for optimizing the initial path include the Dynamic Window Approach (DWA) and differential flat motion constraints.

[0068] The first solution related to this application generates a priori obstacle avoidance path for pothole obstacles by establishing a semantic map as prior information, and then improves the confidence of pothole obstacles based on real-time perception to achieve real-time obstacle avoidance path update.

[0069] The second approach related to this application proposes a safety constraint modeling of the three-dimensional collision potential field of the ontology, which can generate a geometrically occupied grid within the effective observation range of the lidar, and achieve path optimization by combining the obstacle avoidance path generated based on the global map and the target point.

[0070] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application.

[0071] Figure 2 This is a system architecture example diagram provided in this application, including a vehicle terminal 100, a cloud device 200, and a client 300. The vehicle terminal 100 includes an image acquisition module 101, a radar module 102, a positioning module 103, a communication module 104, and an inertial navigation module 105.

[0072] The image acquisition module 101 is used to acquire RGB images and / or depth images from the vehicle end, including sample images for training and images for inference, wherein both the sample images and the images for inference include RGB images and / or depth images.

[0073] The radar module 102 is used to acquire point clouds. The radar module 102 can be, for example, a lidar, a high-precision sensor of the vehicle end 100 that acquires three-dimensional point cloud data of the vehicle's surrounding environment by emitting a laser beam and receiving its reflection. The point cloud acquired by the radar module 102 includes point clouds used for training and point clouds used for inference.

[0074] The positioning module 103 is used to obtain the vehicle's position on the map. The positioning module 103 integrates multiple technologies to achieve accurate vehicle positioning. Specifically, the positioning module 103 calculates the vehicle's precise position, speed, and attitude information by receiving GPS satellite signals and using an inertial measurement unit (IMU).

[0075] The communication module 104 is used to enable communication between the vehicle terminal 100 and the cloud device 200. Specifically, the communication module 104 uploads the sample images and sample point clouds collected by the vehicle terminal 100 to the cloud device 200, and sends the network trained by the cloud device 200 to the vehicle terminal 100.

[0076] The inertial navigation module 105 is used to measure the vehicle body pose of the vehicle end 100. Specifically, the inertial navigation module 105 can calculate the position, velocity and attitude information of the vehicle end 100 through the inertial measurement unit, and provide this information to the positioning module 103 for precise positioning of the vehicle.

[0077] The vehicle terminal 100 includes an image acquisition module 101, a radar module 102, a positioning module 103, a communication module 104, and an inertial navigation module 105. The vehicle terminal 100 is a vehicle that travels on a road and includes the aforementioned modules. It should be noted that the technical solutions provided in this application embodiment are also applicable to robots that include the aforementioned modules; that is, the vehicle terminal 100 in this application can be replaced by a robot.

[0078] The cloud-based unit 200 receives sample images and sample point clouds sent by the vehicle-side unit 100, generates labeled data based on the sample point clouds, and trains the network based on the labeled data. The labeled data includes sample risk fields and sample travel paths.

[0079] The client 300 can be a user's personal computer, laptop, tablet, or smartphone, etc. The user can use the client 300 to view the risk area and travel route of the vehicle 100, and can also control the vehicle 100 through the client 300. It should be noted that the user on the client 300 and the user on the vehicle 100 can be the same person or different people. Unless otherwise specified, the user referred to in this embodiment includes both the user on the vehicle 100 and the user on the client 300.

[0080] Figure 3 This is an example diagram of the system modules provided in this application. The input is the data collected by the sensor. The processing includes a data generation process and a training / inference process. The data generation process includes five parts: aligning the gravity direction, sparse Gaussian prediction, updating the elevation map, generating the risk field, and searching the BEV (Bird's Eye View) spatial path.

[0081] The data generation process is used to generate labeled training data. During data generation, aligning the gravity direction involves transforming the sample point cloud to a gravity coordinate system to align with the gravity direction; sparse Gaussian prediction refers to obtaining the mean and variance of multiple points in space through a sparse Gaussian model; updating the elevation map refers to fusing the elevation maps of multiple sample point clouds to generate a more accurate elevation map; generating the risk field refers to generating a risk field based on the updated elevation map, which is located in the BEV space; searching for a path in the BEV space refers to searching for the optimal obstacle avoidance path in the risk field.

[0082] During training / inference, the output of the neural network includes two branches: a risk field and a travel path. Users can refer to the risk field to optimize the travel path to improve the obstacle avoidance success rate.

[0083] Figure 4 This is an example diagram of the system framework provided in this application, including a data generation framework and a training / inference framework. In the data generation framework, gravity alignment calculations are performed based on the acceleration from IMU inference and data acquired by radar to obtain a localized sample point cloud. The sample point cloud is discretized, generating corresponding grids in the BEV space, thus transforming it to the BEV space. The mean and variance of points corresponding to multiple grids in the BEV space are calculated using radial basis functions, filtering out interference points in the BEV space. An elevation map of each sample point cloud is generated based on the mean and variance of multiple points. Elevation maps of multiple sample point clouds can be fused to generate a more accurate elevation map. The elevation map is calculated based on three cost terms: altitude change, gradient change, and vehicle bumpiness; or two cost terms: altitude change and vehicle bumpiness; or two cost terms: altitude change and gradient change, to generate a risk field. Based on a heuristic algorithm, a bicycle model is used to search within the risk field to generate a travel path. It should be noted that the risk field and travel path generated in the data generation framework are labeled data used for training. Of course, in other possible implementations, the terrain changes represented by the risk site can also be related to changes in site elevation and the degree of bumpiness when vehicles travel, or to changes in site elevation and gradient.

[0084] In the training / inference framework, the input is multiple frames of images. It should be noted that the input can also be point clouds, or a combination of images and point clouds. Figure 4Taking a multi-frame image as an example, the first network in the neural network calculates the projection of the 3D BEV space onto the 2D image to obtain BEV features. These BEV features are then processed to generate multi-scale features. The multi-scale features are the inputs to the second and third networks in the neural network. The second network generates a risk field based on the multi-scale features, and the third network generates a travel path based on the multi-scale features. The travel path and the risk field reside in the BEV space corresponding to the same site.

[0085] The technical solution for the inference side is described in detail below. This solution is applied to the vehicle side 100.

[0086] See Figure 5 This application provides a route generation method applied to a vehicle, which includes the following steps:

[0087] S501. Acquire multiple frames of images of the site and / or point clouds of the site.

[0088] The vehicle's camera captures images, and the captured multi-frame images can be images taken at multiple times from the same viewpoint, or images taken from multiple viewpoints. Each frame in the multi-frame images is used to record at least a portion of the scene.

[0089] S502. Process multiple frames of images and / or the point cloud of the site using a neural network to obtain the risk field of the site and the first passage path in the site, so as to indicate the vehicle's movement in the site.

[0090] The neural network consists of a first network, a second network, and a third network. The output of the first network is the input to the second and third networks. The head of the second network is used to classify the data, and the head of the third network is used to segment the data. The output of the second network is the risk field, and the output of the third network is the first travel path. The risk field represents the terrain changes of the site in the form of a field.

[0091] The first network takes multiple frames of images as input and outputs multi-scale features. The head of the first network is used to generate spatial features. For details, please refer to [link / reference]. Figure 6 The processing of the first network includes the following sub-steps:

[0092] S601. Process multiple frames of images and / or point clouds to generate spatial features;

[0093] Spatial features are BEV features. For a multi-frame input image, subsequent images are projected onto the coordinate system of the first image to generate a temporal fusion image. Spatial features, specifically BEV features, are then generated from this temporal fusion image. The first image is the image that is first in the temporal sequence, and subsequent images are all images except the first one. When generating spatial features from the temporal fusion image, it is projected onto the BEV space. Specifically, the coordinates of the BEV space are used as the query, the pixel coordinates of the temporal fusion image are used as the key, and the pixel values ​​of the temporal fusion image are used as the value. By constructing the relationship between the query, key, and value, the temporal fusion image is projected onto the BEV space to generate spatial features. The BEV space is set according to the driving scenario.

[0094] If processing multi-frame images, subsequent images are projected onto the coordinate system of the first image to generate a temporal fused image. Spatial features are then generated based on this temporal fused image. Subsequent images are any images other than the first image from a predetermined number of images. Specifically, when generating spatial features from the temporal fused image, the coordinates in the BEV space are used as the query, the pixel coordinates of the temporal fused image are used as the key, and the pixel values ​​of the temporal fused image are used as the value. By constructing the relationship between the query, key, and value, the temporal fused image is projected onto the BEV space to generate spatial features.

[0095] If dealing with point clouds, convert the point clouds to BEV space to generate spatial features. If dealing with multiple frames of images and point clouds, merge the spatial features generated from the images and the spatial features generated from the point clouds, and generate multi-scale features from the merged features.

[0096] It should be noted that if the input is a multi-frame image or point cloud, the head of the first network used to process the multi-frame image and the head of the first network used to process the point cloud are different structures. If the input is a multi-frame image and a point cloud, the head of the first network used to process the multi-frame image and the point cloud includes the head structure of the first network used to process the multi-frame image, the head structure of the first network used to process the point cloud, and the structure for merging features.

[0097] S602. Process the spatial features to obtain multi-scale features;

[0098] The third network is a common network model that can output multi-scale features, such as ResNet and multilayer perceptron, which will not be listed here.

[0099] In step S502, both the risk field and the first travel path are represented in the BEV space. As an example, the numerical range is set to 0-1, and this range is divided into 10 equal sub-ranges, representing the intervals containing 10 risk categories. Each risk category is represented by a numerical value between 0 and 10, which can be 0, 1…9. Different risk categories represent different levels of risk. The individual grids representing different risk categories constitute the risk field. In the BEV space, the value of a path point in the first travel path can be represented by 1, and the value of a non-path point can be represented by 0. Multiple path points constitute the first travel path.

[0100] It should be noted that this embodiment can also acquire point clouds and multi-frame sample images of the site. After generating spatial features using multi-frame images, the point cloud is then converted to BEV space to obtain the corresponding spatial features. The spatial features corresponding to the multi-frame images and the spatial features corresponding to the point cloud are merged. By processing the merged features, multi-scale features are obtained.

[0101] It should be noted that this embodiment can also obtain only the point cloud of the site, transform the point cloud to the BEV space to obtain spatial features, and process the spatial features to obtain multi-scale features.

[0102] The output of the neural network consists of two branches: one for generating the risk field and the other for generating the first passable path. Each branch contains a subnetwork: a second network for generating the risk field and a third network for generating the first passable path. The second and third networks can have the same or different structures.

[0103] In the risk field generation branch, the input to the second network is the multi-scale features output by the first network, and the head of the second network classifies these multi-scale features. Within the second network, methods such as pyramiding and multi-scale temporal convolutional transforms can be used to classify the multi-scale features, and multilayer perceptrons or convolutional neural networks can be used to generate the risk field.

[0104] In the branch that generates the first travel path, the input of the third network is the multi-scale features output by the first network. The head of the third network segments the multi-scale features to obtain the first-scale features. In the third network, multi-scale segmentation algorithms, convolutional neural networks, etc., can be used to segment the multi-scale features, and multilayer perceptrons or convolutional neural networks can be used to generate the first travel path. The first-scale feature is any one of the multi-scale features.

[0105] Each scale feature contains semantic and geometric information. When generating the first travel path, the third network determines the first travel path based on the semantic and geometric information in the first scale feature. Terrain obstacles are represented using a traditional symbolic method based on geometric information. The first travel path represents the lowest-risk path for vehicle passage using symbolic obstacles, and the travel path is represented in the BEV space.

[0106] In this embodiment, the output of the neural network includes a risk field and a first travel path. The first travel path is calculated based on first-scale features, and its terrain obstacles rely on a symbolic representation. Since the road surface and the symbolically represented terrain obstacles are discretely distributed, it is difficult to perceive the accurate location of terrain obstacles on the road surface, which can easily lead to missed detections and false detections of terrain obstacles, affecting the vehicle's obstacle avoidance success rate. However, in the risk field, obstacles and the road surface coexist, allowing the accurate location of terrain obstacles on the road surface to be perceived, which helps to improve the detection rate and accuracy of terrain obstacles, thereby improving the vehicle's obstacle avoidance success rate. This embodiment, by outputting a risk field and a first travel path, allows users to refer to the risk field to optimize the first travel path to generate a travel path for driving, thereby improving the obstacle avoidance success rate.

[0107] It should be noted that the first travel path is determined based on the existing symbolization method. Although it is prone to missing or false detection of terrain obstacles, it has a fast inference speed and can meet the needs of scenarios where the impact of terrain obstacles is small. Therefore, the method of outputting both the first travel path and the risk field through the neural network in this embodiment of the application makes it easier for users to plan the first travel path with reference to the terrain obstacles in scenarios where the impact of terrain obstacles is large, so as to generate a travel path for driving and improve the obstacle avoidance success rate.

[0108] As one implementation method, after the neural network generates the risk field and the first travel path, both can be simultaneously displayed on the vehicle's visual interface. This allows the vehicle user to refer to the risk field to plan the first travel path, thereby generating a travel path for driving and improving obstacle avoidance success rate. This implementation method is suitable for scenarios where terrain obstacles have a significant impact on the travel path, allowing the vehicle user to dynamically plan the first travel path by referring to the risk field.

[0109] As one implementation method, after the neural network generates the risk field and the first travel path, only the risk field can be displayed on the vehicle's visual interface. This allows users to refer to the risk field to plan the actual travel path, improving the obstacle avoidance success rate. This implementation method is suitable for scenarios where terrain obstacles have a significant impact on the travel path, enabling vehicle users to plan their travel path based on the risk field.

[0110] After the neural network generates the risk field and the first travel path, the second network within the neural network can also generate a second travel path based on the terrain changes represented by the risk field, and display the second travel path on the site map. As one implementation, the second travel path can be displayed only on the vehicle's visual interface, allowing the vehicle user to drive according to the second travel path, thus improving obstacle avoidance success rate. This implementation is suitable for scenarios where terrain obstacles have a significant impact on the travel path, allowing the vehicle user to drive according to the second travel path generated from the risk field.

[0111] As one implementation method, after the neural network generates the risk field and the first travel path, or after generating the second travel path based on the risk field, only the first travel path can be displayed on the vehicle's visual interface, allowing the vehicle user to drive according to the first travel path. This method is suitable for scenarios where terrain obstacles have little impact on the travel path.

[0112] In one implementation, after the neural network generates a risk field and a first travel path, the second network within the neural network can also generate a second travel path based on the risk field. Both the first and second travel paths are simultaneously displayed on the vehicle's visual interface, allowing the vehicle user to plan a travel path based on both paths, further improving obstacle avoidance success rate. This implementation is suitable for scenarios where terrain obstacles significantly impact the travel path. The vehicle user can optimize the first travel path by referring to the second travel path, or vice versa, to generate a suitable travel path.

[0113] In one implementation, after the neural network generates a risk field and a first travel path, the second network within the neural network can also generate a second travel path based on the risk field. The risk field, the first travel path, and the second travel path are simultaneously displayed on the vehicle's visual interface, allowing the user to plan a travel path based on these elements, further improving obstacle avoidance success rate. This implementation is suitable for scenarios where terrain obstacles significantly impact the travel path. Users can comprehensively plan using the risk field, the first travel path, and the second travel path to generate a suitable travel path.

[0114] In the above embodiments, when displaying at least two of the risk field, the first passage path, and the second passage path to the user at the vehicle end, multiple results can be displayed in the same image or in different images.

[0115] It should be noted that the display of the risk field, the first passage path, and the second passage path also applies to the client side, making it convenient for users on the client side to view one or more of the risk field, the first passage path, and the second passage path, and thus making it convenient for clients on the client side to control the vehicle based on one or more of the displayed risk field, the first passage path, and the second passage path.

[0116] In the above embodiments, when generating a second travel path based on the risk field, a heuristic algorithm can be used to search for paths within the risk field to generate the second travel path. Specifically, in the BEV space, the risk value of each grid cell in the risk field is used as a local cost for path searching. Combined with a heuristic algorithm and a kinematic model, the path is progressively traversed in the forward direction within the risk field to search for the next grid cell with the lowest risk value. A minimum-risk grid path is generated based on these grid cells. The heuristic algorithm includes one or more of distance cost, angle cost, time cost, and collision cost, and the kinematic model uses a bicycle model. After finding a minimum-risk grid path, the coordinates of the grid cells on the minimum-risk grid path are smoothed using a polynomial curve. The smoothed grid cell coordinates are then used as path points, and a travel path is generated based on multiple path points.

[0117] In this embodiment, terrain obstacles and the road surface coexist in the risk field. The risk field allows for the perception of the location of terrain obstacles on the road surface, which helps improve the detection rate and accuracy of terrain obstacles, thereby increasing the obstacle avoidance success rate. Furthermore, the neural network outputs both the first passage path and the risk field, enriching the reference information used for obstacle avoidance and further contributing to the improvement of obstacle avoidance success rate.

[0118] The neural network consists of two branches: a second network for generating the risk field and a third network for generating the first pass path. The training method of the neural network will be described in detail below.

[0119] See Figure 7 This application provides a method for training a neural network, applied in the cloud, which includes the following steps:

[0120] S701. Acquire multiple frames of sample images of the site and / or sample point clouds of the site.

[0121] Images are captured by the vehicle's camera, and point clouds are collected by the LiDAR. Multi-frame sample images are images taken at multiple time points from the same viewpoint, or, alternatively, images taken from multiple viewpoints. Each frame in the multi-frame sample images is used to record at least a portion of the scene.

[0122] S702. Process multiple frames of sample images and / or sample point clouds using a neural network to obtain the risk field of the site and the first passage path in the site.

[0123] The neural network consists of a first network, a second network, and a third network. The output of the first network is the input to the second and third networks. The head of the second network is used to classify the data, and the head of the third network is used to segment the data. The output of the second network is the risk field, and the output of the third network is the first travel path. The risk field represents the terrain changes of the site in the form of a field.

[0124] The first network takes multiple frames of sample images and / or sample point clouds as input and outputs multi-scale features. The head of the first network is used to generate spatial features; for details, please refer to [link to documentation]. Figure 8 The processing of the first network includes the following sub-steps:

[0125] S801. Process multiple frames of sample images and / or sample point clouds to generate spatial features;

[0126] If processing multi-frame sample images, subsequent sample images are projected onto the coordinate system of the first sample image to generate a temporal fusion sample image. Spatial features are then generated based on this temporal fusion sample image. Subsequent sample images are any sample images other than the first sample image from a predetermined number of sample images. Specifically, when generating spatial features from the temporal fusion sample image, the coordinates in the BEV space are used as the query, the pixel coordinates of the temporal fusion sample image are used as the key, and the pixel values ​​of the temporal fusion sample image are used as the value. By constructing the relationship between the query, key, and value, the temporal fusion sample image is projected onto the BEV space to generate spatial features.

[0127] If processing sample point clouds, convert the sample point clouds to BEV space to generate spatial features. If processing multiple frames of sample images and sample point clouds, merge the spatial features generated from the sample images and the spatial features generated from the sample point clouds, and generate multi-scale features from the merged features.

[0128] It should be noted that if the input is a multi-frame sample image or a sample point cloud, the head of the first network used to process the multi-frame sample image and the head of the first network used to process the sample point cloud are different structures. If the input is a multi-frame sample image and a sample point cloud, the head of the first network used to process the multi-frame sample image and the sample point cloud includes the head structure of the first network used to process the multi-frame sample image, the head structure of the first network used to process the sample point cloud, and the structure for merging features.

[0129] S802. Process the spatial features to obtain multi-scale features.

[0130] In step S702, both the risk field and the first travel path are represented in the BEV space. As an example, the numerical range is set to 0-1, and this range is divided into 10 equal sub-ranges, representing the intervals containing 10 risk categories. Each risk category is represented by a numerical value between 0 and 10, which can be 0, 1…9. Different risk categories represent different levels of risk. The individual grids representing different risk categories constitute the risk field. In the BEV space, the value of a path point in the first travel path can be represented by 1, and the value of a non-path point can be represented by 0. Multiple path points constitute the first travel path.

[0131] The output of the neural network consists of two branches: one for generating the risk field and the other for generating the first passable path. Each branch contains a subnetwork: a second network for generating the risk field and a third network for generating the first passable path. The second and third networks can have the same or different structures.

[0132] In the risk field generation branch, the input to the second network is the multi-scale features output by the first network, and the head of the second network classifies these multi-scale features. Within the second network, methods such as pyramiding and multi-scale temporal convolutional transforms can be used to classify the multi-scale features, and multilayer perceptrons or convolutional neural networks can be used to generate the risk field.

[0133] In the branch that generates the first travel path, the input of the third network is the multi-scale features output by the first network. The head of the third network segments the multi-scale features to obtain the first-scale features. In the third network, multi-scale segmentation algorithms, convolutional neural networks, etc., can be used to segment the multi-scale features, and multilayer perceptrons or convolutional neural networks can be used to generate the first travel path. The first-scale feature is any one of the multi-scale features.

[0134] Each scale feature contains semantic and geometric information. When generating the first travel path, the third network determines the first travel path based on the semantic and geometric information in the first scale feature. Terrain obstacles are represented using a traditional symbolic method based on geometric information. The first travel path represents the lowest-risk path for vehicle passage using symbolic obstacles, and the travel path is represented in the BEV space.

[0135] S703. Supervised training of the neural network is performed using the sample risk field and sample travel path to obtain the trained neural network.

[0136] The sample risk field and sample travel path are labeled data of multi-frame sample images and / or sample point clouds of the site. The loss function used by the second network includes focal loss, mean squared error loss, or cross-entropy loss, etc. The loss function used by the third network includes dice loss, mean squared error loss, or cross-entropy loss, etc. The loss of the relevant risk field is determined by using the obtained site risk field and sample risk field, and the loss of the relevant travel path is determined by using the obtained first travel path and sample travel path.

[0137] After obtaining the risk field, the second network within the neural network can generate a second travel path based on the terrain changes represented by the risk field. Specifically, a path search is performed in the obtained risk field using a heuristic algorithm to obtain the second travel path, and the loss for the travel path is determined based on the second travel path and the sample travel paths. The heuristic algorithm includes parameters for representing one or more of the following costs: distance cost, angle cost, time cost, and collision cost.

[0138] In this embodiment, terrain obstacles and the road surface coexist in the risk field, enabling the perception of the terrain obstacle's position on the road surface. This helps improve the detection rate and accuracy of terrain obstacles, thereby increasing the obstacle avoidance success rate. Furthermore, the neural network trained in this embodiment outputs both the travel path and the risk field, allowing users to refer to the risk field to plan their travel path, thus improving the obstacle avoidance success rate.

[0139] The neural network comprises two branches: a first model for generating the risk field and a second model for generating the travel paths. The first model is trained based on the sample risk field, and the second model is trained based on the sample travel paths. The annotation methods for the sample risk field and sample travel paths are described below.

[0140] See Figure 9 This application provides a data generation method applied in the cloud, which includes the following steps:

[0141] S901, Obtain multiple first sample point clouds of the site;

[0142] A point cloud is a collection of all point cloud data within the observation range. In this embodiment, multiple first sample point clouds are used to generate a risk field and a passage path. The observation range refers to a preset length and width range in front of the vehicle. This preset length and width range characterizes the scope of effective data that the vehicle can collect. If the preset length and width range is set too small, it will limit the perception distance and hinder the detection of obstacles ahead. If the preset length and width range is set too large, it will affect the effectiveness of perception. It can be set according to the specific application scenario. For example, in a terrain risk field, it is generally set to about 100 meters. In this embodiment, the first sample point cloud for training is collected using the vehicle's lidar. The collection method is existing technology and will not be described in detail here.

[0143] S902. Process multiple first sample point clouds to obtain the first elevation map of each first sample point cloud;

[0144] In open-pit mine unmanned driving scenarios, the observed point cloud is sparse and cannot cover data from multiple points in a continuous space. For example, for an obstacle less than 50 centimeters away at 40 meters, the number of effective point clouds is usually less than 16. In this application, in order to obtain data from multiple points in space, the mean and variance of multiple points in space are calculated using a sparse Gaussian model. The elevation values ​​of the corresponding points are calculated based on the mean and variance of the points, and a first terrain elevation map is generated based on the elevation values ​​of multiple points. Specifically, the first sample point cloud is discretized, and corresponding grids are generated in the BEV (Bird's Eye View) space. The grid index is input into the sparse Gaussian model to obtain the mean and variance of multiple points in space. The elevation values ​​of multiple points in space are generated based on the mean and variance of multiple points, and a first elevation map is generated in the BEV space based on the elevation values ​​of multiple points.

[0145] The sparse Gaussian model uses radial basis functions to calculate the mean and variance of points in the space. See also... Figure 10 The training method for a sparse Gaussian model provided in this application includes:

[0146] S1001, Obtain multiple second sample point clouds of acceleration and site;

[0147] Acceleration and multiple second sample point clouds are used to train the sparse Gaussian model. The second sample point clouds are acquired by the LiDAR on the vehicle and the acceleration is calculated by the IMU of the vehicle's positioning module. The second sample point clouds can be the same as or different from the first sample point clouds.

[0148] S1002. Compute the gravity transformation matrix based on acceleration and multiple second sample points;

[0149] In this embodiment, the gravity transformation matrix is ​​calculated based on the angle between the point cloud normal vector and the vertical acceleration component at the vehicle end.

[0150] Specifically, the covariance of the second sample point cloud is first calculated, then the covariance is decomposed using Singular Value Decomposition (SVD) to determine the point cloud normal vector, and finally the vertical acceleration is determined based on the acceleration. The point cloud covariance is calculated using existing techniques.

[0151] Assume the second sample point cloud is {P} t The covariance of the second sample point cloud is ∑p. The normal vector n is obtained by decomposing the covariance ∑p using SVD. z where t represents time;

[0152] n z =SVD(∑p)

[0153] Assume the acceleration at time t is The vertical acceleration at time t is calculated using the exponential average.

[0154]

[0155] in, λ is the vertical acceleration at time t-1, Δt is the sampling time interval, and λ is a constant.

[0156] Based on the normal vector n z and vertical acceleration The angle between the two forces is used to generate the gravity transformation matrix δT;

[0157]

[0158] S1003. Transform the second sample point cloud to the gravity-aligned coordinate system using the gravity transformation matrix to generate elevation points;

[0159] Second sample point cloud {P t The elevation point {P} is generated by transforming the coordinates to a gravity-aligned coordinate system by left multiplication with the gravity transformation matrix. h}

[0160] {P h}=δT{P t}

[0161] By transforming the second sample point cloud to a gravity-aligned coordinate system to generate elevation points, the influence of vehicle pose and terrain slopes on radar observation can be eliminated.

[0162] S1004. Decompose the elevation points into plane point coordinates and corresponding observation values;

[0163] Elevation point {P h Each elevation point in the equation is decomposed into a two-dimensional plane point p. i =(x i ,y i ) and the corresponding observed value h i Observation h i This represents the height of the plane perpendicular to the point in the plane, where i is a natural number.

[0164] S1005. Train a sparse Gaussian model based on the coordinates of the points on the plane and the corresponding observations.

[0165] The sparse Gaussian model employs radial basis functions. Iterative training is performed using the coordinates of planar points and their corresponding observations. A trained sparse Gaussian model is obtained when a termination condition is met. The input to the trained sparse Gaussian model is a single first-sample point cloud, and the output is the mean and variance of the points in space.

[0166] In this application, the sparse Gaussian model follows the following distribution:

[0167] f(p)~SGP(m(p),k se (p,p′))

[0168]

[0169] P represents the input point, p' represents the coordinates of the predicted point, l is the weight parameter obtained after model training, and m(p) is the mean function.

[0170] It should be noted that in this embodiment, the first elevation map, the second elevation map, the risk field, and the travel path are all established based on the same BEV space.

[0171] S903. Filter out interfering areas in the first elevation map;

[0172] The interference area includes multiple interference points, which are points where the weighted sum of variance and gradient on the first elevation map is greater than a threshold.

[0173] Affected by sand, dust, etc., points in the interference area exhibit uneven gradients and large variances in the sparse Gaussian process space, showing stronger non-convexity compared to terrain obstacles such as falling rocks, spills, and ruts.

[0174] The region consists of multiple points. Each point in the first elevation map corresponds to a grid. Using the grid as the calculation unit, the gradient and variance of each grid are calculated separately. The gradient and variance of each grid are weighted to obtain a weighted value. Points corresponding to grids with weighted values ​​greater than a threshold are identified as interference points. The region containing multiple interference points is identified as the interference region. A weighted summation method is used to weight the gradient and variance. The weights of each parameter in the weighted summation are set empirically and are not specifically limited here. The threshold is set empirically and represents the critical value between interference points and non-interference points.

[0175] After identifying the interference area, the interference area is filtered on the first elevation map to remove the effects caused by sand, dust, etc.

[0176] In this step, the gradient is calculated using the squared exponential kernel function, and the variance is the variance of the points calculated in step S602 during the generation of the first elevation map.

[0177] To prevent misidentification of interference areas, interference areas are not identified within the target detection bounding box. Here, the target detection bounding box refers to the detection box defined by the target detection module of the intelligent driving system; for example, the target detection bounding box is defined for people or other vehicles.

[0178] Interference areas have been filtered out, which can increase the detection range of lidar for terrain obstacles in the presence of sand, dust and other interference, thereby improving the detection rate and accuracy of terrain obstacles.

[0179] It should be noted that if the impact of sand and dust is minimal, step S903 can be omitted; in other words, step S903 is an optional step.

[0180] S904. Merge multiple first elevation maps to generate a second elevation map;

[0181] LiDAR continuously acquires point clouds, with each first sample point cloud corresponding to a first elevation map. To improve the accuracy of the elevation map used to generate the risk field, multiple consecutive first elevation maps are fused to generate a second elevation map. As an example, the vehicle pose is transformed from a 6-DOF pose to a 3-DOF pose. Multiple elevation values ​​from the multiple first elevation maps are weighted according to variance normalization, and the weighted value is used as the elevation value of a point in the second elevation map, thus generating the second elevation map. As another example, after generating every 5 first elevation maps, these 5 maps are fused to generate one second elevation map.

[0182] S905. Generate a risk field based on the second elevation map;

[0183] The risk field characterizes the terrain changes of the site in the form of a field. In the second elevation map, a risk field is generated based on the elevation changes, gradient changes, and the degree of bumpiness experienced by vehicles traveling on multiple points in the second elevation map, or the elevation changes of the site and the degree of bumpiness experienced by vehicles traveling on the site, or the elevation changes of the site and gradient changes. Here, elevation change represents the degree of passability; for example, the value of elevation change ranges from 0 to 1, where 1 represents the maximum passability and 0 represents the minimum passability. Areas with elevation values ​​greater than 0.4 meters and less than 0.3 meters are set as impassable areas, and the elevation change value at points in these impassable areas is 0. The gradient change uses the gradient from step S903. The degree of bumpiness represents the degree of bumpiness experienced by vehicles traveling on the road surface, measured by the vertical displacement change of acceleration. As an example, the elevation changes, gradient changes, and degree of bumpiness of multiple points in the second elevation map are calculated, and the risk values ​​of multiple points are determined based on these values. The risk field is then generated based on the risk values ​​of these multiple points. For a given point, its risk value can be determined by weighted summation of the changes in height, gradient, and turbulence. The weights of each parameter in the weighted summation are set based on experience and are not specifically limited here.

[0184] As an example, the numerical range is set to 0-1, and then divided into 10 equal sub-ranges, representing the ranges containing 10 risk categories. Each risk category is represented by a numerical value between 0 and 10, which can be 0, 1…9. Different risk categories represent different levels of risk. The individual grids representing different risk categories constitute the risk field. Weighted values ​​of altitude variation, gradient variation, and turbulence level are mapped to the numerical range. The risk category is determined based on the sub-range of the numerical range, and the risk value of each grid is set to the value corresponding to that risk category. Based on the risk values ​​of each point, a risk field is generated in the BEV space.

[0185] S906. Conduct path search in the risk area to determine the access path to the site.

[0186] A path search is performed in the risk field based on a heuristic algorithm to determine a travel path. Specifically, in the BEV space, the risk value of each grid cell in the risk field is used as a local cost for path search. Combining the heuristic algorithm with a kinematic model, the path traverses the risk field progressively in the forward direction to search for the next grid cell with the lowest risk value. A lowest-risk grid path is generated based on these lowest-risk grid cells. The heuristic algorithm includes parameters representing one or more costs among distance cost, angle cost, time cost, and collision cost. The kinematic model uses a bicycle model. After finding a lowest-risk grid path, the coordinates of the grid cells on the lowest-risk grid path are smoothed using a polynomial curve. The smoothed grid cell coordinates are used as path points, and a travel path is generated based on multiple path points. As an example, in the BEV space, the value of a path point can be represented by 1, and the value of a non-path point can be represented by 0.

[0187] It should be noted that the risk fields and travel paths generated in this application embodiment can be applied individually to different scenarios, or simultaneously to the same scenario. In the case of individual use, if only the risk field is needed and the travel path is not required, step S906 can be omitted.

[0188] In this embodiment, the method of representing obstacles using a detection frame is abandoned. Instead, a risk field is used to represent obstacles. Since terrain obstacles and the road surface coexist in the risk field, the location of terrain obstacles on the road surface can be perceived through the risk field, which helps to improve the detection rate and accuracy of terrain obstacles. Furthermore, this embodiment filters out interference areas, which can increase the perception distance of terrain obstacles in the presence of interference such as sand and dust, further improving the detection rate and accuracy of terrain obstacles.

[0189] The risk field and travel path generated in this embodiment are used as labeled data for training. Figure 5 The neural network in the embodiment corresponding to the method.

[0190] See Figure 11 This application provides a path generation device 1100, comprising:

[0191] The acquisition module 1101 is used to acquire multiple frames of images of the site and / or point clouds of the site;

[0192] The vehicle's camera captures images, and the captured multi-frame images can be images taken at multiple times from the same viewpoint, or images taken from multiple viewpoints. Each frame in the multi-frame images is used to record at least a portion of the scene.

[0193] The processing module 1102 is used to process multiple frames of images and / or point clouds of the site through a neural network to obtain the risk field of the site and the first passage path in the site, so as to indicate the vehicle to drive in the site.

[0194] The neural network consists of a first network, a second network, and a third network. The output of the first network is the input to the second and third networks. The head of the second network is used to classify the data, and the head of the third network is used to segment the data. The output of the second network is the risk field, and the output of the third network is the first travel path. The risk field represents the terrain changes of the site in the form of a field.

[0195] The input to the first network is a multi-frame image, and the output is multi-scale features. The head of the first network is used to generate spatial features.

[0196] In this embodiment, both the risk field and the first travel path are represented in the BEV space. As an example, the numerical range is set to 0-1, and the numerical range is divided into 10 equal sub-intervals, representing the intervals where 10 risk categories are located. Each risk category is represented by a numerical value between 0 and 10, which can be 0, 1…9. Different risk categories represent different levels of risk. The individual grids with different risk categories constitute the risk field. In the BEV space, the value of a path point in the first travel path can be represented by 1, and the value of a non-path point can be represented by 0. Multiple path points constitute the first travel path.

[0197] It should be noted that this embodiment can also acquire point clouds and multi-frame sample images of the site. After generating spatial features using multi-frame images, the point cloud is then converted to BEV space to obtain the corresponding spatial features. The spatial features corresponding to the multi-frame images and the spatial features corresponding to the point cloud are merged. By processing the merged features, multi-scale features are obtained.

[0198] In this embodiment, only the point cloud of the site can be acquired, and the point cloud can be transformed into the BEV space to obtain spatial features. By processing the spatial features, multi-scale features can be obtained.

[0199] The path generation device 1100 may further include a display module. In one implementation, after the neural network generates a risk field and a first travel path, both can be simultaneously displayed on the vehicle's visual interface. This allows the vehicle user to refer to the risk field to plan the first travel path, thereby generating a travel path for driving and improving obstacle avoidance success rate. This implementation is suitable for scenarios where terrain obstacles significantly impact the travel path, allowing the vehicle user to dynamically plan the first path by referring to the risk field.

[0200] As one implementation method, after the neural network generates the risk field and the first travel path, only the risk field can be displayed on the vehicle's visual interface. This allows users to refer to the risk field to plan the actual travel path, improving the obstacle avoidance success rate. This implementation method is suitable for scenarios where terrain obstacles have a significant impact on the travel path, enabling vehicle users to plan their travel path based on the risk field.

[0201] The path generation device 1100 may further include a generation module, used to generate a second travel path based on the terrain changes represented by the risk field after the neural network generates a risk field and a first travel path. As one implementation, the display module may display only the second travel path on the vehicle's visual interface, allowing the vehicle user to drive according to the second travel path, thus improving obstacle avoidance success rate. This implementation is suitable for scenarios where terrain obstacles have a significant impact on the travel path, allowing the vehicle user to drive according to the second travel path generated by the risk field.

[0202] As one implementation method, after the neural network generates the risk field and the first travel path, or after generating the second travel path based on the risk field, only the first travel path can be displayed on the vehicle's visual interface, allowing the vehicle user to drive according to the first travel path. This method is suitable for scenarios where terrain obstacles have little impact on the travel path.

[0203] In one implementation, after the neural network generates a risk field and a first travel path, the second network within the neural network can also generate a second travel path based on the risk field. Both the first and second travel paths are simultaneously displayed on the vehicle's visual interface, allowing the vehicle user to plan a travel path based on both paths, further improving obstacle avoidance success rate. This implementation is suitable for scenarios where terrain obstacles significantly impact the travel path. The vehicle user can optimize the first travel path by referring to the second travel path, or vice versa, to generate a suitable travel path.

[0204] In one implementation, after the neural network generates a risk field and a first travel path, the second network within the neural network can also generate a second travel path based on the risk field. The risk field, the first travel path, and the second travel path are simultaneously displayed on the vehicle's visual interface, allowing the user to plan a travel path based on these elements, further improving obstacle avoidance success rate. This implementation is suitable for scenarios where terrain obstacles significantly impact the travel path. Users can comprehensively plan using the risk field, the first travel path, and the second travel path to generate a suitable travel path.

[0205] In the above embodiments, when displaying at least two of the risk field, the first passage path, and the second passage path to the user at the vehicle end, multiple results can be displayed in the same image or in different images.

[0206] The display of the risk field, the first passage path, and the second passage path is also applicable to the client side, making it convenient for users on the client side to view one or more of the risk field, the first passage path, and the second passage path, thereby making it convenient for clients on the client side to control the vehicle based on one or more of the displayed risk field, the first passage path, and the second passage path.

[0207] It should be noted that the specific implementation process of each module included in the embodiments of this application can be found in [the relevant documentation]. Figure 5 The relevant content of each step of the method.

[0208] In this embodiment, terrain obstacles and the road surface coexist in the risk field, enabling the perception of the terrain obstacle's position on the road surface. This helps improve the detection rate and accuracy of terrain obstacles, thereby increasing the obstacle avoidance success rate. Furthermore, the neural network outputs both the first passage path and the risk field, enriching the reference information used for obstacle avoidance and further contributing to the improved obstacle avoidance success rate.

[0209] This application provides a vehicle, including... Figure 11 The path generation device 1100 shown.

[0210] See Figure 12 This application provides a neural network training device 1200, comprising:

[0211] The acquisition module 1201 is used to acquire multiple frames of sample images of the site and / or sample point clouds of the site.

[0212] Images are captured by the vehicle's camera, and point clouds are collected by the LiDAR. Multi-frame sample images are images taken at multiple time points from the same viewpoint, or, alternatively, images taken from multiple viewpoints. Each frame in the multi-frame sample images is used to record at least a portion of the scene.

[0213] The processing module 1202 is used to process multiple frames of sample images and / or sample point clouds through a neural network to obtain the risk field of the site and the first passage path in the site.

[0214] The neural network consists of a first network, a second network, and a third network. The output of the first network is the input to the second and third networks. The head of the second network is used to classify the data, and the head of the third network is used to segment the data. The output of the second network is the risk field, and the output of the third network is the first travel path. The risk field represents the terrain changes of the site in the form of a field.

[0215] The input to the first network is multiple frames of sample images and / or sample point clouds, and the output is multi-scale features. The head of the first network is used to generate spatial features.

[0216] The supervision module 1203 is used to supervise the training of the neural network through the sample risk field and sample travel path to obtain the trained neural network.

[0217] The sample risk field and sample travel path are labeled data of multi-frame sample images and / or sample point clouds of the site. The loss function used by the second network includes focus loss, and the loss function used by the third network includes dice loss. The loss related to the risk field is determined by using the obtained site risk field and sample risk field, and the loss related to the travel path is determined by using the obtained first travel path and sample travel path.

[0218] As one implementation method, after obtaining the risk field, a path search is performed within the obtained risk field based on a heuristic algorithm to obtain a second travel path, and the loss for the relevant travel path is determined based on the second travel path and the sample travel paths. The heuristic algorithm includes parameters for characterizing one or more of the following costs: distance cost, angle cost, time cost, and collision cost.

[0219] It should be noted that the specific implementation process of each module included in the embodiments of this application can be found in [the relevant documentation]. Figure 7 The relevant content of each step of the method.

[0220] In this embodiment, terrain obstacles and the road surface coexist in the risk field, enabling the perception of the terrain obstacle's position on the road surface. This helps improve the detection rate and accuracy of terrain obstacles, thereby increasing the obstacle avoidance success rate. Furthermore, the neural network trained in this embodiment outputs both the passage path and the risk field, enriching the reference information used for obstacle avoidance and further contributing to improved obstacle avoidance success rate.

[0221] The neural network comprises two branches: a first model for generating the risk field and a second model for generating the travel paths. The first model is trained based on the sample risk field, and the second model is trained based on the sample travel paths. The annotation methods for the sample risk field and sample travel paths are described below.

[0222] See Figure 13This application provides a data generation apparatus 1300, comprising:

[0223] The acquisition module 1301 is used to acquire multiple first sample point clouds;

[0224] Multiple first sample point clouds are used to generate risk fields and travel paths, and these multiple first sample point clouds are sequential in time.

[0225] The first generation module 1302 is used to process multiple first sample point clouds to obtain a first elevation map of each first sample point cloud.

[0226] The first sample point cloud is discretized, and a corresponding raster is generated in the BEV (Bird's Eye View) space. The raster index is input into the elevation model to obtain the mean and variance of multiple points in the space. The elevation values ​​of multiple points in the space are generated based on the mean and variance of multiple points. The first elevation map is generated in the BEV space based on the elevation values ​​of multiple points.

[0227] Filtering module 1303 is used to filter out interference areas in the first elevation map;

[0228] The interference area includes multiple interference points, which are points where the weighted sum of variance and gradient on the first elevation map is greater than a threshold.

[0229] Affected by sand, dust, etc., points in the interference area exhibit uneven gradients and large variances in the sparse Gaussian process space, showing stronger non-convexity compared to terrain obstacles such as falling rocks, spills, and ruts.

[0230] The region consists of multiple points. Each point in the first elevation map corresponds to a grid. Using the grid as the calculation unit, the gradient and variance of each grid are calculated separately. The gradient and variance of each grid are weighted to obtain a weighted value. The points corresponding to the grids with weighted values ​​greater than a threshold are identified as interference points. The region where multiple interference points are located is identified as the interference region.

[0231] After identifying the interference area, the interference area is filtered on the first elevation map to remove the effects caused by sand, dust, etc.

[0232] It should be noted that if the impact of sand and dust is minimal, the filter module 1303 may not be necessary; in other words, the filter module 1303 is optional.

[0233] The second generation module 1304 is used to merge multiple first elevation maps to generate a second elevation map;

[0234] The third generation module 1305 is used to generate a risk field based on the second elevation map;

[0235] In the second elevation map, a risk field is generated based on the elevation changes, gradient changes, and the degree of bumpiness during vehicle movement at multiple points on the second elevation map, or the elevation changes and the degree of bumpiness during vehicle movement at the site, or the elevation changes and gradient changes at the site. As an example, the elevation changes, gradient changes, and bumpiness levels at multiple points on the second elevation map are calculated, and the risk values ​​for these points are determined based on these risk values. A risk field is then generated based on these risk values.

[0236] The determination module 1306 is used to perform path search in the risk field to determine the passage path of the site.

[0237] A path search is performed in the risk field based on a heuristic algorithm to determine a travel path. Specifically, in the BEV space, the risk value of each grid cell in the risk field is used as a local cost for path search. Combining the heuristic algorithm with a kinematic model, the path traverses the risk field progressively in the forward direction to search for the next grid cell with the lowest risk value. A minimum-risk grid path is generated based on these grid cells. The heuristic algorithm includes parameters representing one or more costs among distance cost, angle cost, time cost, and collision cost. The kinematic model uses a bicycle model. After finding a minimum-risk grid path, the coordinates of the grid cells on the minimum-risk grid path are smoothed using a polynomial curve. The smoothed grid cell coordinates are used as path points, and a travel path is generated based on multiple path points.

[0238] It should be noted that the specific implementation process of each module included in the embodiments of this application can be found in [the relevant documentation]. Figure 9 The relevant content of each step of the method.

[0239] In this embodiment, the method of representing obstacles using a detection frame is abandoned. Instead, a risk field is used to represent obstacles. Since obstacles and the road surface coexist in the risk field, the location of terrain obstacles on the road surface can be perceived through the risk field, which helps to improve the detection rate and accuracy of terrain obstacles. Furthermore, this embodiment filters out interference areas, which can increase the perception distance of terrain obstacles in the presence of interference such as sand and dust, further improving the detection rate and accuracy of terrain obstacles.

[0240] Please see Figure 14 This is a schematic diagram of an embodiment of the electronic device in this application.

[0241] The electronic device provided in this embodiment can be a processor, server, or dedicated data processing device, etc. The specific form of the device is not limited in this embodiment.

[0242] The electronic device 1400 may vary considerably due to different configurations or performance, and may include one or more processors 1401 and one or more memories 1402, in which programs or data are stored.

[0243] The memory 1402 can be volatile or non-volatile memory. Optionally, the processor 1401 is one or more central processing units (CPUs), graphics processing units (GPUs), or other dedicated processors, such as Ascend. The CPU can be a single-core CPU or a multi-core CPU. The processor 1401 can communicate with the memory 1402 and execute a series of instructions stored in the memory 1402 on the computing device 1400.

[0244] The electronic device 1400 also includes one or more wired or wireless network interfaces 1403, such as Ethernet interfaces.

[0245] Optionally, although Figure 14 As not shown in the diagram, the computing device 1400 may also include one or more power supplies; one or more input / output interfaces, which can be used to connect to a monitor, mouse, keyboard, touch screen device or sensing device, etc. The input / output interfaces are optional components and may or may not be present, and are not limited here.

[0246] In this embodiment, the processor 1401 in the electronic device 1400 can execute the steps of the above method embodiment.

[0247] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, they can be implemented in whole or in part in the form of a computer program product.

[0248] A computer program product includes one or more computer instructions. When the computer program product runs on a processor, the computer loads and executes the computer execution instructions, producing all or part of the processes or functions of the embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0249] Computer-readable storage media can be any usable medium that a computer can store, or a data storage device such as a server or data center that integrates one or more usable media. The usable medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)). The computer-readable storage medium stores a computer program that, when executed on a processor, produces all or part of the processes or functions described in the embodiments of this application.

[0250] The technical solutions provided in this application have been described in detail above. Specific examples have been used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A path generation method, characterized in that, include: Acquire multiple frames of images of the site and / or point clouds of the site; each frame of the multiple frames is used to record at least a portion of the site. The multi-frame images and / or point clouds of the site are processed by a neural network to obtain a risk field of the site and a first travel path in the site, so as to indicate the driving of vehicles in the site; wherein, the neural network includes a first network, a second network and a third network, the output of the first network is the input of the second network and the third network, the head of the second network is used to classify data, the head of the third network is used to segment data, the output of the second network is the risk field, and the output of the third network is the first travel path, the risk field representing the terrain changes of the site in the form of a field.

2. The method as described in claim 1, characterized in that, The terrain changes characterized by the risk locations are related to the changes in elevation, gradient, and the degree of bumpiness experienced by vehicles traveling through the site.

3. The method as described in claim 1 or 2, characterized in that, The multi-frame images are images taken at multiple time points from the same viewpoint, or the multi-frame images are images taken from multiple viewpoints.

4. The method according to any one of claims 1-3, characterized in that, The loss function used by the second network includes Focal Loss, and the loss function used by the third network includes Dice Loss.

5. The method according to any one of claims 1-4, characterized in that, Also includes: Display the risk field and / or the first passage path.

6. The method as described in claim 5, characterized in that, Also includes: Based on the terrain changes characterized by the risk location, a second access route is displayed on the map of the site.

7. The method as described in claim 6, characterized in that, Also includes: The second travel path is obtained by performing a path search in the risk field based on a heuristic algorithm; the heuristic algorithm includes parameters for characterizing one or more of the following costs: distance cost, angle cost, time cost, and collision cost.

8. A method for training a neural network, characterized in that, include: Acquire multiple frames of sample images of the site and / or sample point clouds of the site; Each frame in the multi-frame sample image is used to record at least a portion of the scene; The risk field of the site and a first passage path in the site are obtained by processing the multi-frame sample images and / or the sample point cloud through a neural network; wherein the neural network includes a first network, a second network and a third network, the output of the first network is the input of the second network and the third network, the head of the second network is used to classify data, the head of the third network is used to segment data, the output of the second network is the risk field, the output of the third network is the first passage path, and the risk field represents the terrain change of the site in the form of a field; The neural network is trained under supervision using a sample risk field and a sample travel path to obtain the trained neural network; the sample risk field and the sample travel path are the labeled data of the multi-frame sample images and / or the sample point cloud of the site.

9. The method as described in claim 8, characterized in that, The terrain changes characterized by the risk locations are related to the changes in elevation, gradient, and the degree of bumpiness experienced by vehicles traveling through the site.

10. The method as described in claim 8 or 9, characterized in that, The multi-frame sample images are images taken at multiple time points from the same viewpoint, or the multi-frame sample images are images taken from multiple viewpoints.

11. The method according to any one of claims 8-10, characterized in that, The loss function used by the second network includes FocalLoss, and the loss function used by the third network includes DiceLoss.

12. A data generation method, characterized in that, include: Obtain multiple first-sample point clouds of the site; The multiple first sample point clouds are processed to obtain a first elevation map for each first sample point cloud; Multiple first elevation maps are merged to generate a second elevation map; The risk field of the site is generated based on the second elevation map; The risk field represents the topographic changes of the site in the form of a field; A path search is performed in the risk field to obtain the access path to the site.

13. The method as described in claim 12, characterized in that, Before fusing multiple first elevation maps to generate a second elevation map, the process includes: Filter out interference regions in the first elevation map; the interference regions include interference points, which are points in the first elevation map whose weighted value of variance and gradient is greater than a threshold.

14. The method as described in claim 12 or 13, characterized in that, The generation of the site risk field based on the second elevation map specifically includes: Calculate the height change, gradient change, and turbulence level of multiple points in the second elevation map; calculate the risk value of the multiple points based on the height change, gradient change, and turbulence level; generate the risk field based on the risk values ​​of the multiple points.

15. The method according to any one of claims 12-14, characterized in that, The path search performed in the risk field to determine the access path to the site specifically includes: A path search is performed in the risk field based on a heuristic algorithm to determine the passage path of the field; the heuristic algorithm includes parameters for characterizing one or more of the following costs: distance cost, angle cost, time cost, and collision cost.

16. A path generation device, characterized in that, include: An acquisition module is used to acquire multiple frames of images of the site and / or point clouds of the site; each frame of the multiple frames is used to record at least a portion of the site. A processing module is used to process the multi-frame images and / or the point cloud of the site through a neural network to obtain a risk field of the site and a first travel path in the site, so as to indicate the vehicle's travel in the site; wherein, the neural network includes a first network, a second network and a third network, the output of the first network is the input of the second network and the third network, the head of the second network is used to classify data, the head of the third network is used to segment data, the output of the second network is the risk field, the output of the third network is the first travel path, and the risk field represents the terrain changes of the site in the form of a field.

17. The apparatus as claimed in claim 16, characterized in that, The terrain changes characterized by the risk locations are related to the changes in elevation, gradient, and the degree of bumpiness experienced by vehicles traveling through the site.

18. The apparatus as claimed in claim 16 or 17, characterized in that, The multi-frame images are images taken at multiple time points from the same viewpoint, or the multi-frame images are images taken from multiple viewpoints.

19. The apparatus according to any one of claims 16-18, characterized in that, The loss function used by the second network includes FocalLoss, and the loss function used by the third network includes DiceLoss.

20. The apparatus according to any one of claims 16-19, characterized in that, It also includes a display module: The display module is used to display the risk field and / or the first passage path.

21. The apparatus as claimed in claim 20, characterized in that, It also includes a generation module; The generation module is used to generate a second access path based on the terrain changes characterized by the risk location; The display module is also used to display a second access route on a map of the site.

22. The apparatus as claimed in claim 21, characterized in that, Based on the terrain changes characterized by the risk locations, a second travel path is generated, specifically including: The second travel path is obtained by performing a path search in the risk field based on a heuristic algorithm; the heuristic algorithm includes parameters for characterizing one or more of the following costs: distance cost, angle cost, time cost, and collision cost.

23. A training device for a neural network, characterized in that, include: The acquisition module is used to acquire multiple frames of sample images of the site and / or sample point clouds of the site; Each frame in the multi-frame sample image is used to record at least a portion of the scene; The processing module is used to process the multi-frame sample images and / or the sample point cloud through a neural network to obtain the risk field of the site and the first passage path in the site; wherein, the neural network includes a first network, a second network and a third network, the output of the first network is the input of the second network and the third network, the head of the second network is used to classify data, the head of the third network is used to segment data, the output of the second network is the risk field, the output of the third network is the first passage path, and the risk field represents the terrain change of the site in the form of a field; The supervision module is used to supervise the training of the neural network through the sample risk field and the sample passage path to obtain the trained neural network; the sample risk field and the sample passage path are the labeled data of the multi-frame sample images and / or the sample point cloud of the site.

24. The apparatus as claimed in claim 23, characterized in that, The terrain changes characterized by the risk locations are related to the changes in elevation, gradient, and the degree of bumpiness experienced by vehicles traveling through the site.

25. The apparatus as claimed in claim 23 or 24, characterized in that, The multi-frame sample images are images taken at multiple time points from the same viewpoint, or the multi-frame sample images are images taken from multiple viewpoints.

26. The apparatus according to any one of claims 23-25, characterized in that, The loss function used by the second network includes FocalLoss, and the loss function used by the third network includes DiceLoss.

27. A data generation apparatus, characterized in that, include: The acquisition module is used to acquire multiple first sample point clouds of the site; The first generation module is used to process the plurality of first sample point clouds to obtain a first elevation map of each first sample point cloud. The second generation module is used to fuse multiple first elevation maps to generate a second elevation map; The third generation module is used to generate the site's risk field based on the second elevation map; The risk field represents the topographic changes of the site in the form of a field; The determination module performs path search within the risk field to determine the access routes to the site.

28. The apparatus as claimed in claim 27, characterized in that, It also includes a filtering module; The filtering module is used to filter out interference regions in the first elevation map; the interference regions include interference points, which are points in the first elevation map whose weighted value of variance and gradient is greater than a threshold.

29. The apparatus as claimed in claim 27 or 28, characterized in that, The generation of the site risk field based on the second elevation map specifically includes: Calculate the height change, gradient change, and turbulence level of multiple points in the second elevation map; calculate the risk value of the multiple points based on the height change, gradient change, and turbulence level; generate the risk field based on the risk values ​​of the multiple points.

30. The apparatus according to any one of claims 27-29, characterized in that, The path search performed in the risk field to determine the access path to the site specifically includes: A path search is performed in the risk field based on a heuristic algorithm to determine a travel path; the heuristic algorithm includes parameters for characterizing one or more of the following costs: distance cost, angle cost, time cost, and collision cost.

31. A vehicle, characterized in that, Includes the path generation apparatus as described in any one of claims 16-22.

32. A computer device, characterized in that, include: At least one memory for storing programs; At least one processor for executing the program stored in the memory; When the program stored in the memory is executed, the processor is used to execute the method as described in any one of claims 1-15.

33. A computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method as described in any one of claims 1-15.

34. A computer program product, characterized in that, When the computer program product is run on a processor, the processor causes the processor to perform the method as described in any one of claims 1-15.

Images