Road prediction method and autonomous driving method, vehicle and device

Abstract

The present disclosure relates to a method for road prediction, comprising: receiving laser data indicating an environment of a vehicle, the laser data comprising a plurality of data points associated with one or more objects in the environment; generating a two-dimensional image based on the plurality of data points, the two-dimensional image indicating the environment in a direction substantially parallel to a road surface; extracting features from the two-dimensional image to obtain a feature map having a lower resolution than the two-dimensional image; and reconstructing the feature map to obtain a predicted image having a higher resolution than the feature map, and obtaining a result of road prediction based on the predicted image, the result of road prediction comprising both a contour of the road and a drivable area. The present disclosure also relates to a method, vehicle and device for autonomous driving.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving, and in particular to a method for road prediction and a method, vehicle and equipment for autonomous driving. Background Art

[0002] Autonomous driving means that the vehicle can complete lane keeping, overtaking and merging, stopping at red lights and going at green lights, interaction with lights and horns, obstacle avoidance, turning, etc. under the control of a computing device without the need for guidance and control by the driver. Autonomous driving as referred to in this disclosure includes non-fully autonomous driving of the vehicle (helping the driver to drive) and fully autonomous driving (completely replacing the driver to drive).

[0003] At present, path planning (e.g., trajectory planning) for autonomous driving mainly relies on maps and positioning, such as high-precision maps and high-precision positioning. However, since the map is not updated in real time (hence, it is also called "static map" in this article), if the map does not match the actual road conditions (also called "map failure" in this article) and / or the positioning is inaccurate, it will be difficult for the autonomous vehicle to plan the path. Therefore, real-time prediction of the road model is crucial to the safety performance of the autonomous vehicle.

[0004] In the event of a static map failure and / or inaccurate positioning, for straight roads, the autonomous vehicle can usually continue to drive forward along the current straight road. The "straight roads" referred to in this article include roads whose center lines are basically straight. Compared with straight roads, road contours at intersections, etc. are no longer simple linear geometric shapes, so it is not easy to determine the drivable area ahead. Based on this, it is very important to predict the drivable area at non-straight roads in real time, which is also related to the safety and flexibility of autonomous vehicles. The "non-straight roads" referred to in this article include any form of intersecting roads (including, for example, Figure 1A The intersection shown, Figure 1B The T-junction shown, Figure 1C The road segments with branch roads shown in the figure, etc.) and the turning sections of the road (including, for example, Figure 1D At the road corner shown, Figure 1E bends shown, etc.).

[0005] In the prior art, the polynomial fitting method uses a simple linear extrapolation method to predict the road geometry and drivable area in front of the autonomous vehicle in real time. This method is relatively accurate in predicting straight roads, but when the autonomous vehicle is on a non-straight road, the predicted drivable area is not accurate enough.

[0006] In addition, a particle filter method can be used to estimate the drivable area ahead. This method is based on point cloud data collected from a LiDAR sensor for estimation. This method assumes that there are obstacles where the point cloud is distributed. A straight line is emitted around the self-vehicle, and the line stops when it encounters a point cloud, otherwise it continues to search whether it will collide with the point cloud. However, this method is easily affected by dynamic vehicle occlusion and can only search the drivable area in a single direction, and cannot predict the geometry of the entire drivable area and intersection. Summary of the invention

[0007] One of the objectives of the present disclosure is to provide a method for road prediction and a method, vehicle, and device for autonomous driving.

[0008] According to a first aspect of the present disclosure, a method for road prediction is provided, comprising: receiving laser data indicating an environment of a vehicle, the laser data comprising a plurality of data points associated with one or more objects in the environment; generating a two-dimensional image based on the plurality of data points, the two-dimensional image indicating the environment in a direction substantially parallel to a road surface; performing feature extraction on the two-dimensional image to obtain a feature map having a lower resolution than the two-dimensional image; and reconstructing the feature map to obtain a predicted image having a higher resolution than the feature map, and obtaining a result of road prediction based on the predicted image, the result of road prediction comprising a contour and / or a drivable area of ​​the road.

[0009] According to a second aspect of the present disclosure, a method for road prediction is provided, comprising: receiving laser data indicating an environment of a vehicle, the laser data comprising a plurality of data points associated with one or more objects in the environment; generating a two-dimensional image based on the plurality of data points, the two-dimensional image indicating the environment in a direction substantially parallel to a road surface; and processing the two-dimensional image based on a pre-trained convolutional neural network model to obtain both a contour and a drivable area of ​​the road.

[0010] According to a third aspect of the present disclosure, a method for autonomous driving is provided, comprising: receiving an image indicating an environment in front of a vehicle; determining a road category in front of the vehicle based on the image; in response to determining that the road category is a non-straight road, performing the method described above to obtain a contour and / or a drivable area of ​​the road in front of the vehicle.

[0011] According to a fourth aspect of the present disclosure, there is provided an autonomously driven vehicle, comprising: one or more cameras configured to acquire images indicating an environment in front of the vehicle; and one or more computing devices configured to: receive the images from the cameras; determine a type of road in front of the vehicle based on the images; and in response to determining that the road type is a non-straight road, perform the method described above to obtain a contour and / or a drivable area of ​​the road in front of the vehicle.

[0012] According to a fifth aspect of the present disclosure, there is provided an autonomous driving device, comprising: one or more processors; and one or more memories, wherein the one or more memories are configured to store a series of computer-executable instructions, wherein when the series of computer-executable instructions are executed by the one or more processors, the one or more processors perform the method as described above.

[0013] According to the sixth aspect of the present disclosure, a non-temporary computer-readable storage medium is provided, characterized in that a series of computer-executable instructions are stored on the non-temporary computer-readable storage medium, and when the series of computer-executable instructions are executed by one or more computing devices, the one or more computing devices perform the method as described above.

[0014] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0016] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0017] Figures 1A to 1E is a schematic diagram schematically showing a lane to which the method according to an embodiment of the present disclosure is applicable.

[0018] Figure 2 FIG. 4 is a flowchart schematically illustrating a method for road prediction according to an embodiment of the present disclosure.

[0019] Figure 3 FIG. 4 is a flow chart schematically illustrating a method for autonomous driving according to an embodiment of the present disclosure.

[0020] Figure 4 It is a flowchart schematically showing a part of a method for road prediction according to an embodiment of the present disclosure.

[0021] Figure 5FIG. 4 is a block diagram schematically illustrating a method for road prediction according to an embodiment of the present disclosure.

[0022] Figure 6 is a block diagram schematically illustrating an autonomously driven vehicle according to an embodiment of the present disclosure.

[0023] Figures 7A to 7D It is a schematic diagram schematically illustrating the generation of a two-dimensional image based on laser data in a method for road prediction according to an embodiment of the present disclosure.

[0024] Figures 8A to 8G It is a schematic diagram schematically showing the processing results of each step in the autonomous driving method according to an embodiment of the present disclosure.

[0025] Note that in the embodiments described below, sometimes the same reference numerals are used in common between different drawings to represent the same parts or parts with the same functions, and their repeated descriptions are omitted. In some cases, similar numbers and letters are used to represent similar items, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings. DETAILED DESCRIPTION

[0026] The present disclosure will be described below with reference to the accompanying drawings, wherein the accompanying drawings illustrate several embodiments of the present disclosure. However, it should be understood that the present disclosure can be presented in a variety of different ways and is not limited to the embodiments described below; in fact, the embodiments described below are intended to make the disclosure of the present disclosure more complete and fully illustrate the scope of protection of the present disclosure to those skilled in the art. It should also be understood that the embodiments disclosed herein can be combined in various ways to provide more additional embodiments.

[0027] It should be understood that the terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. All terms (including technical terms and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. For the sake of simplicity and / or clarity, well-known functions or structures may not be described in detail.

[0028] Herein, the term "A or B" includes "A and B" and "A or B" rather than exclusively including only "A" or only "B", unless specifically stated otherwise.

[0029] In this document, the word "exemplary" means "used as an example, instance or illustration" rather than as a "model" to be exactly copied. Any implementation described as an example herein is not necessarily to be construed as preferred or advantageous over other implementations. Moreover, the present disclosure is not limited by any expressed or implied theory given in the above technical field, background technology, summary of the invention or detailed description.

[0030] In addition, the terms "first", "second" and the like may also be used herein for reference purposes only, and thus are not intended to be limiting. For example, the terms "first", "second" and other such numerical terms referring to structures or elements do not imply a sequence or order unless the context clearly indicates otherwise.

[0031] It should also be understood that when the term “include / comprises” is used in this document, it indicates the presence of the specified features, integers, steps, operations, units and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, units and / or components and / or their combinations.

[0032] As mentioned above, in the case of static map failure or inaccurate positioning, it will be difficult for autonomous vehicles to plan paths. Real-time prediction of intersection areas is particularly important for the safety of autonomous driving. Among the existing technical methods, the prediction range of the linear interpolation method is very limited, and this method is only applicable to straight lanes, that is, it cannot predict the drivable area of ​​curves or intersections in real time. The particle filter method in the prior art is easily blocked by dynamic vehicles and can only search the drivable area in a single direction, but cannot predict the geometry of the entire drivable area and intersection.

[0033] In response to the above technical problems, the inventors have proposed a road prediction method and a device for executing the method, which can simultaneously predict both the contour of the road in front of the vehicle and the drivable area based on laser data, and has good prediction accuracy for non-straight roads, especially intersections. The inventors have also proposed an autonomous driving method and a device and vehicle for executing the method, which can perform autonomous driving based on laser data and an image in front of the vehicle taken by a camera, wherein when the road in front is identified as a non-straight road based on the image, the aforementioned road prediction method is used to obtain the contour of the road in front of the vehicle and the drivable area.

[0034] Figure 21 is a flowchart schematically illustrating a method 100 for road prediction according to an embodiment of the present disclosure. The method 100 includes: receiving laser data indicating the environment of a vehicle (step 110), generating a two-dimensional image based on the laser data (step 120), extracting features from the two-dimensional image to generate a feature map (step 130), and reconstructing the feature map to generate a predicted image (step 140), and obtaining a result of road prediction based on the predicted image, wherein the result of road prediction includes both the contour of the road and the drivable area. An autonomous driving vehicle or autonomous driving device can perform driving planning based on the contour of the road and the drivable area. Among them, steps 130 and 140 can be completed by a pre-trained convolutional neural network (CNN) model, that is, the two-dimensional image obtained in step 120 is input into the trained CNN model to obtain both the contour of the output road and the drivable area.

[0035] The vehicle may include one or more LiDAR sensors to collect laser data, wherein the laser data includes multiple data points associated with one or more objects in the environment. One or more computing devices in the vehicle may receive multiple data points indicating the environment of the vehicle in a structured three-dimensional point cloud and / or an unstructured three-dimensional point cloud. In some cases, the computing device may receive data points in a point cloud like a grid. The laser data may include data points containing information indicating the position of an object in the environment relative to the vehicle. In other examples, the laser data may include other information about the environment of the vehicle. The received laser data may undergo some preprocessing, such as denoising and / or fusion, before subsequent processing. Denoising can remove sparse data points, data points that are relatively far from the edge of the road, etc. Fusion processing can fuse data points corresponding to different objects such as the edge of the road, dynamic obstacles, etc., and can also fuse information from other information sources (such as cameras, maps, positioning devices, etc.). Preprocessing may also include coordinate transformations described below.

[0036] Each data point in the laser data has a coordinate value (x, y, z). The coordinate values ​​of the data points mentioned in this disclosure are Figures 7A to 7DThe coordinate values ​​in the Oxyz coordinate system shown, the computing device may perform coordinate conversion on the original coordinate values ​​of the received data points to obtain the coordinate values ​​of each data point in the Oxyz coordinate system. The Oxyz coordinate system mentioned in the present disclosure is a spatial rectangular coordinate system, whose origin O is located on the road surface A where the vehicle is located in the height direction (z direction), and coincides with the observation point of the laser data (for example, which can be considered as the position of the LiDAR sensor) in the direction parallel to the road surface A. For example, the LiDAR sensor can be installed on the roof, 1.5 meters from the ground. If the original coordinate value of each data point in the laser data collected by the LiDAR sensor relative to the LiDAR sensor is (x', y', z'), then the coordinate values ​​converted to the Oxyz coordinate system are x=x', y=y', z=z'+1.5m respectively.

[0037] The environment of the vehicle refers to the three-dimensional space around the vehicle, including the spatial range that needs to be observed at present. For example, the spatial range that needs to be observed at present can be set to be from 90m in front of the vehicle to 10m behind the vehicle (length m is 100m), from 50m on the left of the vehicle to 50m on the right of the vehicle (width n is 100m), and from 10m above the road surface A to 1m below the road surface A (height h is 11m, wherein the height h2 above the road surface A is 10m, and the height h1 below the road surface A is 1m). That is, the three-dimensional space with coordinate values ​​x in the range of -50m to +50m, y in the range of -10m to +90m, and z in the range of -1m to +10m. The computing device can only receive data points in the space that needs to be observed, or it can receive more data points and then filter out the data points located in the space. In the above example, the three-dimensional space is defined as a cuboid. Those skilled in the art should understand that the shape of the three-dimensional space is not limited and can be defined as needed, and it can even be an irregular shape. For example, the height of the space that needs to be observed farther away from the vehicle may be greater than the height closer to the vehicle. Therefore, the three-dimensional space can be defined as one in which the height increases as the distance from the origin O increases.

[0038] Based on the data points in the environment of the vehicle, a two-dimensional image indicating the environment in a direction substantially parallel to the road surface A is generated. The three-dimensional space can be divided into M×N (e.g., 256×256) units U along the direction of the plane substantially parallel to the road surface A, and each unit U extends the height of the three-dimensional space in a direction perpendicular to the plane. Each pixel in the two-dimensional image can correspond one-to-one to each unit U in the three-dimensional space. Figure 7CIn the example shown, the horizontal cross section of each unit U is defined as a rectangle, and the horizontal cross section of each unit U is the same. Those skilled in the art should understand that the shape of the horizontal cross section of the unit is not limited and can be defined as needed, such as a rhombus, a hexagon or other polygon, a circle / ellipse, or even an irregular shape. In addition, the horizontal cross section of each unit can be different, for example, the cross-sectional area of ​​the unit closer to the object of interest can be smaller than the cross-sectional area of ​​the unit farther away from the object of interest, so that the generated two-dimensional image has a higher resolution for the object of interest.

[0039] In one embodiment, the two-dimensional image may be a binary image, for example, the number of data points or coordinate values ​​of the laser data distributed in each unit corresponding to the pixel of the unit U whose number or coordinate value is greater than the threshold is determined as the first value, and the number or coordinate value of the pixel corresponding to the unit U whose number or coordinate value is not greater than the threshold is determined as the second value, thereby generating a binary two-dimensional image with a resolution of M×N (e.g., 256×256) containing only two colors (without grayscale). In one embodiment, the two-dimensional image may be a grayscale image. Based on the number or coordinate value of the data points distributed in each unit by the laser data, the grayscale value of the pixel corresponding to each unit U is determined, thereby generating a two-dimensional image with grayscale of a resolution of M×N.

[0040] The number of data points in the laser data received from the LiDAR sensor is usually large, and there may be objects in the entire environment of the vehicle that do not need to be paid attention to. If all the data points in the entire environment are mapped to a two-dimensional image for subsequent image processing, it may result in a large amount of processing and / or the objects that need to be paid attention to are not prominent enough. In one embodiment, a first image is generated based on the first part of multiple data points, and a second image is generated based on the second part for subsequent processing. In the subsequent processing, the first and second images can be merged (for example, superimposed) to form a two-dimensional image and then processed, or the first and second images can be processed separately. The first part of the data points and the second part of the data points can be continuous, spaced, or at least partially overlapped in spatial position.

[0041] In one embodiment, the objects that need attention may include curbs and green plants growing on both sides of the road (for example, trees), so the first part of the data points may include data points whose height is close to the height of the road surface A, and the second part of the data points may include data points whose height is close to the height of the treetops of the trees beside the road. The approximate height of the treetops can be determined according to the geographical location of the road section where the vehicle is traveling, for example, the height of the treetops of trees in areas closer to the equator may be less than the height of the treetops of trees in areas farther away from the equator. In a specific example, the first part may include data points whose coordinate values ​​z in the three-dimensional space are within the range of -0.25m to +0.25m, and the second part may include data points whose coordinate values ​​z in the three-dimensional space are within the range of +4m to +5m. In one embodiment, the objects that need attention may include curbs and objects on the road, wherein the first part of the data points may be as described above, and the second part of the data points may include data points whose height is close to the height of at least a part of the objects located on the road. For example, in a specific example, the objects on the road mainly include traveling vehicles and walking pedestrians, and the second part may be set to include data points whose coordinate values ​​z in the three-dimensional space are within the range of +0.5m to +2m.

[0042] In some cases, other objects in the vehicle environment besides the object of interest may also need to be considered to avoid information loss. In one embodiment, a first image is generated based on a first portion of a plurality of data points, a second image is generated based on a second portion, and a third image is generated based on a third portion for subsequent processing. The first and second portions may correspond to the object of interest (as described above), and the third portion may correspond to other objects in the environment. For example, the third portion may include all data points in the three-dimensional space except the first and second portions, or may include all data points in the three-dimensional space.

[0043] The following is a specific example to illustrate the generation of a two-dimensional image based on laser data. The laser data in the three-dimensional space is divided into three parts. The first part includes all data points with coordinate values ​​z in the range of -0.25m to +0.25m, the second part includes all data points with coordinate values ​​z in the range of +4m to +5m, and the third part includes all data points with coordinate values ​​z in the range of -1m to 10m. Count the data points in the first part that fall into M×N (in 3D space) Fig.7D The number of cells in each unit U (taking the 5×4 shown as an example) is generated, and the following first original matrix is ​​generated:

[0044] Where N ijIndicates the number of data points in the corresponding unit; the subscripts i and j represent the counts of the rows and columns of the matrix, i = 1, 2, ..., 4 and j = 1, 2, ..., 5. The following formula 1 can be used to map each element in the first original matrix to a value space of 0 to 255 to obtain the grayscale value G of each pixel corresponding to each unit: ij :

[0045] G ij =N ij ×α+β Formula 1

[0046] In formula 1 and

[0047] Thus, the following first gray value matrix, i.e., the first image, is obtained:

[0048]

[0049] Formula 1 gives an example of obtaining a gray value matrix from an original matrix, that is, mapping the element N in the original matrix to a numerical range of 0 to 255 in a linear transformation manner, thereby generating an element G in the gray value matrix. Those skilled in the art should understand that other methods can also be used to map the element N to an element G in a numerical range of 0 to 255 (or a subset of the numerical range) to generate a gray value matrix, and the present disclosure is not limited thereto.

[0050] The coordinate z values ​​of the data points with the largest coordinate z value in each unit U of the three-dimensional space in the second part are counted to generate a second original matrix composed of the maximum z value. And the grayscale value of the pixel corresponding to each unit is calculated according to the method shown in Formula 1, so as to generate a second grayscale value matrix, that is, a second image. In one embodiment, the number K of data points in each unit U of the third part falling into the three-dimensional space can be counted to form a third original matrix. Since the third part includes all data points with coordinate values ​​z in the range of -1m to 10m, the K value may be very large in the unit with dense data points, and the K value may be very small in the unit with sparse data points, so that the grayscale value matrix obtained may have a loss of precision. Therefore, the original matrix can be generated directly without using the K value, but the K value can be transformed to compress its value range to generate the original matrix. In one embodiment, each element in the third original matrix can be a transformation value D, which is obtained based on the logarithm of the number K value of the data points in each unit U, for example, it can be calculated using the following formula 2:

[0051]

[0052] Wherein F is a scaling factor for the logarithmic value of the K value, which can be preset according to the numerical range of K and the required numerical range of D. Formula 2 gives an example of a compressed numerical range of the K value, and those skilled in the art should understand that other methods can also be used to generate the transformation value D to generate the required third original matrix, which is not limited by the present disclosure. Based on the third original matrix composed of the transformation value D, the grayscale value of the pixel corresponding to each unit can be calculated according to the method shown in Formula 1, for example, to generate a third grayscale value matrix, i.e., a third image.

[0053] In subsequent processing, feature extraction is performed on the two-dimensional image, for example, the image is processed with a filter to obtain a feature map with a lower resolution than the two-dimensional image. It should be noted that the two-dimensional image (which may include multiple images, such as the first to third images mentioned above), feature maps, and predicted images in the present disclosure are all stored in the form of data matrices, and each element in the matrix corresponds to each pixel of the corresponding image (or mapping map). Therefore, the "resolution" in the present disclosure can refer not only to the number of pixels in the corresponding image, but also to the number of elements in the matrix corresponding to the image. For some data matrices of intermediate processing results, although the matrix may not be considered to correspond to an image, the number of its elements can still be referred to as "resolution" in the present disclosure.

[0054] Feature extraction can be performed using known techniques, for example, one or more convolution layers in CNN can be used to extract features using convolution kernels (i.e., filters). It should be understood by those skilled in the art that when processed by multiple convolution layers, the two-dimensional image input to CNN can undergo multiple feature extractions. The resolution of the feature map is related to the size of the convolution kernel used, the step size of the convolution, and the number of convolution layers. In some cases, the feature extraction operation itself through the convolution layer will not reduce the resolution of the matrix (i.e., it will not reduce the number of elements of the input matrix), so it is necessary to set a downsampling layer after the convolution layer (for example, a downsampling layer can be set after each convolution layer, or a downsampling layer can be set after multiple convolution layers) to obtain a feature map with a lower resolution. In this case, the resolution of the feature map is also related to the number of layers and parameters of the sampling layer. It should be understood by those skilled in the art that CNN can also include other known functional layers, such as input layer, activation layer, pooling layer, fully connected layer, hidden layer, etc.

[0055] In some embodiments, the two-dimensional image may include multiple images, such as the first to third images mentioned above. In these embodiments, feature extraction may be performed on each image separately, that is, a different convolution kernel from other images may be used for the characteristics of each image in each convolution layer, which is more conducive to extracting features in the image. For example, for a first image corresponding to a curb generated based on the first part of the data points, a convolution kernel suitable for extracting features of the curb may be used; for a second image corresponding to a tree crown generated based on the second part of the data points, a convolution kernel suitable for extracting features of the tree crown may be used, and so on. In this example, feature extraction is performed on the first and second images one or more times using respective one or more convolution kernels to obtain feature maps of the first and second images, i.e., the first and second feature maps, respectively. It may be selected whether to merge the first and second feature maps (e.g., superimpose) as needed, and the first and second feature maps may be subjected to subsequent operations separately, or may be combined to obtain a combined feature map for the first and second images before subsequent operations are performed. For a case where a two-dimensional image includes more than two images (e.g., the case where the first to third images are generated based on the first to third parts of the data points respectively), operations similar to those for two images may be performed, which will not be described in detail.

[0056] Since the result of road prediction needs to be as accurate as possible, it is hoped that the predicted image has a higher resolution, for example, the same resolution as the two-dimensional image input to the CNN model. The feature map can be reconstructed to obtain a predicted image with a higher resolution than the feature map, for example, a feature map with a resolution of 32×32 is reconstructed into a predicted image with a resolution of 256×256, and the contour and drivable area of ​​the road are obtained based on the predicted image. Reconstruction can be performed using known techniques, for example, one or more upsampling layers in CNN can be used for reconstruction. Figure 43 is a flowchart schematically showing a part of a method 300 for road prediction according to an embodiment of the present disclosure. The step of reconstructing 330 includes: performing instance segmentation 331 on the feature map to segment the road in the predicted image, thereby obtaining the outline of the road; and performing binary segmentation 332 on the feature map to mark the drivable area in the predicted image as the first category and the non-drivable area as the second category, thereby obtaining the drivable area. The drivable area generally includes an area within the outline of the road and excluding various objects on the road (such as vehicles, pedestrians, roadblocks, etc.). When training the CNN model, the loss function of the instance segmentation 331 result and the binary segmentation 332 result can be comprehensively considered. For example, a first loss function is obtained based on the difference between the instance segmentation 331 result and the ground truth, and a second loss function is obtained based on the difference between the binary segmentation 332 result and the ground truth, and then the first and second loss functions are combined (for example, added) to obtain a comprehensive loss function, and the parameters of the CNN model are adjusted according to the comprehensive loss function.

[0057] In one embodiment, Figure 4 As shown, the method 300 further includes, before reconstruction 330, redistributing 320 the feature map to reduce the variance of the feature map, and then reconstructing 330 the redistributed feature map. Redistributing 320 to reduce the variance of the feature map can reduce the impact of noise changes on the stability of reconstruction 330. In one embodiment, redistribution 320 may include processing data in a feature map that does not obey a Gaussian distribution (e.g., a messy scattered distribution) to obey a Gaussian distribution. In one embodiment, redistribution 320 may include processing data in a feature map that obeys a Gaussian distribution to obey a Gaussian distribution whose distribution obeys a Gaussian distribution with a smaller variance (e.g., a more concentrated Gaussian distribution). The redistribution 320 process may be performed using one or more redistribution layers located after one or more convolutional layers in a CNN, i.e., the redistribution 320 process may be performed after feature extraction 310; or constraints may be added to the convolutional layer so that the result of its output is a result that obeys a Gaussian distribution, i.e., the redistribution 320 process may be performed during feature extraction 310. Thus, the upsampling layer for the reconstruction 330 process located after the redistribution 320 process is an upsampling layer suitable for reconstructing data that obeys a Gaussian distribution.

[0058] The characteristics of the redistribution 320 process (e.g., the parameters of the redistribution layer) also need to be adjusted during the training of the CNN model, and the parameters of the Gaussian distribution are constrained and optimized through the loss function. The loss function considered here, in addition to the error between the reconstructed predicted image and the true value (e.g., the above-mentioned comprehensive loss function), can also include the difference between the distribution of the latent variable and the unit Gaussian distribution, which can be measured by the KL divergence. The combination of the above-mentioned comprehensive loss function and the KL divergence can be considered to adjust the parameters of the CNN model.

[0059] Combine the following Figure 5 , a method 400 for road prediction according to an embodiment of the present disclosure is described with a specific example. First to third images 411, 412, 413 are generated based on the first to third parts of the plurality of data points of the laser data, respectively, and the results are superimposed after feature extraction is performed on the first to third images 411, 412, 413, respectively, to obtain a combined feature map 420 for the first to third images 411, 412, 413. The feature map 420 is redistributed to reduce the variance of the feature map, and a redistributed feature map 430 is obtained. The redistributed feature map 430 is reconstructed, and the feature map 430 is subjected to instance segmentation processing to obtain a road contour 442, and the feature map 430 is subjected to binary segmentation processing to obtain a drivable area 441.

[0060] Figure 3 2 is a flow chart schematically showing a method 200 for autonomous driving according to an embodiment of the present disclosure. The method 200 includes: receiving an image indicating an environment in front of a vehicle (step 210), determining a road type in front of the vehicle based on the image (step 220), and determining the road type as follows: Figures 1A to 1E In response to determining that the road category is a non-straight road, the method described above (e.g., method 100) is performed to obtain both the contour and the drivable area of ​​the road ahead of the vehicle. In one embodiment, the non-straight road is as follows: Figures 1A to 1C The intersection shown. The autonomous vehicle or device can plan driving based on the contour and drivable area of ​​the road. For example, the autonomous vehicle or device can compare the contour and drivable area of ​​the road with the corresponding road segment in the static map to determine whether the static map is credible, whether there are obstacles in the road segment, etc.

[0061] An autonomous vehicle or device may generally include one or more cameras, which may be configured to acquire images (which may be static or dynamic images) indicating the environment in front of the vehicle. An autonomous vehicle or device may also include one or more computing devices, which may receive such images from a camera and determine the category of the road in front of the vehicle based on the images. Determining the road category may be accomplished by a pre-trained classification recognition model. The classification recognition model may be, for example, a binary classifier or a deep convolution classifier based on a neural network, using the images captured by the camera as input to the classification recognition model. The classification recognition model may identify whether the road in front of the vehicle is a non-straight road (or, in one embodiment, whether it is an intersection). At the same time, the classification recognition model may also output the confidence of its recognition result to measure the reliability of its classification recognition result. When the confidence is higher than a set threshold, the computing device may adopt the recognition result of the classification recognition model.

[0062] Figures 8A to 8G is a schematic diagram schematically showing the processing results of each step in the autonomous driving method according to an embodiment of the present disclosure. According to the autonomous driving method of the embodiment of the present disclosure, the image indicating the environment in front of the vehicle from the camera that can be received by the computing device can be as follows Fig. 8A As shown, the computing device may input the image into the classification recognition model. If the road category output by the classification recognition model is a non-straight road (an intersection in one embodiment), and the confidence of the recognition result is greater than a set threshold, the computing device may receive laser data indicating the environment of the vehicle from the LiDAR sensor, and generate a two-dimensional image indicating the vehicle environment in a direction substantially parallel to the road surface according to the laser data, as shown in FIG. Figure 8B shown. Figure 8B The two-dimensional image shown is a superposition of a first image corresponding to data points near the curb, a second image corresponding to data points near the treetops on both sides of the road, and a third image corresponding to all data points in the vehicle environment. For ease of description, the accompanying drawings only show the process of merging multiple images contained in the two-dimensional image into one image for feature extraction, and do not show the process of extracting features separately. However, those skilled in the art can understand the process of extracting features separately from multiple images based on the accompanying drawings and description.

[0063] Figure 8B After the two-dimensional image shown is input into the pre-trained CNN model, it can pass through multiple convolutional layers for multiple feature extractions, and the feature extraction results of each convolutional layer serve as the input of the next adjacent convolutional layer. Figures 8C-1 to 8C-3 Three feature extractions are shown in sequence (note that the three feature extractions shown may be three non-adjacent feature extractions). It can be seen that after the first feature extraction, Figure 8C-1 The feature maps shown have a lower resolution than the input Figure 8B The resolution of the two-dimensional image shown in the figure is obtained after the second feature extraction. Figure 8C-2 The feature maps shown have a resolution lower than Figure 8C-1 The resolution of the feature map shown in the figure is obtained after the third feature extraction. Figure 8C-3 The feature maps shown have a resolution lower than Figure 8C-2 The resolution of the feature map shown. Those skilled in the art should understand that in a practical CNN model, more than three convolutional layers may be included to perform more than three feature extractions.

[0064] The feature map output by the last convolutional layer in the CNN model can be redistributed and the feature map after redistribution (not shown) can be reconstructed to generate a predicted image. Figures 8D-1 to 8D-3 The results of three upsamplings after binary segmentation are shown in sequence (note that the three upsamplings shown may be three non-adjacent upsamplings). It can be seen that Figures 8D-1 to 8D-3 The resolution increases successively. Fig. 8E The prediction result of the binary segmentation process, i.e., the drivable area, is shown, wherein the light-colored part represents the part marked as the non-drivable area, and the dark-colored part represents the part marked as the drivable area. Figures 8F-1 to 8F-3 The results of three upsamplings for instance segmentation processing are shown in sequence (note that the three upsamplings shown may be three non-adjacent upsamplings). It can be seen that Figures 8F-1 to 8F-3 The resolution increases successively. Figure 8G The predicted result of the example segmentation process is shown, that is, the outline of the road, where the light-colored part is the segmented road part.

[0065] The present disclosure also provides an autonomous driving device (or vehicle). The autonomous driving device includes one or more processors and one or more memories. One or more processors are configured to execute the method according to the embodiment of the present disclosure described above. The memory is configured to store data and programs required by the processor. The program includes a series of computer executable instructions required for the processor to execute the method according to the embodiment of the present disclosure described above. The data includes the input, output and intermediate results of each step such as the image received from the camera, the laser data received from the LiDAR sensor, the pre-processed data points, the generated two-dimensional image, the feature map, and the prediction results described above, and may also include the calculation basis such as the classification recognition model and the CNN model described above. One or more memories can be configured to use one memory to store one of the above contents, and can also be configured to use multiple memories to jointly store one of the above contents, or use one memory to store more than one of the above contents.

[0066] It should be noted that one or more memories may be local memories (e.g., memories loaded on the device (or vehicle)), or cloud memories (e.g., memories in a cloud server), or part of them may be local memories and part of them may be cloud memories. Similarly, one or more processors may be local processors (e.g., processors loaded on the device (or vehicle)), or cloud processors (e.g., processors in a cloud server), or part of them may be local processors and part of them may be cloud processors.

[0067] Figure 6 Schematically illustrates a block diagram of an autonomously driven vehicle 500 according to an embodiment of the present disclosure. Figure 6 Describe vehicle 500, which can also be an example of a hardware device applied to various aspects of the present disclosure. Vehicle 500 can be any machine configured to perform processing and / or computing, and can be but not limited to a workstation, server, desktop computer, laptop computer, tablet computer, personal data assistant, smart phone, car computer, or any combination thereof. The above-mentioned autonomous driving device according to the embodiment of the present disclosure can be implemented in whole or in part by vehicle 500 or similar devices or systems.

[0068] The vehicle 500 may include elements that may be connected to or communicate with the bus 502 via one or more interfaces. For example, the vehicle 500 may include a bus 502, as well as one or more processors 504, one or more input devices 506, and one or more output devices 508. The one or more processors 504 may be any type of processor, and may include, but are not limited to, one or more general-purpose processors and / or one or more special-purpose processors (e.g., special processing chips). The input device 506 may be any type of device that can input information to a computing device, and may include, but are not limited to, a camera, a LiDAR sensor, a mouse, a keyboard, a touch screen, a microphone, and / or a remote control. The output device 508 may be any type of device that can present information, and may include, but are not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer.

[0069] The vehicle 500 may also include or be connected to a non-transitory storage device 510. The non-transitory storage device 510 may be any storage device that is non-transitory and can implement data storage, and may include but is not limited to a disk drive, an optical storage device, a solid-state memory, a floppy disk, a hard disk, a tape or any other magnetic medium, an optical disk or any other optical medium, a ROM (read-only memory), a RAM (random access memory), a cache memory, and / or any other memory chip / chipset, and / or any other medium from which a computer can read data, instructions and / or code. The non-transitory storage device 510 may be detachable from the interface. The non-transitory storage device 510 may have data / instructions / code for implementing the above methods, steps and processes. One or more of the one or more memories described above may be implemented by the non-transitory storage device 510.

[0070] The vehicle 500 may also include a communication device 512. The communication device 512 may be any type of device or system capable of communicating with an external device and / or with a network, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication device, and / or a chipset, such as a Bluetooth device, a 1302.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0071] The vehicle 500 may also be connected to external devices, such as a GPS receiver, sensors for sensing different environmental data, such as an acceleration sensor, a wheel speed sensor, a gyroscope, and the like. In this way, the vehicle 500 may, for example, receive location data and sensor data indicating the driving condition of the vehicle. When the vehicle 500 is used as an on-board device, it may also be connected to other facilities of the vehicle (such as an engine system, wipers, an anti-lock braking system, etc.) to control the operation and operation of the vehicle.

[0072] In addition, the non-transitory storage device 510 may have map information and software elements so that the processor 504 can perform route guidance processing. In addition, the output device 508 may include a display for displaying a map, a location marker of the vehicle, and an image indicating the driving condition of the vehicle. The output device 508 may also include a speaker or an interface with headphones for audio guidance.

[0073] The bus 502 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus. In particular, for vehicle-mounted devices, the bus 502 may also include a Controller Area Network (CAN) bus or other architectures designed for use in vehicles.

[0074] The vehicle 500 may also include working memory 514 , which may be any type of working memory that may store instructions and / or data useful for the operation of the processor 504 , and may include, but is not limited to, random access memory and / or read-only storage devices.

[0075] Software elements may be located in the working memory 514, including but not limited to an operating system 516, one or more application programs 518, drivers, and / or other data and code. Instructions for performing the above methods and steps may be included in one or more application programs 518. The executable code or source code of the instructions of the software elements may be stored in a non-transitory computer-readable storage medium, such as the storage device 510 described above, and may be read into the working memory 514 by compilation and / or installation. The executable code or source code of the instructions of the software elements may also be downloaded from a remote location.

[0076] It should also be understood that changes can be made according to specific requirements. For example, custom hardware can also be used, and / or specific elements can be implemented with hardware, software, firmware, middleware, microcode, hardware description language or any combination thereof. In addition, connections with other computing devices such as network input / output devices can be adopted. For example, some or all of the methods or devices according to the embodiments of the present disclosure can be implemented by using programming hardware (for example, including a field programmable gate array (FPGA) and / or a programmable logic array (PLA)) in assembly language or hardware programming language (such as VERILOG, VHDL, C++) according to the logic and algorithm of the present disclosure.

[0077] It should also be understood that the components of vehicle 500 can be distributed across a network. For example, some processing can be performed using one processor, while other processing can be performed by another processor remote from the one processor. Other components of vehicle 500 can also be similarly distributed. In this way, vehicle 500 can be interpreted as a distributed computing system that performs processing at multiple locations.

[0078] Although various aspects of the present disclosure have been described so far with reference to the accompanying drawings, the above-described methods, systems, and devices are merely illustrative examples, and the scope of the present invention is not limited by these aspects, but only by the following aspects: the attached claims and their equivalents. Various elements may be omitted or replaced with equivalent elements. In addition, the steps may be performed in an order different from the order described in the present disclosure. In addition, various elements may be combined in various ways. It is also important to note that as technology develops, many of the elements described may be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for road prediction, comprising: receiving laser data indicative of an environment of a vehicle, the laser data comprising a plurality of data points associated with one or more objects in the environment; generating a two-dimensional image based on the plurality of data points, the two-dimensional image indicating the environment in a direction substantially parallel to a road surface; Performing feature extraction on the two-dimensional image to obtain a feature map having a lower resolution than the two-dimensional image; as well as The feature map is reconstructed to obtain a predicted image with a higher resolution than the feature map, and a road prediction result is obtained based on the predicted image, wherein the road prediction result includes both a road profile and a drivable area. wherein generating the two-dimensional image based on the plurality of data points comprises: generating first and second images based on first and second parts of the plurality of data points, respectively; The step of extracting features from the two-dimensional image to obtain a feature map comprises: extracting features from the first and second images respectively to obtain feature maps of the first and second images respectively; or extracting features from the first and second images respectively and then superimposing the extracted features to obtain a combined feature map of the first and second images. Among them, generating the two-dimensional image based on the multiple data points includes: dividing the environment into M×N units along the direction of a plane basically parallel to the road surface, each unit extending the height of the environment along the direction perpendicular to the plane; and determining the grayscale value of the pixel corresponding to each unit based on the number or coordinate values ​​of the multiple data points distributed in each unit, thereby generating the two-dimensional image with grayscale with a resolution of M×N.

2. The method according to claim 1, characterized in that Generating the two-dimensional image based on the plurality of data points comprises: generating first to third images based on first to third parts of the plurality of data points, respectively; Extracting features from the two-dimensional image to obtain a feature map includes: extracting features from the first to third images respectively to obtain feature maps of each of the first to third images respectively; or extracting features from the first to third images respectively and then superimposing them to obtain a combined feature map for the first to third images.

3. The method according to claim 1, characterized in that: The first portion includes data points whose heights approximate the height of a road surface; and The second portion includes data points whose heights are close to the height of the crowns of trees beside the road and / or data points whose heights are close to the height of at least a portion of an object located on the road.

4. The method according to claim 2, characterized in that: The first portion includes data points whose heights approximate the height of the road surface; The second portion includes data points whose heights are close to the heights of tree crowns of trees beside the road and / or data points whose heights are close to the heights of at least a portion of an object located on the road; as well as The third portion includes all data points in the plurality of data points except the first and second portions, or all data points in the plurality of data points.

5. The method according to claim 1, characterized in that Also includes: Before the reconstruction, the feature map is redistributed to reduce the variance of the feature map, and the redistributed feature map is reconstructed.

6. The method according to claim 1, characterized in that Reconstructing the feature map includes: Performing instance segmentation processing on the feature map to obtain a road contour; and The feature map is subjected to binary segmentation processing to obtain a drivable area.

7. A method of autonomous driving, comprising: receiving an image indicative of an environment in front of the vehicle; Determining the type of road ahead of the vehicle based on the image; as well as In response to determining that the road category is a non-straight road, the method according to any one of claims 1 to 6 is performed to obtain both the contour and the drivable area of ​​the road ahead of the vehicle.

8. The method according to claim 7, characterized in that The non-straight road is an intersection.

9. An autonomous vehicle comprising: one or more cameras configured to acquire images indicative of an environment in front of the vehicle; as well as One or more computing devices configured to: receiving the image from the one or more cameras; Determining the type of road ahead of the vehicle based on the image; as well as In response to determining that the road category is a non-straight road, the method according to any one of claims 1 to 6 is performed to obtain both the contour and the drivable area of ​​the road ahead of the vehicle.

10. The vehicle according to claim 9, characterized in that The non-straight road is an intersection.

11. An autonomous driving device, comprising: one or more processors; as well as one or more memories configured to store a series of computer-executable instructions, Wherein, when the series of computer-executable instructions are executed by the one or more processors, the one or more processors are caused to perform the method as described in any one of claims 1-8.

12. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores a series of computer-executable instructions, and when the series of computer-executable instructions are executed by one or more computing devices, the one or more computing devices are caused to perform the method as described in any one of claims 1-8.

Images