Road curvature determination method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides a road curvature determination method and device, electronic equipment and a readable storage medium, and relates to the field of data processing, and the method comprises the steps: obtaining environment data around a vehicle; wherein the environment data comprises environment point cloud data and / or environment image data; performing target object detection based on the environment data to obtain a target object detection result; establishing a road geometric model by adopting a target object detection result; a road curvature is determined based on the road geometry model. According to the method, the road geometric model can be established according to the static objects above the ground, the road curvature can be predicted according to the road geometric model, and the static objects above the ground are generally not easy to shield, so that the detection accuracy of the road curvature can be improved under the condition that accurate road surface information is difficult to obtain.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and particularly to a method, device, electronic device and readable storage medium for determining road curvature. Background Art

[0002] With the development of technology, more and more vehicles are equipped with autonomous driving functions or assisted driving functions. When realizing autonomous driving or assisted driving functions, the vehicle needs to detect the road curvature in real time to ensure driving safety.

[0003] In the related art, in one way, a lidar is used to scan the road surface near the vehicle, collect the road surface point cloud data, and determine the corresponding road curvature according to the road surface point cloud data. In another way, a vision camera is used to take pictures of the road surface near the vehicle, identify the lane line information based on the captured road surface image, and then determine the road curvature according to the lane line information.

[0004] However, in some cases, the road surface and lane lines are easily blocked (for example, when the vehicle is in a curve), resulting in missing road surface data, thus making it impossible to accurately detect the road curvature. Summary of the Invention

[0005] In view of this, the present invention aims to provide a method, device, electronic device and readable storage medium for determining road curvature to solve the problem that the road curvature cannot be accurately detected in the prior art.

[0006] To achieve the above object, the technical solution of the present invention is realized as follows:

[0007] In a first aspect, the present invention provides a method for determining road curvature, the method comprising:

[0008] Obtain the environmental data around the vehicle; wherein, the environmental data includes environmental point cloud data and / or environmental image data;

[0009] Perform target object detection based on the environmental data to obtain a target object detection result; wherein, the target object includes stationary objects above the road surface;

[0010] Establish a road geometry model by using the target object detection result;

[0011] Determine the road curvature based on the road geometry model.

[0012] Optionally, the performing target object detection based on the environmental data to obtain a target object detection result includes:

[0013] Generate a point cloud bird's-eye view based on the environmental point cloud data, and generate an image bird's-eye view based on the environmental image data;

[0014] Feature encode the bird's-eye view of the point cloud and the bird's-eye view of the image through an encoder to respectively obtain a first feature corresponding to the bird's-eye view of the point cloud and a second feature corresponding to the bird's-eye view of the image;

[0015] Fuse the first feature and the second feature to obtain a third feature;

[0016] Decode the third feature through a decoder to obtain the target object detection result.

[0017] Optionally, the target object detection result includes the category information, size information, and position information of the target object. Using the target object detection result to establish a road geometric model includes:

[0018] Fit the size information and position information corresponding to the target objects with the same category information to obtain the number of types of the category information of the first geometric models;

[0019] Determine the road geometric model based on the first geometric model.

[0020] Optionally, the determining the road curvature based on the road geometric model includes:

[0021] Input the road geometric model into a road curvature prediction model to obtain the road curvature output by the road geometric model.

[0022] Optionally, the method further includes:

[0023] Obtain the original point cloud data around the vehicle;

[0024] Identify the ground point cloud points in the original point cloud data;

[0025] Filter out the ground point cloud points from the original point cloud data to obtain the environmental point cloud data.

[0026] Optionally, the method further includes:

[0027] Obtain the original image data around the vehicle;

[0028] Identify the ground area in the original image data;

[0029] Remove the ground area from the original image data to obtain the environmental image data.

[0030] Optionally, the method further includes:

[0031] Obtain the driving road information;

[0032] Determine the road visibility distance based on the driving road information;

[0033] When the visible distance of the road is less than or equal to the first distance, perform the step of acquiring the environmental data around the vehicle.

[0034] In a second aspect, the present invention provides a road curvature determination device, and the device includes:

[0035] An acquisition module, configured to acquire environmental data around the vehicle; wherein, the environmental data includes environmental point cloud data and / or environmental image data;

[0036] A detection module, configured to perform target object detection based on the environmental data to obtain a target object detection result; wherein, the target object includes stationary objects in the air above the road surface;

[0037] A geometric model module, configured to establish a road geometric model by using the target object detection result;

[0038] A road curvature module, configured to determine the road curvature based on the road geometric model.

[0039] Optionally, the detection module includes:

[0040] An aerial view sub-module, configured to generate a point cloud aerial view based on the environmental point cloud data and generate an image aerial view based on the environmental image data;

[0041] A feature sub-module, configured to perform feature encoding on the point cloud aerial view and the image aerial view through an encoder to respectively obtain a first feature corresponding to the point cloud aerial view and a second feature corresponding to the image aerial view;

[0042] A feature fusion sub-module, configured to fuse the first feature and the second feature to obtain a third feature;

[0043] A detection result sub-module, configured to decode the third feature through a decoder to obtain the target object detection result.

[0044] Optionally, the target object detection result includes category information, size information, and position information of the target object, and the geometric model module includes:

[0045] A first geometric model sub-module, configured to fit the size information and position information corresponding to the target objects with the same category information to obtain the number of types of the category information of first geometric models;

[0046] A road geometric model sub-module, configured to determine the road geometric model based on the first geometric model.

[0047] Optionally, the road curvature module is further configured to input the road geometry model into a road curvature prediction model to obtain the road curvature output by the road geometry model.

[0048] Optionally, the apparatus further includes:

[0049] An original point cloud data module, configured to obtain original point cloud data around the vehicle;

[0050] A ground point cloud point module, configured to identify ground point cloud points in the original point cloud data;

[0051] An environmental point cloud data module, configured to filter out the ground point cloud points from the original point cloud data to obtain the environmental point cloud data.

[0052] Optionally, the apparatus further includes:

[0053] An original image data module, configured to obtain original image data around the vehicle;

[0054] A ground area module, configured to identify a ground area in the original image data;

[0055] An environmental image data module, configured to remove the ground area from the original image data to obtain the environmental image data.

[0056] Optionally, the apparatus further includes:

[0057] A driving road information module, configured to obtain driving road information;

[0058] A road visible distance module, configured to determine a road visible distance based on the driving road information;

[0059] The obtaining module is further configured to execute the step of obtaining environmental data around the vehicle when the road visible distance is less than or equal to a first distance.

[0060] In a third aspect, the present invention provides a readable storage medium. When instructions in the readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned road curvature determination method.

[0061] In a fourth aspect, an embodiment of the present invention provides an electronic device, which includes a processor and a memory. The memory stores a program or instructions that can run on the processor. When the program or instructions are executed by the processor, the steps of the method in the first aspect are implemented.

[0062] Fifth aspect, the present invention provides a vehicle controller, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above-mentioned road curvature determination method is implemented.

[0063] Sixth aspect, the present invention provides a vehicle, which includes the above-mentioned vehicle controller.

[0064] Compared with the prior art, the road curvature determination method, device, electronic device, and readable storage medium of the present invention have the following advantages:

[0065] In summary, the embodiment of the present invention provides a road curvature determination method, which includes: obtaining environmental data around the vehicle; wherein, the environmental data includes environmental point cloud data and / or environmental image data; performing target object detection based on the environmental data to obtain a target object detection result; establishing a road geometric model using the target object detection result; and determining the road curvature based on the road geometric model. It is possible to establish a road geometric model according to stationary objects above the ground and predict the road curvature according to the road geometric model. Since stationary objects above the ground are usually not easily blocked, the detection accuracy of the road curvature can be improved in the case where it is difficult to obtain accurate road surface information. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0067] Figure 1 is a flowchart of the steps of a road curvature determination method provided by an embodiment of the present invention;

[0068] Figure 2 is a schematic top view of a world coordinate system provided by an embodiment of the present invention;

[0069] Figure 3 is another flowchart of the steps of a road curvature determination method provided by an embodiment of the present invention;

[0070] Figure 4 is a schematic diagram of the visible distance of a road provided by an embodiment of the present invention;

[0071] Figure 5 is a schematic diagram of a model structure provided by an embodiment of the present invention;

[0072] Figure 6 is a schematic diagram of a first geometric model provided by an embodiment of the present invention;

[0073] Figure 7The structural block diagram of a road curvature determination device provided by an embodiment of the present invention. Detailed implementation manners

[0074] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other.

[0075] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0076] Referring to Figure 1 , a flowchart of the steps of a road curvature determination method provided by an embodiment of the present invention is shown.

[0077] Step 101, obtain the environmental data around the vehicle; wherein, the environmental data includes environmental point cloud data and / or environmental image data.

[0078] During the driving process of the vehicle, for reasons such as improving driving safety and enriching the user experience, it is usually necessary to identify the road conditions around the vehicle. For example, as more and more vehicles are equipped with autonomous driving and assisted driving functions, during the autonomous driving and assisted driving processes of the vehicle, in order to enable the vehicle to drive correctly on the road, it is necessary to determine the road data in front of the vehicle in real time, which includes the road model and road curvature.

[0079] However, in some road conditions (such as curves, slopes, etc.), the road in front of the vehicle may be blocked, and at this time, it is difficult for the vehicle to obtain accurate road data, thereby affecting the safe and stable operation of systems such as autonomous driving and assisted driving. Therefore, an embodiment of the present invention provides a method for inferring relevant road data based on the environmental data around the vehicle, so that relatively accurate road data can still be obtained in the case of road blockage, and the safe and stable operation of vehicle systems such as autonomous driving and assisted driving can be maintained. It should be noted that this solution can not only be applied when the road surface in front of the vehicle is blocked, but also can be applied when the line of sight of the road surface in front is poor, and even can be applied under normal circumstances. The embodiment of the present invention does not specifically limit the usage scenarios.

[0080] In the embodiment of the present invention, the environmental data may include environmental point cloud data and / or environmental image data. The environmental point cloud data can be collected by devices such as a three-dimensional laser scanner or a lidar (LiDAR) carried by the vehicle. The environmental point cloud data can provide a three-dimensional representation of the environment around the vehicle. Each point in the environmental point cloud data may include X, Y, and Z coordinates, and may also include other information, such as reflectivity, color, etc. The environmental image data can be collected by a camera carried by the vehicle. The environmental image data can provide a two-dimensional visual representation of the environment around the vehicle. The environmental image data may include, but is not limited to, color images, grayscale images, infrared images, etc.

[0081] It should be noted that the vehicle can synchronously collect the above environmental data at preset time intervals (such as 0.1 - second intervals) or fixed frame rates (such as 30 fps) to form a continuous sequence of environmental point clouds and / or a sequence of environmental images in time. When collecting environmental point cloud data and environmental image data simultaneously, the acquisition times of a corresponding set of data in the two sequences obtained are the same.

[0082] In addition, the acquisition direction of the environmental data can be the vehicle's driving direction. For example, when the vehicle is driving forward, it can collect environmental data within a 120 - degree viewing angle range in front of the vehicle. For the convenience of subsequent processing, the viewing ranges of the environmental data can overlap with each other.

[0083] Step 102: Perform target object detection based on the environmental data to obtain a target object detection result; wherein, the target object includes stationary objects above the road surface.

[0084] In the embodiments of the present invention, after obtaining the environmental data, target object detection can be performed based on the environmental data to obtain a target object detection result. The target object can be stationary objects existing above the road area and positions near the road, and can include, but are not limited to, roadside trees, road signs, street lamps, buildings, utility poles, etc. Since these objects are relatively tall, their lines of sight are usually not easily blocked. When a certain section of the road is not visible, the target objects above its area and the nearby areas can usually still be observed from the vehicle's position, that is, they can be captured by the environmental data. Thus, the road data below the target object can be inferred from the target object detection result obtained by performing target object detection on the environmental point cloud data and the environmental image data.

[0085] In the embodiments of the present application, the environmental point cloud data or the environmental image data can be directly used for target object detection to obtain a target object detection result. Specifically, for the environmental point cloud data, a point cloud detection model based on the point cloud data can be set up, and the environmental point cloud data is input into the point cloud detection model to obtain the target object detection result output by the point cloud detection model. For the environmental image data, an image detection model based on the image data can be set up, and the environmental image data is input into the image detection model to obtain the target object detection result output by the image detection model.

[0086] It is also possible to perform target object detection based on the environmental point cloud data to obtain a first detection result, perform target object detection based on the environmental image data to obtain a second detection result, and then merge and process the first detection result and the second detection result to obtain the target object detection result. Among them, the above - mentioned merging and processing methods can include, but are not limited to, determining the result intersection or result union of the first detection result and the second detection result, and can also include averaging or weighted averaging the first detection result and the second detection result. The embodiments of the present invention do not make specific limitations.

[0087] In another embodiment, environmental point cloud data and environmental image data can also be input into a multi-modal object detection network to directly obtain object detection results. Among them, the multi-modal object detection network is a network that combines data from multiple sensors (such as cameras and LiDAR) for object detection. By integrating different types of data, this network can utilize the advantages of various sensors to improve the accuracy and robustness of object detection.

[0088] In the embodiment of the present invention, the object detection result can at least indicate the position of the object to be detected in a three-dimensional space (such as a vehicle three-dimensional coordinate system or a world three-dimensional coordinate system).

[0089] Step 103, establish a road geometry model using the object detection result.

[0090] After obtaining the object detection result, a road geometry model can be established according to the object detection result. Specifically, since the object detection result contains each object and the corresponding position information of the object, in a three-dimensional coordinate system such as the world coordinate system or the vehicle coordinate system, a road geometry model can be formed by fitting according to the position information corresponding to each object. It should be noted that when fitting, objects of the same type can be fitted, or the types of objects can be not distinguished, and directly fit various types of objects to obtain a road geometry model.

[0091] Refer to Figure 2 , Figure 2 shows a schematic top view of the world coordinate system provided by the embodiment of the present invention. As Figure 2 shown, if 12 objects (street lights) in front of the vehicle are recognized, these objects can be mapped to the world coordinate system according to the position information of these 12 objects, and a schematic top view of the world coordinate system as shown in Figure 2 is obtained, and these 12 objects 21 are fitted in the world coordinate system to obtain a road geometry model 22.

[0092] In the embodiment of the present invention, a road generation neural network model can also be pre-trained. This road generation neural network model can use the object detection result as the model input and output the corresponding road geometry model. Those skilled in the art can also use other methods to generate the road geometry model, and the embodiment of the present invention does not make specific limitations.

[0093] It should be noted that the above road geometric model can be a two-dimensional plane model or a three-dimensional solid model, and the embodiments of the present invention do not make specific limitations. For example, a two-dimensional road geometric model can be fitted based on the projection position of the target object in the three-dimensional coordinate system, and a three-dimensional road geometric model can be constructed based on the projection position and height information of the target object in the three-dimensional coordinate system.

[0094] Step 104, determine the road curvature based on the road geometric model.

[0095] The road curvature refers to the degree of bending of the road along the vehicle driving direction. The road curvature describes the bending characteristics of the road, that is, the change rate of the road tangent direction in the horizontal and vertical directions. The road curvature can be expressed as the reciprocal of the curvature radius (arc length radius), and the unit can be meters (m) or radians measured by its reciprocal (rad / m). Road curvature prediction is of great significance in modern transportation and autonomous driving systems. Accurately predicting the road curvature is the key to realizing autonomous driving functions and assisted driving functions such as precise path planning, lane keeping, and turning control. By obtaining and analyzing road curvature information in real time, the autonomous driving system can better plan the driving path of the vehicle, ensure smooth turning and reasonable vehicle speed adjustment, and improve the safety and performance of autonomous driving.

[0096] In the embodiments of the present invention, after obtaining the road geometric model, the corresponding road curvature can be determined according to the road geometric model. Specifically, in one implementation manner, the curvature of one side of the road geometric model along the road extension direction can be used as the road curvature, or the average value of the curvatures of the two sides of the road geometric model along the road extension direction can be used as the road curvature. In another implementation manner, the road center line can also be determined based on the road geometric model, and then the curvature of the road center line can be determined as the road curvature.

[0097] In summary, the embodiments of the present invention provide a method for determining road curvature, including: obtaining environmental data around the vehicle; wherein the environmental data includes environmental point cloud data and / or environmental image data; performing target object detection based on the environmental data to obtain a target object detection result; establishing a road geometric model using the target object detection result; and determining the road curvature based on the road geometric model. It is possible to establish a road geometric model based on stationary objects above the ground, and predict the road curvature according to the road geometric model. Since stationary objects above the ground are usually not easily blocked, the detection accuracy of the road curvature can be improved in the case where it is difficult to obtain accurate road surface information.

[0098] Refer to Figure 3 , Figure 3 shows a flowchart of steps of another method for determining road curvature provided by the embodiments of the present invention.

[0099] Step 201: Obtain driving road information.

[0100] In an embodiment of the present invention, when the road visibility is good, the traditional method can be used to determine the road curvature. When the road visibility is poor, the static objects in the air above the road surface can be used to predict the road curvature. Specifically, during the driving of the vehicle, the road data in the driving direction of the vehicle can be collected in real time through a lidar and / or a vision camera to obtain the driving road information. The above driving road information may include road image information and / or road point cloud information, which is not specifically limited in the embodiments of the present invention.

[0101] Step 202: Determine the road visible distance based on the driving road information.

[0102] After obtaining the driving road information, the road visible distance can be determined based on the driving road information. Among them, the road visible distance represents the distance between the farthest end of the road where the road curvature can be accurately determined through the road surface information and the vehicle itself.

[0103] Specifically, the driving road information can be processed by partitioning, and the proportion of the road surface in each partition can be determined. The closest distance between the partition with the road surface proportion less than or equal to the preset proportion and the vehicle itself is determined as the road visible distance. It is also possible to determine the road surface area in each partition, and the closest distance between the partition with the road surface area less than or equal to the preset area and the vehicle itself is determined as the road visible distance.

[0104] Refer to Figure 4 , Figure 4 shows a schematic diagram of the road visible distance provided by an embodiment of the present invention. As Figure 4 shown, when the driving road information is road image information, starting from the vehicle 41, the road image can be divided into 5 regions from 42 to 46. If the road surface proportion of regions 42 and 43 is 100%, the road surface proportion of region 44 is 80%, the road surface proportion of region 45 is 60%, and the road surface proportion of region 46 is 40%, when the preset proportion is 50%, the distance between region 46 and the vehicle can be determined as the road visible distance.

[0105] Step 203: When the road visible distance is less than or equal to the first distance, obtain the environmental data around the vehicle.

[0106] In an embodiment of the present invention, if the visible distance of the road is less than or equal to the first distance, it indicates that the current road condition is insufficient to meet the requirement of determining the road curvature through road surface information. At this time, the environmental data around the vehicle can be obtained, and the road curvature can be predicted based on the target object detection result obtained from the environmental data, so as to improve the accuracy of road area detection under the current road condition. The first distance can be flexibly set by those skilled in the art according to actual needs. For example, it can be 20 meters, and the embodiment of the present invention does not make specific limitations.

[0107] Further, in order to consider the accuracy impact of environmental light intensity on the traditional road curvature detection method, the first distance can also be adjusted according to the current environmental light intensity, so that a preset relationship is satisfied between the current environmental light intensity and the first distance (for example, the first distance can be made to decrease as the environmental light intensity increases), and the first distance is determined in real time according to the current environmental light intensity and the above preset relationship.

[0108] By obtaining the driving road information, determining the visible distance of the road based on the driving road information, and obtaining the environmental data around the vehicle when the visible distance of the road is less than or equal to the first distance, it is possible to timely adopt the method of detecting target objects to determine the road curvature in the case of poor road visibility, avoid the loss and inaccuracy of road curvature data during the vehicle driving process, and contribute to improving the driving safety of the vehicle.

[0109] Optionally, the environmental point cloud data can be obtained in the following manner of step A1 to step A3.

[0110] Step A1: Obtain the original point cloud data around the vehicle.

[0111] In an embodiment of the present invention, the original point cloud data represents the original data obtained by scanning the area around the vehicle (such as in front of the vehicle) by a lidar mounted on the vehicle, which includes the point cloud data formed by the reflection of all objects within the lidar scanning range. Since the target object is a stationary object above the road surface, the point cloud data near the ground has no reference value and can be filtered in advance to eliminate the interference it causes to subsequent steps, so as to improve the accuracy and efficiency of subsequent target object detection.

[0112] Step A2: Identify the ground point cloud points in the original point cloud data.

[0113] In an embodiment of the present invention, since the pose of the lidar installed on a specific vehicle model is usually fixed, the ground area in the original point cloud data scanned by its lidar is usually also fixed. Therefore, the ground point cloud area corresponding to the point cloud data can be recorded during the lidar calibration stage, and the ground point cloud area can be applied to the original point cloud data to determine the ground point cloud points overlapping with the ground point cloud area in the original point cloud data.

[0114] Due to the uncertainty of road conditions (such as road undulations) and the pose offset of the lidar, etc., the actual ground point cloud area may shift. Therefore, a ground detection model can also be used to determine the ground point cloud points in the original point cloud data. Specifically, a large amount of road condition training data can be collected in advance by actually driving on the road, and a model to be trained based on a convolutional neural network can be established. The model to be trained is trained with the road condition training data to obtain a ground detection model that can output the ground area according to the point cloud data. In actual application, the original point cloud data is input into the ground detection model, and the ground area output by the ground detection model is obtained, so as to determine the ground point cloud points in the ground area.

[0115] Step A3, filter out the ground point cloud points from the original point cloud data to obtain the environmental point cloud data.

[0116] In the embodiment of the present invention, after determining the ground point cloud points, the ground point cloud points can be filtered out from the original point cloud data, so as to obtain environmental point cloud data that does not contain ground point cloud points.

[0117] Furthermore, since the target object is usually located above the ground, in order to further filter out unnecessary point cloud points from the original point cloud data, the ground point cloud points in the original point cloud data and all the point cloud points below the ground point cloud points can also be filtered out to obtain environmental point cloud data, so that the amount of data of the environmental point cloud data is less, and the stability and efficiency of the subsequent processing process are further improved.

[0118] By acquiring the original point cloud data around the vehicle, identifying the ground point cloud points in the original point cloud data, and filtering out the ground point cloud points from the original point cloud data to obtain the environmental point cloud data, it is possible to make the environmental point cloud data contain less useless information and improve the accuracy and efficiency of subsequent processing.

[0119] Optionally, the environmental image data can be acquired in the following manner of Step B1 to Step B3.

[0120] Step B1, acquire the original image data around the vehicle.

[0121] In the embodiment of the present invention, the original image data represents the original image data obtained by a vision camera mounted on the vehicle by photographing the area around the vehicle (such as in front of the vehicle). Since the target object is a stationary object above the road surface, the image area near the ground has no reference value and can be filtered out in advance to eliminate the interference caused to the subsequent steps, so as to improve the accuracy and efficiency of subsequent target object detection.

[0122] Step B2, identify the ground area in the original image data.

[0123] In one embodiment, the corresponding ground image area can be recorded during the visual camera calibration phase, and the ground image area can be applied to the original image data to determine the ground area in the original image data. In another embodiment, a large amount of road condition training data can be collected in advance through actual road driving, and a model to be trained based on a convolutional neural network can be established. The model to be trained is trained with the road condition training data to obtain a ground detection model that can output the ground area according to the image data. In actual application, the original image data is input into the ground detection model to obtain the ground area output by the ground detection model.

[0124] Step B3, remove the ground area from the original image data to obtain the environmental image data.

[0125] In the embodiment of the present invention, after the ground area is determined, the ground area can be removed from the environmental image data to obtain environmental image data that does not contain ground content.

[0126] Further, since the target object is usually located above the ground, in order to further remove unnecessary images from the original image data, the ground area in the original image data and all images below the ground area can also be removed to obtain the environmental image data, so that the amount of data of the environmental image data is less, and the stability and efficiency of the subsequent processing process are further improved.

[0127] It should be noted that when the viewing ranges of the original image data and the original point cloud data are the same, the ground areas included in the two are the same. Therefore, in this case, the ground area can be determined based on the original image data or the original point cloud data, and then the ground area in the original image data is removed according to the ground area, and the ground area in the original point cloud data is filtered to avoid repeated determination of the ground area and improve the processing efficiency.

[0128] By acquiring the original image data around the vehicle, identifying the ground area in the original image data, and removing the ground area from the original image data to obtain the environmental image data, the environmental image data can contain less useless information and improve the accuracy and efficiency of subsequent processing.

[0129] Step 204, generate a point cloud bird's-eye view based on the environmental point cloud data and generate an image bird's-eye view based on the environmental image data.

[0130] In the embodiments of the present invention, in order to further improve the speed of the subsequent processing process and meet the low-latency requirements for real-time determination of road curvature at the vehicle end, the environmental data can be two-dimensionally processed to generate a point cloud bird's-eye view based on the environmental point cloud data and an image bird's-eye view based on the environmental image data. Those skilled in the art can generate the above-mentioned bird's-eye views in various ways, such as the projection method, the method of eliminating the Z-axis coordinate, and the embodiments of the present invention do not make specific limitations.

[0131] Further, before generating the point cloud bird's-eye view and the image bird's-eye view, preprocessing operations such as downsampling can also be performed on the environmental data to further reduce the amount of data to be processed subsequently and improve the processing efficiency.

[0132] Step 205: Feature-encode the point cloud bird's-eye view and the image bird's-eye view through an encoder to respectively obtain a first feature corresponding to the point cloud bird's-eye view and a second feature corresponding to the image bird's-eye view.

[0133] In the embodiments of the present invention, the point cloud bird's-eye view and the image bird's-eye view can be feature-encoded through an encoder. It should be noted that different encoders can be respectively used for feature-encoding the point cloud bird's-eye view and the image bird's-eye view. Among them, the encoder can be composed of multiple convolutional layers, pooling layers, and fully connected layers, and is used to extract features from the point cloud bird's-eye view and the image bird's-eye view.

[0134] The above encoder can be a part of an object detection model constructed based on a neural network model. Refer to Figure 5 , Figure 5 which shows a schematic diagram of a model structure provided by the embodiments of the present invention. As shown in Figure 5 , after preprocessing the original point cloud data and the original image data, a point cloud bird's-eye view and an image bird's-eye view are respectively obtained. The image bird's-eye view is input into the first encoder to obtain a first feature output by the first encoder, and the image bird's-eye view is input into the second encoder to obtain a second feature output by the second encoder.

[0135] Step 206: Fuse the first feature and the second feature to obtain a third feature.

[0136] In the embodiments of the present invention, the first feature and the second feature can be fused to obtain a third feature. By combining the feature information from different sources, the complexity and diversity of the vehicle driving environment can be better captured, thereby improving the accuracy of subsequent object detection.

[0137] Specifically, the first feature and the second feature can be concatenated, that is, fused by concatenation to obtain a third feature. The first feature and the second feature can also be fused through one or more dense layers, so that the third feature can inherit the non-linear relationship between the first feature and the second feature. Those skilled in the art can also adopt other methods for feature fusion according to actual business needs, which are not specifically limited in the embodiments of the present invention.

[0138] As Figure 5 shown, in the embodiments of the present invention, the target detection model may include a feature fusion module, and the first feature and the second feature can be fused through this feature fusion module to obtain a third feature.

[0139] Step 207, decode the third feature through a decoder to obtain the target object detection result.

[0140] In the embodiments of the present invention, the third feature can be decoded through a decoder. The decoder can be composed of multiple convolutional layers, pooling layers, and fully connected layers, and is used to determine the required target object detection result from the third feature.

[0141] The above decoder can be a part of the target detection model constructed based on the neural network model. As Figure 5 shown, after obtaining the third feature, the third feature can be input into the decoder to obtain the target object detection result output by the decoder.

[0142] In the embodiments of the present invention, a point cloud bird's-eye view can be generated based on the environmental point cloud data, and an image bird's-eye view can be generated based on the environmental image data. The point cloud bird's-eye view and the image bird's-eye view are feature-encoded through an encoder to respectively obtain the first feature corresponding to the point cloud bird's-eye view and the second feature corresponding to the image bird's-eye view. The first feature and the second feature are fused to obtain a third feature. The third feature is decoded through a decoder to obtain the target object detection result. It is possible to realize multi-modal target detection of environmental data through the encoder and the matching decoder, which helps to improve the accuracy of the target object detection result. And by feature-encoding the point cloud bird's-eye view and the image bird's-eye view, the object height information in the environmental data can be ignored, which helps to improve the efficiency of target detection and meet the real-time requirements of the vehicle terminal for the target object detection result.

[0143] Step 208, fit the size information and position information corresponding to the target objects with the same category information to obtain the number of types of the category information first geometric models.

[0144] In the embodiments of the present invention, the target objects to be detected can include various categories, such as street lamps, trees, road signs, etc. Therefore, in the target object detection results, various information such as the category information, size information, and position information of the target objects can be included. Thus, a first geometric model of a road can be generated according to the target object detection results of the same category, and the number of first geometric models obtained is the same as the number of categories of the target objects involved in the target object detection results.

[0145] Specifically, the target object detection results with the same category information can be summarized, and then the size information and position information corresponding to the target objects with the same category information are fitted to obtain the first geometric model corresponding to this category. It should be noted that in the fitting process, not only the position of the target object but also the size of the target object can be considered, so as to fit a more accurate geometric model.

[0146] Refer to Figure 6 , Figure 6 shows a schematic diagram of a first geometric model provided by an embodiment of the present invention. As Figure 6 shown, it contains target objects of two categories. Based on the position and size of the target object 61 of the street lamp category, the first geometric model 62 can be fitted, and based on the position and size of the target object 63 of the road sign type, the first geometric model 63 can be fitted.

[0147] Furthermore, for different object categories, different methods can be adopted to generate the corresponding first geometric model, so that the generation process can better meet the requirements of this object category and obtain a more accurate first geometric model. For example, a dedicated geometric generation model can be trained through machine learning technology. A geometric generation model can receive the size information and position information of a specific type of target object and generate the corresponding first geometric model. By using the trained geometric generation model, a first geometric model with higher accuracy can be obtained.

[0148] Step 209, determining the road geometric model based on the first geometric model.

[0149] After determining the first geometric models corresponding to various category information, the final road geometric model can be determined according to these first geometric models.

[0150] In one embodiment, all the first geometric models can be averaged or weighted averaged to obtain a final road geometric model. During the weighted averaging process, the proportion of the target objects corresponding to the first geometric model in all the target object detection results can be used to determine the weight of the first geometric model, that is, the more the number of target objects used to determine the first geometric model, the higher the weight corresponding to the first geometric model. For example, if 20 street lamps, 30 trees, and 10 traffic signs are identified after the above steps, a first geometric model 1 is determined based on these 20 street lamps, a first geometric model 2 is determined based on these 30 trees, and a first geometric model 3 is determined based on these 10 traffic signs. The weight of the first geometric model 1 can be 20 / 60 = 1 / 3, the weight of the first geometric model 2 can be 30 / 60 = 1 / 2, and the weight of the first geometric model 3 can be 10 / 60 = 1 / 6.

[0151] Those skilled in the art can also use other methods to process the first geometric model to obtain a road geometric model, which is not specifically limited in the embodiments of the present invention. For example, the average similarity between each first geometric model and other first geometric models can be calculated, and the first geometric model with the largest average similarity is determined as the road geometric model.

[0152] It should be noted that in the embodiments of the present invention, the category differences of the target objects may not be distinguished, and all the identified target objects can be directly fitted to obtain a road geometric model.

[0153] By fitting the size information and position information corresponding to the target objects with the same category information, the number of types of category information first geometric models can be obtained. Determining the road geometric model based on the first geometric models corresponding to multiple object categories can improve the accuracy of the road geometric model.

[0154] Step 210: Input the road geometric model into a road curvature prediction model to obtain the road curvature output by the road geometric model.

[0155] Since the actual road is relatively complex, for example, the road may have a situation of unilateral narrowing (change in the number of lanes), in order to improve the accuracy of road curvature prediction, the road geometric model can be input into a road curvature prediction model to obtain the road curvature output by the road geometric model.

[0156] Specifically, a technician can pre-construct an initial neural network model and collect training samples. The training samples can include a sample road geometry model and the corresponding sample road curvature of the sample road geometry model. Then, the sample road geometry model is input into the initial neural network model to obtain the model road curvature output by the model. Based on the model road curvature and the sample road curvature, the model loss is calculated, and the model parameters of the initial neural network model are adjusted based on the model loss. The above steps are repeatedly executed until the model loss converges, and a road curvature prediction model is obtained. Among them, the above initial neural network model can include, but is not limited to, a regression model, a classification model, etc. In the embodiments of the present invention, the above road geometry model can be input into the road curvature prediction model to obtain the road curvature output by the road curvature prediction model, thereby improving the accuracy of road curvature prediction. As Figure 5 shown, the road geometry model can be input into the road curvature prediction model to obtain the road curvature output by the road curvature prediction model.

[0157] In summary, the embodiments of the present invention provide another method for determining road curvature, including: obtaining environmental data around the vehicle; wherein, the environmental data includes environmental point cloud data and / or environmental image data; performing target object detection based on the environmental data to obtain a target object detection result; establishing a road geometry model by using the target object detection result; and determining the road curvature based on the road geometry model. It is possible to establish a road geometry model according to stationary objects in the air above the ground and predict the road curvature according to the road geometry model. Since stationary objects in the air above the ground are usually not easily blocked, the detection accuracy of road curvature can be improved in the case where it is difficult to obtain accurate road surface information.

[0158] Based on the above embodiments, the embodiments of the present invention further provide a device for determining road curvature.

[0159] Referring to Figure 7 , Figure 7 shows a structural block diagram of a device for determining road curvature provided by the embodiments of the present invention:

[0160] An acquisition module 701, configured to acquire environmental data around the vehicle; wherein, the environmental data includes environmental point cloud data and / or environmental image data;

[0161] A detection module 702, configured to perform target object detection based on the environmental data to obtain a target object detection result; wherein, the target object includes stationary objects in the air above the road surface;

[0162] A geometry model module 703, configured to establish a road geometry model by using the target object detection result;

[0163] A road curvature module 704, configured to determine the road curvature based on the road geometry model.

[0164] Optionally, the detection module includes:

[0165] An aerial view sub-module, configured to generate a point cloud aerial view based on the environmental point cloud data and generate an image aerial view based on the environmental image data;

[0166] A feature sub-module, configured to perform feature encoding on the point cloud aerial view and the image aerial view through an encoder, respectively obtaining a first feature corresponding to the point cloud aerial view and a second feature corresponding to the image aerial view;

[0167] A feature fusion sub-module, configured to fuse the first feature and the second feature to obtain a third feature;

[0168] A detection result sub-module, configured to decode the third feature through a decoder to obtain the target object detection result.

[0169] Optionally, the target object detection result includes the category information, size information, and position information of the target object, and the geometric model module includes:

[0170] A first geometric model sub-module, configured to fit the size information and position information corresponding to the target objects with the same category information to obtain the number of types of the category information of first geometric models;

[0171] A road geometric model sub-module, configured to determine the road geometric model based on the first geometric model.

[0172] Optionally, the road curvature module is further configured to input the road geometric model into a road curvature prediction model to obtain the road curvature output by the road geometric model.

[0173] Optionally, the device further includes:

[0174] A raw point cloud data module, configured to obtain the raw point cloud data around the vehicle;

[0175] A ground point cloud point module, configured to identify the ground point cloud points in the raw point cloud data;

[0176] An environmental point cloud data module, configured to filter out the ground point cloud points from the raw point cloud data to obtain the environmental point cloud data.

[0177] Optionally, the device further includes:

[0178] A raw image data module, configured to obtain the raw image data around the vehicle;

[0179] A ground area module, configured to identify the ground area in the raw image data;

[0180] An environmental image data module, configured to remove the ground area from the original image data to obtain the environmental image data.

[0181] Optionally, the device further includes:

[0182] A driving road information module, configured to obtain driving road information;

[0183] A road visible distance module, configured to determine the road visible distance based on the driving road information;

[0184] The obtaining module is further configured to execute the step of obtaining the environmental data around the vehicle when the road visible distance is less than or equal to a first distance.

[0185] In summary, an embodiment of the present invention provides a road curvature determination device, including: an obtaining module, configured to obtain the environmental data around the vehicle; wherein the environmental data includes environmental point cloud data and / or environmental image data; a detection module, configured to perform target object detection based on the environmental data to obtain a target object detection result; wherein the target object includes a stationary object above the road surface; a geometric model module, configured to establish a road geometric model by using the target object detection result; a road curvature module, configured to determine the road curvature based on the road geometric model. It can establish a road geometric model according to the stationary object above the ground, and predict the road curvature according to the road geometric model. Since the stationary object above the ground is usually not easily blocked, the detection accuracy of the road curvature can be improved when it is difficult to obtain accurate road surface information.

[0186] An embodiment of the present invention further provides a readable storage medium. When the instructions in the readable storage medium are executed by a processor of an electronic device, the electronic device can execute the above-mentioned road curvature determination method.

[0187] An embodiment of the present invention further provides an electronic device, where the electronic device includes a processor and a memory, the memory stores a program or instructions running on the processor, and when the program or the instructions are executed by the processor, the above-mentioned road curvature determination method is implemented.

[0188] An embodiment of the present invention further provides a vehicle controller. The vehicle controller includes a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the above-mentioned road curvature determination method.

[0189] An embodiment of the present invention further provides a vehicle, including the above-mentioned vehicle controller.

[0190] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing device embodiments and will not be elaborated herein.

[0191] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0192] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present invention and should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for determining road curvature, characterized in that, The method includes: Obtaining environmental data around the vehicle; wherein, the environmental data includes environmental point cloud data and / or environmental image data; Performing target object detection based on the environmental data to obtain a target object detection result; wherein, the target object includes stationary objects above the road surface; Establishing a road geometric model using the target object detection result; Determining the road curvature based on the road geometric model.

2. The method according to claim 1, wherein The performing target object detection based on the environmental data to obtain a target object detection result includes: Generating a point cloud bird's-eye view based on the environmental point cloud data and generating an image bird's-eye view based on the environmental image data; Performing feature encoding on the point cloud bird's-eye view and the image bird's-eye view through an encoder to respectively obtain a first feature corresponding to the point cloud bird's-eye view and a second feature corresponding to the image bird's-eye view; Fusing the first feature and the second feature to obtain a third feature; Decoding the third feature through a decoder to obtain the target object detection result.

3. The method according to claim 1, wherein The target object detection result includes the category information, size information, and position information of the target object. The establishing a road geometric model using the target object detection result includes: Fitting the size information and position information corresponding to the target objects with the same category information to obtain a first geometric model for each type of the category information; Determining the road geometric model based on the first geometric model.

4. The method according to claim 1, wherein The determining the road curvature based on the road geometric model includes: Inputting the road geometric model into a road curvature prediction model to obtain the road curvature output by the road geometric model.

5. The method according to claim 1, wherein The method further includes: Obtaining the original point cloud data around the vehicle; Identifying the ground point cloud points in the original point cloud data; Filtering out the ground point cloud points from the original point cloud data to obtain the environmental point cloud data.

6. The method according to claim 1, wherein The method further includes: Obtaining the original image data around the vehicle; Identifying the ground area in the original image data; Removing the ground area from the original image data to obtain the environmental image data.

7. The method according to claim 1, characterized in that The method further includes: Obtaining driving road information; Determining the road visible distance based on the driving road information; Performing the step of obtaining the environmental data around the vehicle when the road visible distance is less than or equal to a first distance.

8. A road curvature determination device, characterized in that, The device includes: An obtaining module, configured to obtain environmental data around the vehicle; wherein, the environmental data includes environmental point cloud data and / or environmental image data; A detecting module, configured to perform target object detection based on the environmental data to obtain a target object detection result; wherein, the target object includes stationary objects above the road surface; A geometric model module, configured to establish a road geometric model using the target object detection result; A road curvature module, configured to determine the road curvature based on the road geometric model.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory. The memory stores a program or instruction that runs on the processor. When the program or the instruction is executed by the processor, the road curvature determination method according to any one of claims 1 to 7 is implemented.

10. A readable storage medium, characterized in that, When the instructions in the readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the road curvature determination method according to any one of claims 1 to 7.

Images