Roadside detection method, device, vehicle and computer-readable storage medium

Abstract

The present application discloses a curb detection method, device, vehicle and computer-readable storage medium, the method comprising: obtaining a first image to be predicted corresponding to a plurality of surround view cameras of a first vehicle and a second image to be predicted corresponding to a plurality of surround view cameras; based on the first image to be predicted and the second image to be predicted, determining first curb prediction data corresponding to the camera perspective and / or second curb prediction data corresponding to the bird's-eye view perspective; based on the first curb prediction data and / or the second curb prediction data, determining target curb prediction data. The present application can perform purely visual curb prediction through the first image to be predicted and the second image to be predicted, thereby improving the accuracy and precision of the curb prediction data, and reducing the cost of curb detection compared to curb detection using a laser radar.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a curb detection method, device, vehicle and computer-readable storage medium. Background Art

[0002] In autonomous driving vehicles, sensors are needed to perceive the environment around the vehicle to obtain perception information to control the vehicle, such as controlling vehicle driving, obstacle avoidance, parking, etc.

[0003] Currently, when parking a vehicle, ultrasound can be used to detect potential parking spaces, and then a path can be planned to accurately park the vehicle at the corresponding location. However, when detecting potential parking spaces, only the length and estimated width of the parking space can be obtained, and the curb of the parking position cannot be sensed. Therefore, there is a risk of hitting the curb when following the planned path. Summary of the invention

[0004] The main purpose of the present application is to provide a curb detection method, device, vehicle and computer-readable storage medium, aiming to solve the technical problem of low accuracy of curb recognition in related technologies.

[0005] To achieve the above object, the present application provides a roadside detection method, the roadside detection method comprising the following steps:

[0006] Acquire a first image to be predicted corresponding to a plurality of surround view cameras of a first vehicle and a second image to be predicted corresponding to a plurality of surround view cameras;

[0007] Determine, based on the first image to be predicted and the second image to be predicted, first road edge prediction data corresponding to a camera perspective and / or second road edge prediction data corresponding to a bird's-eye view;

[0008] Based on the first curb prediction data and / or the second curb prediction data, target curb prediction data is determined.

[0009] Further, the determining, based on the first image to be predicted and the second image to be predicted, first road edge prediction data corresponding to the camera viewing angle includes:

[0010] The first image to be predicted and the second image to be predicted are input into a neural network model of the camera perspective for model prediction to obtain first curb prediction data, wherein the first curb prediction data includes first curb position data, first curb height data and a first occupancy grid.

[0011] Furthermore, before the step of inputting the first image to be predicted and the second image to be predicted into the neural network model of the camera perspective for model prediction to obtain the first road edge prediction data, the step further includes:

[0012] Acquire first surrounding view images corresponding to a plurality of surrounding view cameras of a second vehicle and first surrounding view images corresponding to a plurality of surround view cameras;

[0013] Performing a labeling operation on the first surrounding view image and the first surround view image respectively to obtain first roadside labeling information;

[0014] The first roadside marking information, the first surrounding image and the first surround image are input into a neural network model to be trained in the camera perspective for model training to obtain the neural network model.

[0015] Further, the obtaining, based on the first image to be predicted and the second image to be predicted, determining the second roadside prediction data corresponding to the bird's-eye view, includes:

[0016] The first image to be predicted and the second image to be predicted are input into the BEV network model with a bird's-eye view for model prediction to obtain the second curb prediction data, wherein the second curb prediction data includes second curb position data, second curb height data and a second occupancy grid.

[0017] Furthermore, before the step of inputting the first image to be predicted and the second image to be predicted into the BEV network model with a bird's-eye view for model prediction to obtain the second roadside prediction data, the method further includes:

[0018] Acquire second surrounding view images corresponding to a plurality of surrounding view cameras of a second vehicle and second surrounding view images corresponding to a plurality of surround view cameras;

[0019] performing a labeling operation on the second surrounding view image and the second surround view image respectively to obtain second roadside labeling information;

[0020] The second roadside marking information, the second surrounding image and the second surround image are input into the to-be-trained bird's-eye view network model for model training to obtain the BEV network model of the bird's-eye view.

[0021] Furthermore, the step of respectively labeling the second surrounding image and the second surround image to obtain the second roadside labeling information includes:

[0022] Acquire first point cloud data corresponding to the laser radar of the second vehicle;

[0023] Obtaining third roadside annotation information corresponding to the first point cloud data;

[0024] Based on the third roadside labeling information, the second roadside labeling information is determined.

[0025] Furthermore, the step of obtaining the third roadside annotation information corresponding to the point cloud data includes:

[0026] Performing grid processing on the three-dimensional space corresponding to the first point cloud data to obtain first grid data;

[0027] Based on the first point cloud data, determining an occupied grid in the third roadside annotation information in the first grid data;

[0028] The first point cloud data is annotated based on a preset network model to obtain the curb position data and curb height data in the third curb annotation information.

[0029] In addition, to achieve the above-mentioned purpose, the present application also provides a vehicle, the vehicle comprising:

[0030] An acquisition module, used to acquire first images to be predicted corresponding to a plurality of surround view cameras of the vehicle and second images to be predicted corresponding to a plurality of surround view cameras;

[0031] A prediction module, used to determine first road edge prediction data corresponding to a camera perspective and / or second road edge prediction data corresponding to a bird's-eye view based on the first image to be predicted and the second image to be predicted;

[0032] A determination module is used to determine target curb prediction data based on the first curb prediction data and / or the second curb prediction data.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a curb detection device, which includes: a memory, a processor, and a curb detection program stored in the memory and executable on the processor, and the curb detection program implements the steps of the aforementioned curb detection method when executed by the processor.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which a curb detection program is stored. When the curb detection program is executed by a processor, the steps of the aforementioned curb detection method are implemented.

[0035] The present application obtains first images to be predicted corresponding to multiple surround-view cameras of a first vehicle and second images to be predicted corresponding to multiple surround-view cameras; then, based on the first images to be predicted and the second images to be predicted, determines first curb prediction data corresponding to the camera perspective and / or second curb prediction data corresponding to the bird's-eye view perspective; and then, based on the first curb prediction data and / or the second curb prediction data, determines target curb prediction data. Purely visual curb prediction can be performed through the first images to be predicted and the second images to be predicted, thereby improving the accuracy and precision of curb prediction data and reducing the cost of curb detection compared to curb detection using lidar. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a structural schematic diagram of a curb detection device in a hardware operating environment involved in an embodiment of the present application;

[0037] Figure 2 This is a schematic diagram of the process of the first embodiment of the curb detection method of the present application;

[0038] Figure 3 This is a schematic diagram of the functional modules of a vehicle according to an embodiment of the present application.

[0039] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0040] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] like Figure 1 As shown, Figure 1 It is a structural diagram of a curb detection device in a hardware operating environment involved in an embodiment of the present application.

[0042] The roadside detection device in the embodiment of the present application may be a vehicle. Figure 1As shown, the roadside detection device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0043] Optionally, the roadside detection device may further include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, etc. Among them, the sensor may be a light sensor, a motion sensor, or other sensors. Of course, the roadside detection device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be described in detail here.

[0044] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation on the curb detection device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0045] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a roadside detection program.

[0046] exist Figure 1 In the curb detection device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the curb detection program stored in the memory 1005.

[0047] In this embodiment, the curb detection device includes: a memory 1005, a processor 1001, and a curb detection program stored in the memory 1005 and executable on the processor 1001, wherein the processor 1001 calls the curb detection program stored in the memory 1005 and executes the steps of the curb detection method in the following embodiments.

[0048] The present application also provides a roadside detection method, referring to Figure 2 , Figure 2 This is a flow chart of the first embodiment of the curb detection method of the present application.

[0049] The first vehicle is provided with a plurality of surround-view cameras and a plurality of surround-view cameras, wherein the surround-view cameras can be fisheye cameras, and the number of the surround-view cameras and the surround-view cameras can be reasonably set. The plurality of surround-view cameras can realize image capture of the 360° environment around the first vehicle, and the plurality of surround-view cameras can also realize image capture of the 360° environment around the first vehicle. For example, the surround-view cameras can be set to 7, which are respectively arranged on the left and right sides of the front end of the first vehicle (to capture the image corresponding to the front of the vehicle), the left and right sides of the middle part of the first vehicle (to capture the image corresponding to the middle part of the vehicle), the left and right sides of the rear end of the first vehicle (to capture the image corresponding to the rear of the vehicle) and the top of the first vehicle, and the surround-view surround-view cameras can be set to 4, which are respectively arranged on the left and right sides of the front end of the first vehicle and the left and right sides of the rear end of the first vehicle.

[0050] The roadside detection method comprises:

[0051] Step S101, obtaining first images to be predicted corresponding to multiple surround view cameras of a first vehicle and second images to be predicted corresponding to multiple surround view cameras;

[0052] Step S102, determining first road edge prediction data corresponding to a camera perspective and / or second road edge prediction data corresponding to a bird's-eye view based on the first image to be predicted and the second image to be predicted;

[0053] Step S103: determining target roadside prediction data based on the first roadside prediction data and / or the second roadside prediction data.

[0054] When the vehicle needs to perform parallel parking, the driver can trigger the side parking command through the vehicle's display device or APP, or for an autonomous driving vehicle, it can determine that side parking is required based on current environmental perception information, and then automatically trigger the side parking command. At this time, multiple first images to be predicted are obtained through multiple surround-view cameras of the first vehicle, and multiple second images to be predicted are obtained through multiple surround-view cameras of the first vehicle.

[0055] After acquiring the first image to be predicted and the second image to be predicted, the first vehicle acquires first roadside prediction data corresponding to the camera perspective and second roadside prediction data corresponding to the bird's-eye view perspective based on the first image to be predicted and the second image to be predicted, respectively. Further, in a possible implementation, step S102 includes:

[0056] Step S1021, input the first image to be predicted and the second image to be predicted into the neural network model of the camera perspective for model prediction, and obtain first curb prediction data, wherein the first curb prediction data includes first curb position data, first curb height data and a first occupied grid.

[0057] Further, in another possible implementation, step S102 includes:

[0058] Step S1022, input the first image to be predicted and the second image to be predicted into the BEV network model with a bird's-eye view for model prediction, and obtain second curb prediction data corresponding to the vehicle, wherein the second curb prediction data includes second curb position data, second curb height data and a second occupied grid.

[0059] It should be noted that before performing the curb prediction, model training is first performed to obtain a neural network model and / or a BEV (Bird's Eye View) network model, and the neural network model and / or the BEV network model is loaded onto the first vehicle.

[0060] If the vehicle is currently loaded with a neural network model of the camera perspective, the first image to be predicted and the second image to be predicted are input into the neural network model of the camera perspective for model prediction to obtain first curb prediction data, that is, the output of the neural network model of the camera perspective is used as the first curb prediction data, so as to accurately obtain the first curb prediction data through the neural network model, realize pure visual curb detection, improve the convenience of curb detection, and reduce the cost of curb detection, wherein the first curb prediction data includes first curb position data, first curb height data, and a first occupied grid, the first curb position data is the position data of the curb in the parking area corresponding to the first vehicle, the first curb height data is the height data of the curb in the parking area corresponding to the first vehicle, and the first occupied grid is the grid information occupied by the curb in the parking area corresponding to the first vehicle.

[0061] If the vehicle is currently loaded with a BEV network model with a bird's-eye view, the first image to be predicted and the second image to be predicted are input into the BEV network model with a bird's-eye view for model prediction to obtain second curb prediction data, that is, the output of the BEV network model with a bird's-eye view is used as the second curb prediction data, so as to accurately obtain the second curb prediction data through the BEV network model, realize pure visual curb detection, improve the convenience of curb detection, and reduce the cost of curb detection, wherein the second curb prediction data includes second curb position data, second curb height data and a second occupied grid, the second curb position data is the position data of the curb in the parking area corresponding to the first vehicle, the second curb height data is the height data of the curb in the parking area corresponding to the first vehicle, and the second occupied grid is the grid information occupied by the curb in the parking area corresponding to the first vehicle.

[0062] Among them, when the BEV network model performs model prediction from a bird's-eye view, the convolutional neural network in the BEV network model is used to extract features from each input (the first image to be predicted and the second image to be predicted), and then the transformer in the BEV network model is used to convert the extracted features from the camera perspective to a bird's-eye view to obtain a bird's-eye view. The BEV network model uses the bird's-eye view to make a prediction and obtain the second roadside prediction data.

[0063] After obtaining the first curb prediction data and / or the second curb prediction data, the first vehicle determines the target curb prediction data based on the first curb prediction data and / or the second curb prediction data. For example, if either the first curb prediction data or the second curb prediction data is obtained, the target curb prediction data is the first curb prediction data or the second curb prediction data. If the first curb prediction data and the second curb prediction data are obtained at the same time, the target curb prediction data includes the first curb prediction data and the second curb prediction data; or, the target curb prediction data is obtained by performing mean processing on the first curb prediction data and the second curb prediction data, that is, the mean between the first curb position data and the second curb position data is the curb position data in the target curb prediction data, the mean between the first curb height data and the second curb height data is the curb height data in the target curb prediction data, and the mean between the first occupied grid and the second occupied grid is the occupied grid in the target curb prediction data; or, the second curb prediction data is corrected by the first curb prediction data or the first curb prediction data is corrected by the second curb prediction data to obtain the target curb prediction data, so as to further improve the accuracy and precision of the curb prediction.

[0064] After obtaining the target curb prediction data, the first vehicle may perform a parallel parking operation based on the target curb prediction data, that is, the target curb prediction data may be used to assist the vehicle in performing parallel parking. For example, for an autonomous driving vehicle, the first vehicle may directly perform parallel parking based on the target curb prediction data. For a non-autonomous driving vehicle, the first vehicle may display the target curb prediction data on a display device, thereby improving the accuracy of the vehicle's parallel parking.

[0065] It should be noted that the curb detection method can be applied to the first vehicle, or a server that is in communication with the first vehicle, and the server sends the target curb prediction data to the first vehicle.

[0066] By acquiring a first image to be predicted corresponding to a plurality of surround-view cameras of a first vehicle and a second image to be predicted corresponding to a plurality of surround-view cameras; then determining, based on the first image to be predicted and the second image to be predicted, first curb prediction data corresponding to the camera perspective and / or second curb prediction data corresponding to the bird's-eye view perspective; and then determining target curb prediction data based on the first curb prediction data and / or the second curb prediction data, a purely visual curb prediction can be performed through the first image to be predicted and the second image to be predicted, thereby improving the accuracy and precision of the curb prediction data and reducing the cost of curb detection compared to curb detection using a lidar.

[0067] Based on the first embodiment, a second embodiment of the roadside detection method of the present application is proposed, wherein before step S1021, the roadside detection method further includes:

[0068] Step S201, obtaining first surrounding view images corresponding to a plurality of surrounding view cameras of a second vehicle and first surrounding view images corresponding to a plurality of surround view cameras;

[0069] Step S202, performing a labeling operation on the first surrounding view image and the first surround view image respectively to obtain first roadside labeling information;

[0070] Step S203: input the first roadside marking information, the first surrounding image and the first surround image into a neural network model to be trained in the camera perspective to perform model training and obtain the neural network model.

[0071] Among them, the setting method of each surround-view camera and each surround-view camera in the second vehicle is the same as the setting method of each surround-view camera and each surround-view camera in the first vehicle, and the second vehicle can be a data collection vehicle.

[0072] When performing model training of the neural network model, first surrounding view images corresponding to multiple surrounding view cameras of the second vehicle and first surrounding view images corresponding to multiple surround view cameras are obtained, wherein the first surrounding view images and the first surround view images can be images at multiple moments, so as to obtain multiple groups of surrounding view images and surround view images for model training, each group of images including surround view images taken by multiple surround view cameras at the same moment and surrounding view images taken by multiple surround view cameras.

[0073] After acquiring the first peripheral view image and the first surround view image, the first peripheral view image and the first surround view image are respectively annotated to obtain first curb annotation information, wherein the first curb annotation information includes curb position annotation data, curb height annotation data and occupied grid annotation data. Specifically, the first curb annotation information can be determined by the point cloud data of the laser radar set on the second vehicle, wherein the first curb annotation information includes curb position annotation data, curb height annotation data and occupied grid annotation data. The method for acquiring the first curb annotation information is the same as the method for acquiring the second curb annotation information. Specifically, refer to the process of acquiring the second curb annotation information, which will not be repeated here.

[0074] After obtaining the first curb annotation information, the first curb annotation information, the first surround image, and the first surround image are input into the neural network model to be trained for model training to obtain the neural network model. Specifically, the model training is performed according to each of the multiple groups of images corresponding to the first surround image and the first surround image, and the loss function of the trained model is calculated by the prediction result of each group of images and the first curb annotation information. When the loss function is less than the preset loss function, the currently trained neural network model to be trained is used as the neural network model, otherwise, the currently trained neural network model to be trained is used as the neural network model to be trained, and the process returns to step S203.

[0075] It should be noted that the training process of the neural network model can be applied to a server that is communicatively connected to the first vehicle. The server can obtain the first surround image collected by the second vehicle and the first surround image. After obtaining the neural network model, the server sends the neural network model to multiple associated vehicles including the first vehicle (for example, all vehicles that can be communicatively linked to the server).

[0076] By acquiring first peripheral view images corresponding to multiple peripheral view cameras of a second vehicle and first surround view images corresponding to multiple surround view cameras; then respectively annotating the first peripheral view images and the first surround view images to obtain first curb annotation information; then inputting the first curb annotation information, the first peripheral view images and the first surround view images into a neural network model to be trained in the camera perspective for model training to obtain the neural network model, and performing model training through the first peripheral view images, the first surround view images and the annotated data to obtain the neural network model, thereby improving the accuracy of the neural network model and further improving the accuracy and precision of curb prediction through the neural network model.

[0077] Based on the first embodiment, a third embodiment of the curb detection method of the present application is proposed, wherein before step S1022, the curb detection method further includes:

[0078] Step S301, obtaining second surrounding view images corresponding to a plurality of surrounding view cameras of a second vehicle and second surrounding view images corresponding to a plurality of surround view cameras;

[0079] Step S302, performing a labeling operation on the second surrounding view image and the second surround view image respectively to obtain second roadside labeling information;

[0080] Step S303: input the second roadside marking information, the second surrounding image and the second surround image into a to-be-trained bird's-eye view network model for model training to obtain the BEV network model of the bird's-eye view.

[0081] Among them, the setting method of each surround-view camera and each surround-view camera in the second vehicle is the same as the setting method of each surround-view camera and each surround-view camera in the first vehicle, and the second vehicle can be a data collection vehicle.

[0082] When performing model training of the BEV network model, second surrounding view images corresponding to multiple surrounding view cameras of the second vehicle and second surrounding view images corresponding to multiple surround view cameras are obtained, wherein the second surrounding view images and second surrounding view images may be images at multiple moments, so as to obtain multiple groups of surrounding view images and surround view images for model training, each group of images including surround view images taken by multiple surround view cameras at the same moment and surrounding view images taken by multiple surround view cameras.

[0083] After acquiring the second surrounding image and the second surround image, the second surrounding image and the second surround image are respectively annotated to obtain second curb annotation information, where the second curb annotation information includes curb position annotation data, curb height annotation data, and occupied grid annotation data. Specifically, in a possible implementation, step S302 includes:

[0084] Step S3021, obtaining first point cloud data corresponding to the laser radar of the second vehicle;

[0085] Step S3022, obtaining third roadside annotation information corresponding to the first point cloud data;

[0086] Step S3023: Determine the second roadside marking information based on the third roadside marking information.

[0087] Among them, the second vehicle is equipped with a laser radar, and after acquiring the second peripheral view image and the second surround view image, the first point cloud data corresponding to the laser radar of the second vehicle is acquired. The first point cloud data may be acquisition data of the laser radar at multiple times, and the time of acquisition of each data in the first point cloud data is the same as the shooting time of the image in the second peripheral view image.

[0088] After the first point cloud data is obtained, the third curb annotation information corresponding to the first point cloud data is obtained, wherein the second curb annotation information includes curb position annotation data, curb height annotation data, and occupied grid annotation data. Specifically, in a possible implementation, step S3022 includes:

[0089] Step a, performing grid processing on the three-dimensional space corresponding to the first point cloud data to obtain first grid data;

[0090] Step b, determining an occupied grid in the third roadside annotation information in the first grid data based on the first point cloud data;

[0091] Step c: annotate the first point cloud data based on a preset network model to obtain the curb position data and curb height data in the third curb annotation information.

[0092] After obtaining the first point cloud data, the three-dimensional space corresponding to the first point cloud data is gridded to obtain the first grid data. For example, the three-dimensional space corresponding to the first point cloud data is a three-dimensional space of 20m*20m*5m. The three-dimensional space corresponding to the first point cloud data is gridded according to a preset resolution (such as 0.1m) to obtain the first grid data. Then, based on the first point cloud data, the occupied grids in the third curb annotation information are determined in the first grid data, that is, the grids in the first grid data that exist in the point cloud in the first point cloud data are taken as occupied grids. Of course, the intensity of each occupied grid (the number of point cloud data) can also be counted.

[0093] After obtaining the occupied grid, the first point cloud data is annotated based on the preset network model to obtain the curb position data and the curb height data in the third curb annotation information. Specifically, a large-scale network for lidar information is used to automatically annotate the first point cloud data to obtain the curb position data and the curb height data in the third curb annotation information, wherein the large-scale network is a trained network to accurately obtain the third curb annotation information.

[0094] After obtaining the third curb annotation information, the second curb annotation information is determined based on the third curb annotation information. Specifically, the third curb annotation information can be converted into the second curb annotation information according to the relationship between the point cloud data and the second surrounding image and the second surrounding image.

[0095] After obtaining the second curb annotation information, the second curb annotation information, the second surrounding image, and the second surround image are input into the to-be-trained bird's-eye view network model for model training to obtain the BEV network model of the bird's-eye view. Specifically, model training is performed according to each of the multiple groups of images corresponding to the second surrounding image and the second surround image, and the loss function of the trained model is calculated by the prediction result of each group of images and the second curb annotation information. When the loss function is less than the preset loss function, the currently trained bird's-eye view network model to be trained is used as the BEV network model, otherwise, the currently trained bird's-eye view network model to be trained is used as the to-be-trained bird's-eye view network model, and the process returns to step S303.

[0096] It should be noted that the training process of the bird's-eye view network model can be applied to a server that is communicatively connected to the first vehicle. The server can obtain the first surround view image and the first surround view image collected by the second vehicle. After obtaining the BEV network model, the server sends the BEV network model to multiple associated vehicles including the first vehicle (for example, all vehicles that can be communicatively linked to the server only).

[0097] By acquiring the second surrounding view images corresponding to multiple surrounding view cameras of the second vehicle and the second surround view images corresponding to multiple surround view cameras; then respectively annotating the second surrounding view images and the second surround view images to obtain second curb annotation information; then inputting the second curb annotation information, the second surrounding view images and the second surround view images into a to-be-trained bird's-eye view network model for model training to obtain the BEV network model of the bird's-eye view, and performing model training through the second surrounding view images, the second surround view images and the annotated data to obtain the BEV network model, thereby improving the accuracy of the BEV network model and further improving the accuracy and precision of curb prediction through the BEV network model.

[0098] In addition, the present application also proposes a vehicle, referring to Figure 3 , the vehicle comprises:

[0099] An acquisition module 10 is used to acquire first images to be predicted corresponding to a plurality of surround view cameras of the vehicle and second images to be predicted corresponding to a plurality of surround view cameras;

[0100] A prediction module 20, configured to determine first road edge prediction data corresponding to a camera perspective and / or second road edge prediction data corresponding to a bird's-eye view based on the first image to be predicted and the second image to be predicted;

[0101] The determination module 30 is used to determine target roadside prediction data based on the first roadside prediction data and / or the second roadside prediction data.

[0102] The methods executed by the above-mentioned program units can refer to the various embodiments of the curb detection method of the present application, and will not be repeated here.

[0103] In addition, an embodiment of the present application further proposes a computer-readable storage medium, on which a curb detection program is stored. When the curb detection program is executed by a processor, the steps of the curb detection method described above are implemented.

[0104] In addition, an embodiment of the present application also proposes a computer program product, which includes a curb detection program. When the curb detection program is executed by a processor, the steps of the curb detection method described above are implemented.

[0105] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0106] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0108] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A road edge detection method, characterized in that: The roadside detection method comprises the following steps: Acquire a first image to be predicted corresponding to a plurality of surround view cameras of a first vehicle and a second image to be predicted corresponding to a plurality of surround view cameras; Determine, based on the first image to be predicted and the second image to be predicted, first road edge prediction data corresponding to a camera perspective and / or second road edge prediction data corresponding to a bird's-eye view; Determining target curb prediction data based on the first curb prediction data and / or the second curb prediction data; The determining, based on the first image to be predicted and the second image to be predicted, first curb prediction data corresponding to the camera perspective includes: inputting the first image to be predicted and the second image to be predicted into a neural network model of the camera perspective for model prediction to obtain first curb prediction data, wherein the first curb prediction data includes first curb position data, first curb height data, and a first occupied grid; The method of determining the second curb prediction data corresponding to the bird's-eye view based on the first image to be predicted and the second image to be predicted includes: inputting the first image to be predicted and the second image to be predicted into the BEV network model of the bird's-eye view for model prediction to obtain the second curb prediction data, wherein the second curb prediction data includes second curb position data, second curb height data and a second occupied grid.

2. The road edge detection method according to claim 1, characterized in that: Before the step of inputting the first image to be predicted and the second image to be predicted into the neural network model of the camera perspective for model prediction to obtain the first road edge prediction data, the step further includes: Acquire first surrounding view images corresponding to a plurality of surrounding view cameras of a second vehicle and first surrounding view images corresponding to a plurality of surround view cameras; Performing a labeling operation on the first surrounding view image and the first surround view image respectively to obtain first roadside labeling information; The first roadside marking information, the first surrounding image and the first surround image are input into a neural network model to be trained in the camera perspective for model training to obtain the neural network model.

3. The road edge detection method according to claim 1 or 2, characterized in that: Before the step of inputting the first image to be predicted and the second image to be predicted into the BEV network model with a bird's-eye view for model prediction to obtain the second roadside prediction data, the method further includes: Acquire second surrounding view images corresponding to a plurality of surrounding view cameras of a second vehicle and second surrounding view images corresponding to a plurality of surround view cameras; performing a labeling operation on the second surrounding view image and the second surround view image respectively to obtain second roadside labeling information; The second roadside marking information, the second surrounding image and the second surround image are input into the to-be-trained bird's-eye view network model for model training to obtain the BEV network model of the bird's-eye view.

4. The road edge detection method according to claim 3, characterized in that: The step of respectively labeling the second surrounding image and the second surround image to obtain the second roadside labeling information comprises: Acquire first point cloud data corresponding to the laser radar of the second vehicle; Obtaining third roadside annotation information corresponding to the first point cloud data; Based on the third roadside labeling information, the second roadside labeling information is determined.

5. The road edge detection method according to claim 4, characterized in that: The step of obtaining the third roadside annotation information corresponding to the point cloud data comprises: Performing grid processing on the three-dimensional space corresponding to the first point cloud data to obtain first grid data; Based on the first point cloud data, determining an occupied grid in the third roadside annotation information in the first grid data; The first point cloud data is annotated based on a preset network model to obtain the curb position data and curb height data in the third curb annotation information.

6. A vehicle, characterized in that: The vehicle comprises: An acquisition module, used to acquire first images to be predicted corresponding to a plurality of surround view cameras of the vehicle and second images to be predicted corresponding to a plurality of surround view cameras; A prediction module, used to determine first road edge prediction data corresponding to a camera perspective and / or second road edge prediction data corresponding to a bird's-eye view based on the first image to be predicted and the second image to be predicted; A determination module, configured to determine target curb prediction data based on the first curb prediction data and / or the second curb prediction data; The prediction module is further used to input the first image to be predicted and the second image to be predicted into a neural network model of a camera perspective for model prediction to obtain first curb prediction data, wherein the first curb prediction data includes first curb position data, first curb height data, and a first occupied grid; The first image to be predicted and the second image to be predicted are input into the BEV network model with a bird's-eye view for model prediction to obtain the second curb prediction data, wherein the second curb prediction data includes second curb position data, second curb height data and a second occupancy grid.

7. A road edge detection device, characterized in that: The curb detection device comprises: a memory, a processor, and a curb detection program stored in the memory and executable on the processor. When the curb detection program is executed by the processor, the steps of the curb detection method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a curb detection program, and when the curb detection program is executed by a processor, the steps of the curb detection method according to any one of claims 1 to 5 are implemented.

Images