Method for vehicle visual rendering, in-vehicle device and vehicle

Abstract

The present disclosure relates to a method for vehicle visual presentation, an in-vehicle device, and a vehicle, which can be applied to ordinary vehicles or autonomous vehicles. The above method includes: acquiring image data of the surrounding environment collected in real time by a camera device of the vehicle; performing obstacle detection on the above image data at each moment to obtain a detection result of whether there is an obstacle and the corresponding obstacle type when there is an obstacle; when the above detection result indicates the presence of an obstacle, determining the three-dimensional position information of the above obstacle according to the above image data, the parameters of the above camera device, and the real-time position information of the above vehicle; and rendering the corresponding obstacle in the vehicle driving map loaded in real time on the display interface of the in-vehicle device of the above vehicle according to the above three-dimensional position information and the above obstacle type. The presentation effect is intuitive for non-technical users and helps to improve the user's perception of the road conditions outside the vehicle during driving, and has portability.

Claims

1. A method for vehicle visual presentation, characterized in that, it includes: Obtaining image data of the surrounding environment collected in real time by a camera device of the vehicle; Performing obstacle detection on the image data at each moment to obtain a detection result of whether there is an obstacle and the corresponding obstacle type when there is an obstacle; When the detection result indicates the existence of an obstacle, determining the three-dimensional position information of the obstacle according to the image data, the parameters of the camera device, and the real-time position information of the vehicle; and Rendering the corresponding obstacle in the vehicle driving map loaded in real time on the display interface of the vehicle's on-board device according to the three-dimensional position information and the obstacle type, including: calculating the covering position and covering area of the primitive to be rendered corresponding to the obstacle in the vehicle driving map according to the three-dimensional position information and the obstacle type of the obstacle; determining whether there is a first target primitive within the display range of the display interface and a second target primitive outside the display range of the display interface among the primitives to be rendered according to the covering position and the covering area; when there is the first target primitive among the primitives to be rendered, rendering the first target primitive in 3D on the display interface; when there is the second target primitive among the primitives to be rendered, storing the second target primitive in a rendering list, and refreshing the storage state of the second target primitive in the rendering list according to the real-time display range of the display interface, and when the second target primitive is within the real-time display range, deleting the second target primitive from the rendering list and outputting it to the display interface for rendering.

2. The method according to claim 1, characterized in that, the vehicle driving map is loaded in the following manner: Obtaining the planned path of the vehicle; Determining the map data to be displayed from the current driving position of the vehicle to the subsequent planned path; Calculating a target map interval adapted to the current display parameters in the display interface of the on-board device according to the map data, where the current display parameters are obtained by being pre-configured or updated in real time by the user; and Loading the data corresponding to the target map interval in real time on the display interface to obtain the vehicle driving map.

3. The method according to claim 1, characterized in that, it further includes: When the detection result indicates the existence of an obstacle, extracting the corresponding attention features of the obstacles in the image data according to the preset attention features for each obstacle type and the obstacle type corresponding to the existing obstacles; The preset attention features are used to characterize the identity differences of the same type of obstacles; Comparing the similarity of the attention features extracted from the same type of obstacles in the image data at two consecutive moments to obtain an identity recognition result of whether there is the same obstacle in the image data at the two consecutive moments; According to the identity recognition result, assigning the same type identifier and respective distinct identity identifiers to the same type of obstacles, and assigning different type identifiers to different types of obstacles; Among them, rendering the corresponding obstacle in the vehicle driving map that is real-time loaded on the display interface of the in-vehicle device of the vehicle according to the three-dimensional position information and the obstacle type includes: Rendering the obstacle in the vehicle driving map that is real-time loaded on the display interface of the in-vehicle device of the vehicle according to the type identifier, identity identifier, and three-dimensional position information of the obstacle.

4. The method according to claim 3, wherein, when the type identifiers and identity identifiers of the obstacles at two consecutive moments are the same, rendering the obstacles at the two consecutive moments with the same 3D graphics in the vehicle driving map; when the type identifiers of the obstacles at two consecutive moments are different, rendering the obstacles at the two consecutive moments with differentiated 3D graphics in the vehicle driving map, and the differentiated 3D graphics are at least different in shape; when the type identifiers of the obstacles at two consecutive moments are the same and the identity identifiers are different, rendering the obstacles at the two consecutive moments with the same 3D graphics and differentiated rendering effects in the vehicle driving map, and the differentiated rendering effects are used to visually distinguish the same 3D graphics.

5. The method according to claim 1, wherein, rendering the corresponding obstacle in the vehicle driving map that is real-time loaded on the display interface of the in-vehicle device of the vehicle according to the three-dimensional position information and the obstacle type further includes: determining whether there is a third target primitive among the primitives to be rendered, where part of the image is within the display range of the display interface and part of the image is outside the display range of the display interface according to the coverage position and the coverage area; when there is the third target primitive among the primitives to be rendered, rendering the primitive area within the display range of the display interface of the third target primitive in the vehicle driving map; or, when there is the third target primitive among the primitives to be rendered, proportionally reducing the coverage area of the third target primitive until the entire area of the third target primitive can be within the display range of the display interface; and rendering the reduced third target primitive in the vehicle driving map.

6. The method according to claim 1, wherein, determining the three-dimensional position information of the obstacle according to the image data, the parameters of the imaging device, and the real-time position information of the vehicle includes: determining the relative position information of the obstacle in the world coordinate system where the vehicle is located according to the image data and the parameters of the imaging device; and determining the three-dimensional position information of the obstacle according to the real-time position information of the vehicle and the relative position information.

7. The method according to claim 6, wherein, the imaging device is a binocular imaging device; determining the relative position information of the obstacle in the world coordinate system where the vehicle is located according to the image data and the parameters of the imaging device includes: determining the coordinates of the matching image points of the two cameras of the binocular imaging device for the same obstacle; Determine the projection matrices corresponding to the two cameras according to the calibrated parameters of the two cameras; Constructing a conversion equation between a world coordinate system and an image plane coordinate system according to the projection matrix and the coordinates of the matching image points; The transformation equations between the two cameras are solved by least squares to obtain the three-dimensional coordinates of the obstacle in the world coordinate system, where the three-dimensional coordinates are the relative position information.

8. The method according to claim 7, It is characterized in that The parameters of the camera device include: the focal length value and center distance of the two cameras of the binocular camera device after calibration and stereo calibration, the parallax value between the two cameras, and the offset parameters between the coordinate systems of the left image plane and the right image plane corresponding to the two cameras and the origin in the world coordinate system after calibration and stereo calibration.

9. The method according to claim 1, It is characterized in that The performing obstacle detection on the image data to obtain a detection result of whether an obstacle exists and a corresponding obstacle type when an obstacle exists includes: Performing image segmentation on the image data based on the grayscale threshold of the pixel points to obtain one or more candidate pixel regions; Performing dilation processing on the candidate pixel area; Performing region segmentation based on the boundaries of the candidate pixel region after dilation processing, calculating the area of ​​each region obtained by segmentation, and eliminating the noise point region whose area does not meet the preset threshold range to obtain the target pixel region; The target pixel area is input into a pre-trained obstacle detection model, and a detection result of whether there is an obstacle in the target pixel area and the corresponding obstacle type when an obstacle exists is output.

10. The method according to claim 1, It is characterized in that Obstacle detection is performed on the image data to obtain a detection result of whether an obstacle exists and a corresponding obstacle type when an obstacle exists, including: Performing image segmentation on the image data based on the grayscale threshold of the pixel points to obtain one or more candidate pixel regions; Eliminate interference pixels in the candidate pixel area according to at least one of a difference in the initial color features corresponding to the candidate pixel area and whether the candidate pixel area covers a pixel area outside the road, and obtain a target pixel area after eliminating the interference pixels; The target pixel area is input into a pre-trained obstacle detection model, and a detection result of whether there is an obstacle in the target pixel area and the corresponding obstacle type when an obstacle exists is output.

11. The method according to claim 9 or 10, It is characterized in that The obstacle detection model is trained in the following way: Acquire road condition calibration data, the road condition calibration data including: a road condition target area obtained after object recognition of a road condition scene image; and a real result of an obstacle type corresponding to the road condition target area; Using the real result as a training label, train the parameters of the obstacle detection model with parameters to be trained. The input of the obstacle detection model with parameters to be trained is the road condition target area, and the output of the obstacle detection model with parameters to be trained is: whether there is an obstacle in the road condition target area and the prediction result of the corresponding obstacle prediction type when it is predicted that there is an obstacle; Wherein, when the training reaches a set number of times or when the loss function representing the gap between the training label and the prediction result is lower than a set value, the training is regarded as ended.

12. A vehicle-mounted device, Characterized in that, Comprising: A data acquisition module, configured to acquire image data of the surrounding environment collected in real time by a camera device of the vehicle; An obstacle detection module, configured to perform obstacle detection on the image data at each moment to obtain a detection result of whether there is an obstacle and the corresponding obstacle type when there is an obstacle; A position calculation module, configured to, when the detection result indicates the existence of an obstacle, determine the three-dimensional position information of the obstacle according to the image data, the parameters of the camera device, and the real-time position information of the vehicle; And A display module, configured to render corresponding obstacles in a vehicle driving map loaded in real time on a display interface of the vehicle-mounted device of the vehicle according to the three-dimensional position information and the obstacle type, including: calculating a coverage position and a coverage area of a primitive to be rendered corresponding to the obstacle in the vehicle driving map according to the three-dimensional position information and the obstacle type of the obstacle; determining whether there are a first target primitive located within a display range of the display interface and a second target primitive located outside the display range of the display interface in the primitive to be rendered according to the coverage position and the coverage area; when there is the first target primitive in the primitive to be rendered, rendering the first target primitive in 3D on the display interface; when there is the second target primitive in the primitive to be rendered, storing the second target primitive in a rendering list, and refreshing a storage state of the second target primitive in the rendering list according to a real-time display range of the display interface, and when the second target primitive is located within the real-time display range, deleting the second target primitive from the rendering list and outputting it to the display interface for rendering.

13. A vehicle, Characterized in that, It is used to execute the method according to any one of claims 1-11 or includes the vehicle-mounted device according to claim 12.

Images