Detection method and device of vanishing point, vehicle controller and vehicle

Abstract

The invention relates to the technical field of vanishing point detection, and discloses a vanishing point detection method and device, a vehicle controller and a vehicle, and the vanishing point detection method comprises the steps: obtaining a to-be-detected road image; inputting the road image into a feature extraction model for feature extraction to obtain a feature image; the feature image is input into a vanishing point detection model, feature vectors of multiple channels are obtained, the feature vectors of the multiple channels comprise feature vectors of missing channels and feature vectors of multiple quadrant channels, and the feature vectors of the missing channels are used for representing whether vanishing points exist in the feature image or not; the feature vector of the quadrant channel is used for representing the feature of the feature image in the preset area; and when the feature image has a vanishing point, determining a target vanishing point based on the feature vectors of the plurality of quadrant channels. Through the vanishing point detection model outputting the feature vectors of the multiple channels, the vanishing point detection precision can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vanishing point detection, and in particular to a vanishing point detection method, device, vehicle controller and vehicle. Background Art

[0002] In autonomous driving applications, determining the vanishing point can enable real-time dynamic calibration of the camera's external parameters, which helps improve the accuracy of coordinate transformation based on the camera's internal and external parameters, thereby improving vehicle driving safety.

[0003] Currently, deep learning models can be used to automatically learn features of large amounts of image data to determine the vanishing point. However, since the vanishing point accounts for a very small proportion of the entire image, the features are difficult to learn, resulting in low accuracy of the vanishing point finally determined. Summary of the invention

[0004] In view of this, the present invention provides a vanishing point detection method, device, vehicle controller and vehicle to improve the problem of low vanishing point detection accuracy.

[0005] In a first aspect, the present invention provides a vanishing point detection method, the method comprising: acquiring a road image to be detected; inputting the road image into a feature extraction model for feature extraction to obtain a feature image; inputting the feature image into a vanishing point detection model to obtain feature vectors of multiple channels, wherein the feature vectors of the multiple channels include feature vectors of missing channels and feature vectors of multiple quadrant channels, the feature vectors of the missing channels are used to characterize whether there is a vanishing point in the feature image, and the feature vectors of the quadrant channels are used to characterize the features of the feature image in a preset area; when there is a vanishing point in the feature image, determining the target vanishing point based on the feature vectors of the multiple quadrant channels.

[0006] The vanishing point detection method provided in this embodiment, after acquiring a road image, inputs the road image into a feature extraction module for feature extraction to obtain a feature image, and then inputs the feature image into a vanishing point detection model to obtain feature vectors of multiple channels, and when there is a vanishing point in the feature image, the target vanishing point is determined based on the feature vectors of multiple quadrant channels. In this embodiment, through the vanishing point detection model, the vanishing point can be determined using feature vectors corresponding to multiple quadrant areas covering the global scene, which improves the situation where the vanishing point accounts for a small proportion of the image and the accuracy of the detection model is low, thereby improving the detection accuracy of the vanishing point.

[0007] In an optional implementation, the number of quadrant channels is four, and the four quadrant channels correspond one-to-one to the four quadrant areas of the feature image; determining the target vanishing point based on the feature vectors of multiple quadrant channels includes: determining the target vanishing point based on the intersection of the feature vectors of the four quadrant channels.

[0008] In this embodiment, the number of quadrant channels is limited to four, which can reduce the optimization difficulty and time consumption of the vanishing point detection model and improve the detection efficiency.

[0009] In an optional implementation, the vanishing point detection model is a first deep learning model, the first deep learning model is composed of multiple convolutional layers, the input of the first deep learning model is a feature image, and the output of the first deep learning model is a feature vector of multiple channels.

[0010] In an optional implementation, the loss function of the vanishing point detection model is a mean absolute error function.

[0011] In an optional embodiment, before determining the target vanishing point based on the feature vectors of multiple quadrant channels, the method also includes: inputting the feature image into a lane line detection model to obtain confidences of multiple feature points, offsets of multiple feature points, and heights of multiple feature points; determining two target lane lines based on the confidences of the multiple feature points, the offsets of the multiple feature points, and the heights of the multiple feature points; determining the target vanishing point based on the feature vectors of multiple quadrant channels, including: determining the intersection of the feature vectors of the multiple quadrant channels as the initial vanishing point; optimizing the vanishing point detection model and / or the lane line detection model with the goal that the distance between the initial vanishing point and the intersection of the two target lane lines is less than a preset distance to obtain the target vanishing point.

[0012] The vanishing point detection method provided in this embodiment, after determining the feature image, inputs the feature image into the vanishing point detection model to obtain the feature vectors of multiple channels, and inputs the feature image into the lane line detection model to obtain the confidence of multiple feature points, the offset of multiple feature points and the height of multiple feature points, and determines two target lane lines based on the confidence of multiple feature points, the offset of multiple feature points and the height of multiple feature points, and then determines the target vanishing point based on the intersection of the feature vectors of multiple quadrant channels and the two target lane lines. In this embodiment, the vanishing point is optimized by using the geometric structure relationship between the vanishing point and the lane line, which can further improve the detection accuracy of the vanishing point.

[0013] In an optional implementation, the confidence loss function in the lane line detection model is a binary cross entropy function, and the offset and height loss functions in the lane line detection model are mean square error functions.

[0014] In an optional embodiment, before determining the target vanishing point based on the feature vectors of multiple quadrant channels, the method also includes: determining the maximum boundary of the feature vectors of multiple quadrant channels based on the Hough line detection method; and determining the intersection of the feature vectors of multiple quadrant channels based on the maximum boundary of the feature vectors of multiple quadrant channels.

[0015] In an optional implementation, before acquiring the road image to be detected, the method further includes: acquiring a front view image taken by a vehicle-mounted camera; and performing a homography transformation on the front view image to obtain a road image.

[0016] In this implementation, after acquiring the front view image taken by the vehicle-mounted camera, the front view image is subjected to a homography transformation to obtain a road image. Through the homography transformation, the internal and external parameters of cameras on different vehicles can be unified, which can improve the generalization of the vanishing point detection model and the lane line detection model.

[0017] In an optional implementation, the feature extraction model is a residual network.

[0018] In a second aspect, the present invention provides a vanishing point detection device, comprising: a first acquisition module, used to acquire a road image to be detected; a first processing module, used to input the road image into a feature extraction model for feature extraction to obtain a feature image; a second processing module, used to input the feature image into a vanishing point detection model to obtain feature vectors of multiple channels, wherein the feature vectors of the multiple channels include feature vectors of missing channels and feature vectors of multiple quadrant channels, the feature vectors of the missing channels are used to characterize whether there is a vanishing point in the feature image, and the feature vectors of the quadrant channels are used to characterize the features of the feature image in a preset area; a first determination module, used to determine the target vanishing point based on the feature vectors of the multiple quadrant channels when there is a vanishing point in the feature image.

[0019] In an optional implementation, the number of quadrant channels is four, and the four quadrant channels correspond one-to-one to the four quadrant areas of the feature image; the first determination module includes: a first determination unit, used to determine the target vanishing point based on the intersection of the feature vectors of the four quadrant channels.

[0020] In an optional implementation, the vanishing point detection model is a first deep learning model, the first deep learning model is composed of multiple convolutional layers, the input of the first deep learning model is a feature image, and the output of the first deep learning model is a feature vector of multiple channels.

[0021] In an optional implementation, the loss function of the vanishing point detection model is a mean absolute error function.

[0022] In an optional embodiment, the device also includes: a third processing module, used to input the feature image into the lane line detection model to obtain the confidence of multiple feature points, the offset of multiple feature points and the height of multiple feature points; a second determination module, used to determine two target lane lines based on the confidence of multiple feature points, the offset of multiple feature points and the height of multiple feature points; the first determination module includes: a second determination unit, used to determine the intersection of feature vectors of multiple quadrant channels as the initial vanishing point; an optimization unit, used to optimize the vanishing point detection model and / or the lane line detection model with the distance between the initial vanishing point and the intersection of the two target lane lines being less than a preset distance, to obtain the target vanishing point.

[0023] In an optional implementation, the confidence loss function in the lane line detection model is a binary cross entropy function, and the offset and height loss functions in the lane line detection model are mean square error functions.

[0024] In an optional embodiment, the device also includes: a third determination module, which is used to determine the maximum boundary of the feature vectors of multiple quadrant channels based on the Hough line detection method; and a fourth determination module, which is used to determine the intersection of the feature vectors of multiple quadrant channels based on the maximum boundary of the feature vectors of multiple quadrant channels.

[0025] In an optional implementation, the device further includes: a second acquisition module, used to acquire a front view image taken by a vehicle-mounted camera; and a fourth processing module, used to perform a homography transformation on the front view image to obtain a road image.

[0026] In an optional implementation, the feature extraction model is a residual network.

[0027] In a third aspect, the present invention provides a vehicle controller comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0028] In a fourth aspect, the present invention provides a vehicle, the vehicle includes the device for detecting a vanishing point according to the second aspect or any corresponding embodiment thereof, or the vehicle includes a vehicle controller for detecting a vanishing point according to the third aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 is a schematic flow chart of a method for detecting vanishing points according to an embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of a residual network according to an embodiment of the present invention;

[0032] Figure 3 is a schematic diagram of four quadrant areas according to an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of feature vectors of four quadrant channels according to an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of the position of a vanishing point according to an embodiment of the present invention;

[0035] Figure 6 is a schematic diagram of another position of a vanishing point according to an embodiment of the present invention;

[0036] Figure 7 is a flow chart of another method for detecting vanishing points according to an embodiment of the present invention;

[0037] Figure 8 is a schematic diagram of a position of a target lane line according to an embodiment of the present invention;

[0038] Fig. 9 is a schematic diagram of another position of a target lane line according to an embodiment of the present invention;

[0039] Fig.10 is a schematic diagram of a position of another target lane line according to an embodiment of the present invention;

[0040] Fig.11 is a schematic diagram of an optimization process of an initial vanishing point according to an embodiment of the present invention;

[0041] Fig.12 is a flow chart of another method for detecting vanishing points according to an embodiment of the present invention;

[0042] Fig.13 is a schematic diagram of a specific process of a method for detecting a vanishing point according to an embodiment of the present invention;

[0043] Fig.144 is a structural block diagram of a vanishing point detection device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0045] To facilitate understanding of the present solution, prior to describing the embodiments of the present invention, the relevant technologies in the embodiments of the present application are described.

[0046] In autonomous driving applications, coordinate transformation is used in scenarios such as processing sensor data, realizing vehicle positioning, path planning, and obstacle detection. Coordinate transformation is calculated based on the camera's internal parameters (such as focal length and principal point position, etc.) and external parameters (such as camera position and attitude, etc.). However, due to the impact of factors such as bumps during driving, the accuracy of external parameters obtained by offline calibration of the camera before the vehicle leaves the factory is low. Therefore, when the vehicle is driving, it is necessary to perform real-time dynamic calibration of the camera's external parameters based on the vanishing point to ensure the accuracy of coordinate transformation.

[0047] Among them, the vanishing point is the intersection point in two-dimensional space of parallel lines in three-dimensional space after perspective transformation. An image scene may have multiple vanishing points. The vanishing point that is associated with the main geometric structure in the image scene and can convey a strong sense of three-dimensional space or depth information to the observer is called the main vanishing point.

[0048] There are many types of traditional vanishing point detection algorithms, such as edge-based detection methods, region-based detection methods, and texture-based detection methods. The edge-based detection method uses line segment detectors or Hough detection methods to detect relevant edges, and then uses voting or clustering methods to find the final vanishing point. However, the voting method is very time-consuming and difficult to meet the real-time requirements of autonomous driving. The region-based detection method solves the road boundary based on the different color difference textures of the images on both sides of the road boundary, and finally obtains the vanishing point; the texture-based detection method extracts a large amount of texture from the image, and then uses cumulative pixel voting to detect the vanishing point. However, the edge-based or texture-based methods rely heavily on the quality of the image. For images with uneven lighting or poor quality, the extracted texture quality is low, which affects the detection of the vanishing point.

[0049] Compared with traditional vanishing point detection algorithms, key point detection methods based on deep learning can automatically learn features for large amounts of image data, and detect vanishing points by regressing the coordinates of vanishing points on the image, which is more robust. However, the road structure and traffic conditions are complex and changeable during vehicle driving, and the vanishing points account for a very small proportion of the image, making it difficult to learn features, and the accuracy of the deep learning detection model is low.

[0050] The present invention provides a method for detecting vanishing points. By outputting a vanishing point detection model with feature vectors of multiple channels, multiple quadrant areas covering the global scene can be used to detect vanishing points, thereby improving the situation where the vanishing points account for a small proportion in the image and the accuracy of the detection model is low.

[0051] The vanishing point detection method provided by the present invention can be applied to autonomous driving scenarios. As a basic task of autonomous driving, it can provide a basis for upstream tasks such as vehicle positioning, planning, and control. Improving the detection accuracy of vanishing points is of great significance to autonomous driving.

[0052] According to an embodiment of the present invention, a method for detecting a vanishing point is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a vehicle controller, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0053] In this embodiment, a method for detecting a vanishing point is provided, which can be used in a vehicle controller. Figure 1 FIG. 1 is a flow chart of a method for detecting a vanishing point according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0054] Step S101, obtaining a road image to be detected.

[0055] Exemplarily, the road image may be a front view or a bird's-eye view captured by a camera or other device during driving, or may be a preprocessed front view or a bird's-eye view, wherein the preprocessing includes at least one of cropping, scaling, and pixel value normalization.

[0056] In this embodiment, by cropping and scaling the front view or the bird's-eye view, the image can be suitable for a preset feature extraction model, and by normalizing the pixel values ​​of the front view or the bird's-eye view, the pixels in the image can be limited to a preset range, thereby facilitating subsequent feature extraction and network regression processing of the image.

[0057] Step S102: input the road image into a feature extraction model to extract features and obtain a feature image.

[0058] Specifically, there may be interference factors such as noise or lighting changes in road images. Inputting road images into feature extraction models for feature extraction can highlight relevant features in road images that are useful for vanishing point detection and lane line detection, suppress irrelevant information, reduce the computational complexity of subsequent vanishing point detection, and improve the accuracy of vanishing point detection. For example, if the road image is an image with uneven lighting, the feature extraction model can weaken the influence of lighting and focus more on the geometric structure in the road image.

[0059] Exemplarily, the feature extraction model can be a backbone network of a convolutional neural network (CNN), including but not limited to a residual network (such as ResNet34) and the like.

[0060] Specifically, Figure 2 As shown in the figure, the residual network can include multiple convolutional layers. The residual network transmits residual information through cross-layer connections, which can effectively alleviate problems such as overfitting, gradient disappearance or gradient explosion caused by too deep a model, and improve the accuracy of feature extraction.

[0061] Step S103: input the feature image into a vanishing point detection model to obtain feature vectors of multiple channels.

[0062] The feature vectors of the multiple channels include feature vectors of missing channels and feature vectors of multiple quadrant channels. The feature vectors of missing channels are used to characterize whether there are vanishing points in the feature image, and the feature vectors of quadrant channels are used to characterize the features of preset areas of the feature image.

[0063] Exemplarily, the vanishing point detection model may be an artificial intelligence model whose input is a feature image and whose output is a feature vector of multiple channels. Figure 3 As shown, the number of the multiple quadrant channels is four, and the four quadrant channels correspond to the four quadrant regions of the feature image one by one. The four quadrant regions can be the upper left quadrant region, the upper right quadrant region, the lower left quadrant region, and the lower right quadrant region. The vanishing point detection model learns the structural information of the four quadrant regions to segment the road image to determine the vanishing points in the feature image. Figure 3 Four quadrant channels are taken as an example, but the present invention is not limited thereto. For example, the number of quadrant channels may be 8 or 16.

[0064] Specifically, if there is a vanishing point in the feature image, each pixel on the feature image is assigned to one of the quadrants; if it is difficult to identify the vanishing point in the feature image (such as a crossroad), the pixel on the feature image is assigned to the missing channel. For example, the feature vectors of the four quadrant channels can be as follows Figure 4 shown.

[0065] Step S104: when there is a vanishing point in the feature image, determine a target vanishing point based on feature vectors of multiple quadrant channels.

[0066] The target vanishing point is the vanishing point corresponding to the target vehicle, the target vehicle is the vehicle corresponding to the acquired road image, the vanishing point is the point where the lane lines on both sides of the target vehicle and the tangent lines of the lane lines intersect on the ground plane, and the vanishing point is around the farthest point of the visible lane, which can be used to provide the global geometric context of the scene and help infer the position of the lane lines in the road image. For example, the position of the vanishing point can be Figure 5 The position of point A in the middle and the position of the vanishing point can also be Figure 6 The position of point B in the middle.

[0067] Specifically, after determining the feature vectors of multiple channels, it is possible to determine whether the feature image has a vanishing point based on the feature vector of the missing channel. If the feature image has a vanishing point, the target vanishing point can be determined based on the intersection of the feature vectors of multiple quadrant channels.

[0068] Exemplarily, the number of quadrant channels may be four, and the intersection of the feature vectors of the four quadrant channels may be determined as the target vanishing point.

[0069] The vanishing point detection method provided in this embodiment, after acquiring a road image, inputs the road image into a feature extraction module for feature extraction to obtain a feature image, and then inputs the feature image into a vanishing point detection model to obtain feature vectors of multiple channels, and when there is a vanishing point in the feature image, the target vanishing point is determined based on the feature vectors of multiple quadrant channels. In this embodiment, through the vanishing point detection model, the vanishing point can be determined using feature vectors corresponding to multiple quadrant areas covering the global scene, which improves the situation where the vanishing point accounts for a small proportion of the image and the accuracy of the detection model is low, thereby improving the detection accuracy of the vanishing point.

[0070] In this embodiment, a method for detecting a vanishing point is provided, which can be used in a vehicle controller. Figure 7 FIG. 1 is a flow chart of another method for detecting vanishing points according to an embodiment of the present invention. Figure 7 As shown, the method comprises the following steps:

[0071] Step S701, obtaining a road image to be detected.

[0072] For details, please see Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0073] Step S702: Input the road image into a feature extraction model to extract features and obtain a feature image.

[0074] For details, please see Figure 1Step S102 of the illustrated embodiment will not be described in detail here.

[0075] Step S703: input the feature image into a vanishing point detection model to obtain feature vectors of multiple channels.

[0076] Exemplarily, the vanishing point detection model may be a first deep learning model, the first deep learning model is composed of multiple convolutional layers, the input of the first deep learning model is a feature image, and the output of the first deep learning model is a feature vector of multiple channels. Specifically, if the number of quadrant channels is 4, the output of the first deep learning model is a feature vector of five channels.

[0077] Among them, the loss function of the vanishing point detection model is the mean absolute error function.

[0078] Exemplarily, the loss function of the vanishing point detection model can be shown as formula (1):

[0079]

[0080] in, represents the loss of the vanishing point detection model, y i represents the predicted vanishing point corresponding to the i-th feature image, Represents the annotated vanishing point corresponding to the i-th feature image.

[0081] Step S704: input the feature image into the lane detection model to obtain the confidences of the multiple feature points, the offsets of the multiple feature points, and the heights of the multiple feature points.

[0082] Specifically, the lane line detection model can be understood as an artificial intelligence model, which can be any appropriate neural network (NN) model that can be used to predict the confidence, offset and height of the feature image. The lane line detection model can be a LeNet model, an AlexNet model, a GoogleNet model, a ResNets model, a VGG model, a GAN network, an R-CNN network and a recurrent neural network (RNN), etc.

[0083] A feature point can be understood as a point on a lane line. The offset of a feature point represents the offset of the feature point relative to a preset point to further refine the location information of the feature point. Multiple feature points are located at different heights. The height refers to the distance from the feature point to the ground. The longitudinal distances of multiple feature points may be different.

[0084] The confidence represents the probability that the corresponding feature point is a point on the lane line. For example, the confidence is represented by 0-1. The larger the confidence, the greater the probability that the feature point is a point on the lane line.

[0085] Exemplarily, the confidence loss function in the lane line detection model is a binary cross entropy function, and the offset and height loss functions in the lane line detection model are mean square error functions.

[0086] Specifically, the confidence loss function can be shown as formula (2):

[0087]

[0088] in, represents the loss of confidence, s1 represents the number of grids divided in the width direction of the feature image, and s2 represents the number of grids divided in the height direction of the feature image. represents the probability of whether there is a vanishing point in the i-th feature image predicted by the vanishing point detection model, p i Represents the annotation label of the vanishing point in the i-th feature image.

[0089] The loss function of the offset can be shown as formula (3):

[0090]

[0091] in, Indicates the loss of offset. If there is a vanishing point in the i-th feature image, 1 obj =1; if the i-th feature image does not have a vanishing point, 1 obj =0; represents the offset of the i-th feature image predicted by the lane detection model, Δy i Represents the offset of the i-th feature image annotation.

[0092] The height loss function can be shown as formula (4):

[0093]

[0094] in, Represents the loss of height, h i represents the height of the i-th feature image annotation, Represents the height of the vanishing point of the i-th feature image predicted by the lane detection model.

[0095] Step S705 , determining two target lane lines based on the confidences of the multiple feature points, the offsets of the multiple feature points, and the heights of the multiple feature points.

[0096] The two target lane lines are lane lines located on both sides of the target vehicle. For example, the target lane lines can be Figure 8 The line segments L1 and L2 shown in the figure, the target lane line can be Fig. 9 Line segments L3 and L4 are shown.

[0097] Specifically, in order to improve the generalization and accuracy of lane line detection, lane lines are predicted based on 3D feature points, such as Fig.10 As shown, the feature image in the three-dimensional space is divided into a certain number of grids, and the boundary points of the grids are used as feature points. After determining the confidence, offset and height of the feature points, the feature points with a confidence greater than a preset confidence can be determined as points on the lane line, and the lane line is determined based on the offset of the feature points with a confidence greater than the preset confidence and the position information of the highly refined feature points. Fig.10 The line L5 in the figure represents the lane line.

[0098] Step S706: when there is a vanishing point in the feature image, determine a target vanishing point based on feature vectors of multiple quadrant channels.

[0099] Specifically, the above step S706 includes:

[0100] Step S7061: determine the intersection of the feature vectors of multiple quadrant channels as the initial vanishing point.

[0101] Exemplarily, taking the number of quadrant channels as four as an example, after determining the feature vectors of the four quadrant channels, the boundaries of the four quadrant channels can be determined based on the Hough detection algorithm, and the intersection of the boundaries can be determined as the initial vanishing point.

[0102] Step S7062, with the goal of making the distance between the initial vanishing point and the intersection of the two target lane lines less than a preset distance, the vanishing point detection model and / or the lane line detection model is optimized to obtain a target vanishing point.

[0103] The preset distance may be set by a designer based on experience. For example, the preset distance may be 0 mm, 0.2 mm, or 0.5 mm.

[0104] Exemplarily, a line-point fusion module is pre-configured in the vehicle controller. The line-point fusion module can understand the artificial intelligence model and is used to optimize the position of the vanishing point by utilizing the geometric relationship between the lane line and the vanishing point in two-dimensional space (that is, the vanishing point can be the intersection of two target lane lines), thereby improving the detection accuracy of the vanishing point and the lane line.

[0105] Specifically, Fig.11 As shown, the initial vanishing point and the two target lane lines are input into the line point fusion module. The line point fusion module projects the features of the lane line branches into the two-dimensional space, and then performs curve fitting on the key points of the two-dimensional lane lines. Finally, the two target lane lines and the vanishing point are restricted in the same space to minimize the distance between the intersection of the two target lane lines and the vanishing point (that is, the distance is less than the preset distance).

[0106] Among them, the loss function of the line-point fusion module can be a mean square error function.

[0107] For example, the loss function of the line-point fusion module can be shown as formula (5):

[0108]

[0109] in, represents the loss of the line-point fusion module, y vp represents the coordinates of the initial vanishing point, y cp Represents the coordinates of the intersection of two target lane lines.

[0110] The vanishing point detection method provided in this embodiment, after determining the feature image, inputs the feature image into the vanishing point detection model to obtain the feature vectors of multiple channels, and inputs the feature image into the lane line detection model to obtain the confidence of multiple feature points, the offset of multiple feature points and the height of multiple feature points, and determines two target lane lines based on the confidence of multiple feature points, the offset of multiple feature points and the height of multiple feature points, and then determines the target vanishing point based on the intersection of the feature vectors of multiple quadrant channels and the two target lane lines. In this embodiment, the vanishing point is optimized by using the geometric structure relationship between the vanishing point and the lane line, which can further improve the detection accuracy of the vanishing point.

[0111] In this embodiment, a method for detecting a vanishing point is provided, which can be used in a vehicle controller. Fig.12 FIG. 4 is a flow chart of another method for detecting vanishing points according to an embodiment of the present invention. Fig.12 As shown, the method comprises the following steps:

[0112] Step S1201, obtaining a front view image taken by a vehicle-mounted camera.

[0113] Specifically, a camera is provided on the target vehicle, and a front view image of the target vehicle in a driving scene can be acquired through the camera.

[0114] Step S1202: Perform homography transformation on the front view image to obtain a road image.

[0115] Specifically, homography transformation is a geometric transformation commonly used in computer vision and image processing. Homography transformation can be understood as a mapping relationship. In this embodiment, homography transformation is used to convert an image under the coordinates of the vehicle-mounted camera (front view image) into an image under the coordinates of the virtual camera (road image).

[0116] In this embodiment, through homography transformation, the front view images taken by cameras at different positions or angles can be converted into images with specific viewing angles or geometric relationships, and the internal and external parameters of cameras on different vehicles can be unified, thereby ensuring the consistency of the spatial relationship of the on-board cameras, reducing the difference in data distribution, and effectively improving the learning efficiency of the vanishing point detection model, thereby ensuring the robustness of the vanishing point detection model.

[0117] Step S1203: input the road image into a feature extraction model to extract features and obtain a feature image.

[0118] For details, please see Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0119] Step S1204: input the feature image into a vanishing point detection model to obtain feature vectors of multiple channels.

[0120] For details, please see Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0121] Step S1205 , based on the Hough line detection method, determine the maximum boundary of the feature vectors of multiple quadrant channels.

[0122] Specifically, Hough Line Detection is an algorithm used to detect straight lines in images in image processing and computer vision. The basic idea of ​​Hough Line Detection is to transform the straight line representation in the image space into the parameter space, thereby transforming the straight line detection problem into the problem of finding peaks in the parameter space, and can accurately detect straight line information from complex images.

[0123] Step S1206, determining the intersection of the feature vectors of the multiple quadrant channels based on the maximum boundaries of the feature vectors of the multiple quadrant channels.

[0124] Step S1207: when there is a vanishing point in the feature image, determine a target vanishing point based on the feature vectors of multiple quadrant channels.

[0125] For details, please see Figure 7 Step S706 of the illustrated embodiment will not be described in detail here.

[0126] In this implementation, after acquiring the front view image taken by the vehicle-mounted camera, the front view image is subjected to a homography transformation to obtain a road image. Through the homography transformation, the internal and external parameters of cameras on different vehicles can be unified, which can improve the generalization of the vanishing point detection model and the lane line detection model.

[0127] The following assumes that the number of quadrant channels is four. Fig.13The vanishing point detection method provided by the present invention is described in detail.

[0128] Specifically, Fig.13 As shown in the figure, during the driving process of the vehicle, the front view image in the vehicle driving scene is first obtained through the on-board camera, and then the front view image is transformed by homography to obtain the front view image under the virtual camera (that is, the road image mentioned above), so as to unify the internal and external parameters of the front view cameras on different vehicles and improve the generalization of the vanishing point detection model.

[0129] After determining the road image, the road image is input into the residual network, and the residual network is used as a feature extraction module to extract and learn the features of the road image to obtain a feature image. After determining the feature image, the feature image is sent to the lane detection model and the vanishing point detection model respectively. The vanishing point detection model divides the feature image into four quadrants: upper left, upper right, lower left, and lower right. By learning the structural information of the four quadrants, the feature image is segmented, and the intersection of the feature vectors of the four quadrants is determined as the initial vanishing point. The lane detection model converts the feature image into a three-dimensional space through a space conversion module, and then predicts the confidence, offset, and height of the 3D key points (also called feature points) of the lane line in the three-dimensional space, and determines the two target lane lines based on the confidence, offset, and height of the feature points. Since there is certain visual information in the geometric structure of the lane line and the vanishing point on the two-dimensional graphic plane, then based on the line-point fusion model, the two-dimensional information of the target lane line is used to constrain the initial vanishing point to optimize the vanishing point position finally predicted by the network.

[0130] In this embodiment, a vanishing point detection device is also provided, which is used to implement the above embodiments and preferred implementations, and will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0131] This embodiment provides a vanishing point detection device, such as Fig.14 As shown, including:

[0132] The first acquisition module 1401 is used to acquire a road image to be detected;

[0133] The first processing module 1402 is used to input the road image into the feature extraction model to extract features and obtain a feature image;

[0134] The second processing module 1403 is used to input the feature image into the vanishing point detection model to obtain feature vectors of multiple channels, wherein the feature vectors of the multiple channels include feature vectors of missing channels and feature vectors of multiple quadrant channels, the feature vectors of missing channels are used to characterize whether there are vanishing points in the feature image, and the feature vectors of quadrant channels are used to characterize features of the feature image in a preset area;

[0135] The first determination module 1404 is configured to determine a target vanishing point based on the feature vectors of multiple quadrant channels when there is a vanishing point in the feature image.

[0136] In some optional implementations, the number of quadrant channels is four, and the four quadrant channels correspond one to one to the four quadrant areas of the feature image; the first determination module 1404 includes:

[0137] The first determining unit is used to determine the target vanishing point based on the intersection of the feature vectors of the four quadrant channels.

[0138] In some optional embodiments, the vanishing point detection model is a first deep learning model, the first deep learning model is composed of multiple convolutional layers, the input of the first deep learning model is a feature image, and the output of the first deep learning model is a feature vector of multiple channels.

[0139] In some optional implementations, the loss function of the vanishing point detection model is a mean absolute error function.

[0140] In some optional embodiments, the device further comprises:

[0141] A third processing module is used to input the feature image into the lane line detection model to obtain the confidence of multiple feature points, the offset of multiple feature points and the height of multiple feature points;

[0142] A second determination module is used to determine two target lane lines based on the confidence levels of the multiple feature points, the offsets of the multiple feature points, and the heights of the multiple feature points;

[0143] The first determining module 1404 includes:

[0144] A second determining unit, configured to determine an intersection point of feature vectors of a plurality of quadrant channels as an initial vanishing point;

[0145] The optimization unit is used to optimize the vanishing point detection model and / or the lane line detection model with the goal of making the distance between the initial vanishing point and the intersection of the two target lane lines less than a preset distance to obtain the target vanishing point.

[0146] In some optional implementations, the confidence loss function in the lane line detection model is a binary cross entropy function, and the offset and height loss functions in the lane line detection model are mean square error functions.

[0147] In some optional embodiments, the device further comprises:

[0148] A third determination module is used to determine the maximum boundary of the feature vectors of multiple quadrant channels based on the Hough line detection method;

[0149] The fourth determination module is used to determine the intersection of the feature vectors of the multiple quadrant channels based on the maximum boundaries of the feature vectors of the multiple quadrant channels.

[0150] In some optional embodiments, the device further comprises:

[0151] The second acquisition module is used to acquire the front view image taken by the vehicle-mounted camera;

[0152] The fourth processing module is used to perform homography transformation on the front view image to obtain a road image.

[0153] In some optional embodiments, the feature extraction model is a residual network.

[0154] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0155] The vanishing point detection device in this embodiment is presented in the form of a functional unit, where the unit refers to an application specific integrated circuit (ASIC), a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0156] An embodiment of the present invention also provides a vehicle controller, which includes: one or more processors, a memory, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in a computer device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories.

[0157] The processor may be a central processing unit, a network processor or a combination thereof. The processor may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0158] The memory stores instructions executable by at least one processor, so that the at least one processor executes the method shown in the above embodiment.

[0159] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0160] The memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid state drive; the memory may also include a combination of the above types of memory.

[0161] The present invention also provides a vehicle, the vehicle includes the vanishing point detection device provided by any of the above embodiments, or the vehicle includes the vehicle controller provided by any of the above embodiments.

[0162] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the present invention.

Claims

1. A method for detecting a vanishing point, characterized in that: The method comprises: Acquire a road image to be detected; Inputting the road image into a feature extraction model to extract features and obtain a feature image; Input the feature image into a vanishing point detection model to obtain feature vectors of multiple channels, wherein the feature vectors of the multiple channels include feature vectors of missing channels and feature vectors of multiple quadrant channels, the feature vectors of the missing channels are used to characterize whether there are vanishing points in the feature image, and the feature vectors of the quadrant channels are used to characterize features of the feature image in a preset area; When a vanishing point exists in the feature image, a target vanishing point is determined based on the feature vectors of the multiple quadrant channels.

2. The method according to claim 1, characterized in that The number of the quadrant channels is four, and the four quadrant channels correspond to the four quadrant areas of the feature image one by one; and determining the target vanishing point based on the feature vectors of the multiple quadrant channels includes: The target vanishing point is determined based on the intersection of the feature vectors of the four quadrant channels.

3. The method according to claim 1, characterized in that The vanishing point detection model is a first deep learning model, which is composed of multiple convolutional layers. The input of the first deep learning model is the feature image, and the output of the first deep learning model is the feature vectors of the multiple channels.

4. The method according to claim 3, characterized in that The loss function of the vanishing point detection model is a mean absolute error function.

5. The method according to any one of claims 1 to 4, characterized in that Before determining the target vanishing point based on the feature vectors of the multiple quadrant channels, the method further includes: Inputting the feature image into a lane line detection model to obtain confidences of multiple feature points, offsets of the multiple feature points, and heights of the multiple feature points; Determining two target lane lines based on the confidences of the multiple feature points, the offsets of the multiple feature points, and the heights of the multiple feature points; The determining of the target vanishing point based on the feature vectors of the multiple quadrant channels comprises: Determine the intersection of the feature vectors of the multiple quadrant channels as an initial vanishing point; With the goal that the distance between the initial vanishing point and the intersection of the two target lane lines is less than a preset distance, the vanishing point detection model and / or the lane line detection model is optimized to obtain the target vanishing point.

6. The method according to claim 5, characterized in that The confidence loss function in the lane line detection model is a binary cross entropy function, and the offset and height loss functions in the lane line detection model are mean square error functions.

7. The method according to any one of claims 1 to 4, characterized in that Before determining the target vanishing point based on the feature vectors of the multiple quadrant channels, the method further includes: Determine the maximum boundary of the feature vectors of the multiple quadrant channels based on the Hough line detection method; Based on maximum boundaries of the feature vectors of the multiple quadrant channels, intersection points of the feature vectors of the multiple quadrant channels are determined.

8. The method according to any one of claims 1 to 4, characterized in that Before acquiring the road image to be detected, the method further includes: Get the front view image taken by the vehicle camera; Performing homography transformation on the front view image to obtain the road image.

9. The method according to any one of claims 1 to 4, characterized in that The feature extraction model is a residual network.

10. A vanishing point detection device, characterized in that: The device comprises: A first acquisition module, used to acquire a road image to be detected; A first processing module, used for inputting the road image into a feature extraction model to extract features and obtain a feature image; A second processing module is used to input the feature image into a vanishing point detection model to obtain feature vectors of multiple channels, wherein the feature vectors of the multiple channels include feature vectors of missing channels and feature vectors of multiple quadrant channels, the feature vectors of the missing channels are used to characterize whether there are vanishing points in the feature image, and the feature vectors of the quadrant channels are used to characterize features of the feature image in a preset area; The first determination module is configured to determine a target vanishing point based on the feature vectors of the multiple quadrant channels when there is a vanishing point in the feature image.

11. A vehicle controller, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 9 by executing the computer instructions.

12. A vehicle, characterized in that: The vehicle includes the vanishing point detection device according to claim 10 , or the vehicle includes the vehicle controller according to claim 11 .

Images