A method for training a correlation network, a method for detecting lane lines, and a system

Abstract

The present application discloses a correlation network training method, a lane line detection method and a system, wherein the correlation network is used to calculate the correlation between vehicle features and lane line features, and the method includes: obtaining vehicle features and lane line features in a sample image; based on lane line features and vehicle features, using feature fusion to respectively calculate the lane-vehicle attribution relationship matrix and the vehicle-line attribution relationship matrix; based on the difference between the lane-vehicle attribution relationship matrix and the vehicle-line attribution relationship matrix, calculating the attribution loss function, and using the attribution loss function to correct the total loss function of the correlation network; based on the sample image, using the corrected total loss function to train the correlation network. The technical solution in the present application helps to improve the accuracy of lane line detection by taking advantage of the correlation between the spatial positions of vehicles and lane lines in an image.

Description

Technical Field

[0001] The present application relates to the technical field of autonomous driving, and more specifically, to a correlation network training method, a lane line detection method, and a lane line detection system. Background Art

[0002] With the continuous development of autonomous driving technology, major manufacturers are paying more and more attention to the perception of vehicle driving road scenes. Not only do they need to perceive obstacles and vehicles in the scene, but they also need to be able to perceive lane lines in order to better plan the path of the vehicle during autonomous driving. Such perception methods usually use target detection networks to output detected lane lines based on images collected by the vehicle.

[0003] For example, a certain network architecture uses a general feature extraction network to obtain the features of the input image, and then sends the features to each head (detection head) to complete their respective detection / prediction tasks. This network architecture can simultaneously complete: fitting lane line detection, lane topology prediction, 2D traffic element detection, and lane line and traffic element attribution relationship prediction.

[0004] Although such existing technologies can perform multi-task target detection, they do not consider the spatial correlation between different detection targets and do not fully utilize the information in the perceived scene. For example, when detecting lane lines, they do not consider the correlation between the vehicle and the lane lines, which helps to improve the accuracy of target detection. Summary of the invention

[0005] The purpose of this application is to train a lane detection model based on the correlation between the spatial positions of vehicles and lane lines in an image to improve the accuracy of lane detection.

[0006] The technical solution of the first aspect of the present application is: to provide a correlation network training method, the correlation network is used to calculate the correlation between vehicle features and lane line features, the method comprising: obtaining a vehicle feature F in a sample image car And the lane feature F lane ; Based on lane feature F lane 、Vehicle characteristics F car , using feature fusion to calculate the line car ownership relationship matrix Mat lane_car , car line belonging relationship matrix Mat car_lane ; Based on the line car ownership matrix Mat lane_car , car line belonging relationship matrix Mat car_lane The attribution loss function is calculated based on the difference between the values ​​of the image and the image, and the attribution loss function is used to correct the total loss function of the association network. Based on the sample image, the association network is trained using the corrected total loss function.

[0007] In any of the above technical solutions, further, based on the vehicle features and the lane line features, a feature fusion method is adopted to calculate the vehicle-line attribute relationship matrix and the lane-vehicle attribute relationship matrix respectively, specifically including: based on the vehicle feature F car , Lane line feature F lane , using the first activation function ReLU, respectively calculate the line-vehicle feature array (Q1, K1, V1) and the vehicle-line feature array (Q2, K2, V2), wherein the first query feature in the line-vehicle feature array is determined by the lane line feature, the first key feature and the first true value feature in the line-vehicle feature array are determined by the vehicle feature, the second query feature in the vehicle-line feature array is determined by the vehicle feature, and the second key feature and the second true value feature in the line-vehicle feature array are determined by the lane line feature; based on the line-vehicle feature array (Q1, K1, V1) and the vehicle-line feature array (Q2, K2, V2), using the second activation function ReLU, adopting the feature fusion method, respectively calculate the lane line feature F after feature fusion lane_car , and the vehicle feature F after feature fusion car_lane ; Lane line feature F based on feature fusion lane_car , and the vehicle feature F after feature fusion car_lane , using the third activation function sigmoid, calculate the car line belonging relationship matrix Mat car_lane , Line car ownership relationship matrix Mat lane_car .

[0008] In any of the above technical solutions, further, the method further includes: adopting an iterative method, based on the lane line feature F after feature fusion lane_car , and the vehicle feature F after feature fusion car_lane , using the first activation function ReLU, recalculate the line-car feature array (Q1, K1, V1), the car-line feature array (Q2, K2, V2), and calculate the corresponding car-line attribution relationship matrix Mat car_lane , Line car ownership relationship matrix Mat lane_car .

[0009] In any of the above technical solutions, further, when it is determined that the sample image does not contain the attribution labels of the vehicle and the lane line, the calculation formula of the attribution loss function is:

[0010]

[0011] In the formula, Loss Mat is the attribution loss function, i=1,2,…n, i is the lane number, j=1,2,…n, j is the vehicle number, Mat lane_car is the line car ownership matrix, Mat car_laneis the car line attribution relationship matrix.

[0012] In any of the above technical solutions, further, when it is determined that the sample image contains attribution labels of the vehicle and the lane line, the calculation formula of the attribution loss function is:

[0013]

[0014] In the formula, Loss Mat is the attribution loss function, i=1,2,…n, i is the lane number, j=1,2,…n, j is the vehicle number, Mat lane_car is the line car ownership matrix, Mat car_lane is the car line attribution matrix, Mat gt [i][j] is the probability that the i-th lane line is related to the j-th vehicle.

[0015] In any of the above technical solutions, further, the method also includes: setting a random perturbation function, and determining whether the value of the random perturbation function belongs to a preset range; if so, the first query feature in the line-vehicle feature array is determined by the lane line feature, the first key feature and the first true value feature in the line-vehicle feature array are determined by the vehicle feature, the second query feature in the vehicle-lane feature array is determined by the vehicle feature, and the second key feature and the second true value feature in the line-vehicle feature array are determined by the lane line feature; if not, the first query feature in the line-vehicle feature array is determined by the vehicle feature, the first key feature and the first true value feature in the line-vehicle feature array are determined by the lane line feature, the second query feature in the vehicle-lane feature array is determined by the lane line feature, and the second key feature and the second true value feature in the line-vehicle feature array are determined by the vehicle feature.

[0016] The technical solution of the second aspect of the present application is: a lane line detection method is provided, the method comprising: obtaining vehicle features and corresponding vehicle coordinates, lane line features and lane line coordinates in the current image; based on the association network, calculating the association matrix between the vehicle features and the lane line features; based on the association matrix, selecting associated vehicle coordinates in the vehicle coordinates; based on the associated vehicle coordinates and the lane line coordinates, using coordinate fitting to generate fitted lane lines.

[0017] In any of the above technical solutions, further, the method also includes: based on vehicle features and lane line features, using the activation function ReLU, calculating the vehicle lane feature array (Q4, K4, V4), wherein the fourth query feature in the vehicle lane feature array is determined by the vehicle features, and the fourth key feature and the fourth true value feature in the lane vehicle feature array are determined by the lane line features; based on the vehicle lane feature array (Q4, K4, V4), using dot product operation, calculating the vehicle correction feature, and inputting the vehicle correction feature into the vehicle detection head to obtain the corrected vehicle coordinates; based on the association matrix, selecting the associated vehicle coordinates from the corrected vehicle coordinates.

[0018] In any of the above technical solutions, further, the method also includes: based on vehicle features and lane line features, using the activation function ReLU, calculating the line-vehicle feature array (Q3, K3, V3), wherein the third query feature in the line-vehicle feature array is determined by the lane line feature, and the third key feature and the third true value feature in the line-vehicle feature array are determined by the vehicle features; based on the line-vehicle feature array (Q3, K3, V3), using dot product operation, calculating the lane line correction feature, and inputting the lane line correction feature into the lane line detection head to obtain the corrected lane line coordinates; based on the associated vehicle coordinates and the corrected lane line coordinates, generating a fitted lane line.

[0019] The technical solution of the third aspect of the present application is: a lane line detection system is provided, which includes: a feature acquisition module, the feature acquisition module is configured to obtain vehicle features and corresponding vehicle coordinates, lane line features and lane line coordinates in the current image; an association matrix calculation module, the association matrix calculation module is configured to calculate the association matrix between vehicle features and lane line features based on the association network; a coordinate extraction module, the coordinate extraction module is configured to select associated vehicle coordinates from vehicle coordinates based on the association matrix; a fitting module, the fitting module is configured to generate a fitted lane line by using coordinate fitting based on the associated vehicle coordinates and lane line coordinates.

[0020] The beneficial effects of this application are:

[0021] The technical solution in the present application first integrates the vehicle features and the lane line features by using two network branches, fully utilizes the correlation between the spatial positions of the vehicle and the lane line, obtains the corresponding attribution relationship matrix, and performs network training based on the difference between the two matrices, thereby obtaining the correlation between the vehicle coordinates and the lane line coordinates, so that the vehicle coordinates can be introduced when fitting the lane line, and the fitting result is used as the lane line detection result, thereby improving the accuracy of lane line detection.

[0022] At the same time, during the lane line fitting process, the present application also uses the vehicle-line feature array and the line-vehicle feature array to detect the vehicle head and the lane line head in the network, so as to obtain the corrected corresponding coordinates to correct the prediction results and improve the accuracy of the fitted lane lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0024] Figure 1 is a schematic flow chart of a method for training an association network according to an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of an overall process according to an embodiment of the present application;

[0026] Figure 3 is a schematic flow chart of a lane line detection method according to another embodiment of the present application;

[0027] Figure 4 is a schematic diagram of an overall process according to another embodiment of the present application;

[0028] Figure 5 is a schematic block diagram of a lane detection system according to yet another embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0030] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.

[0031] Embodiment 1:

[0032] like Figure 1 As shown, this embodiment provides a correlation network training method and a lane line detection method, the method comprising:

[0033] Step 10: Get the vehicle feature F in the sample image car And the lane feature F lane ;

[0034] Specifically, an existing neural network is selected as the basic backbone of the model, and a lane detection head and a vehicle detection head are added to the network so that the vehicle feature F in the sample image can be detected. car And the lane feature F lane .

[0035] It should be noted that the sample images selected in this embodiment are collected by a vehicle-mounted camera, and the vehicle-mounted camera may be a monocular camera, a binocular camera, or a surround-view camera.

[0036] The sample image is input into the above neural network model, the corresponding features are extracted through the feature extraction network, and then the extracted features F are sent to the lane line detection head and the vehicle detection head to obtain the vehicle features F in the sample image car And the lane feature F lane .

[0037] It should be noted that the final prediction results of the above two detection heads are all lane line coordinates and all vehicle coordinates. In this embodiment, the features of the previous layer of the prediction results are retained to participate in the subsequent calculation of the attribution relationship matrix.

[0038] Set the maximum number of lane lines predicted by the model to n and the maximum number of vehicles to m. However, it should be noted that not all scenarios contain n lane lines and m vehicles, so for the part without vehicles / lane lines, we need to ensure that the model's prediction result is empty.

[0039] Set the dimension of the feature to: vehicle feature Lane line features Among them, d1 and d2 represent the number of columns in the corresponding dimension respectively, and the values ​​can be set manually.

[0040] Step 20: Based on lane feature F lane 、Vehicle characteristics F car , using feature fusion to calculate the line car ownership relationship matrix Mat lane_car , car line belonging relationship matrix Mat car_lane ;

[0041] Step 30: Based on the line vehicle ownership matrix Mat lane_car , car line belonging relationship matrix Mat car_lane , calculate the attribution loss function, and use the attribution loss function to correct the total loss function of the association network;

[0042] Step 40: Based on the sample image, the relevance network is trained using the modified total loss function.

[0043] Specifically, the feature fusion method proposed in this embodiment can be used to combine the vehicle feature F car Fusion to lane feature F lane In the above example, the fused features are further calculated to obtain the line-car ownership relationship matrix Mat lane_car ; Similarly, the lane feature F lane Fusion to vehicle feature F car In the above example, we can further obtain the car lane attribution matrix Mat car_lane .

[0044] Theoretically, the ownership relationship matrix obtained by the above two fusion methods should be the same. Therefore, in this embodiment, the line vehicle ownership relationship matrix Mat lane_car Mat car_lane The difference is used as the basis for calculating the loss function to obtain the attribution loss function, and the attribution loss function is used to correct the total loss function of the association network.

[0045] It should be noted that since the vehicle feature F is needed in the association network car And the lane feature F lane Therefore, the network should include the loss functions corresponding to the lane detection results and vehicle detection results. Therefore, the weighted sum of the three loss functions can be used as the corrected total loss function to train the network so that the vehicle coordinates associated with the lane line can be added to the lane fitting process based on the correlation between the spatial positions of the vehicle and the lane line to improve the accuracy of lane detection.

[0046] like Figure 2 As shown, considering that the sample images in the model training process require a large amount of data annotation, and the existing sample database usually does not contain the spatial position relationship between the vehicle and the lane line, this embodiment also shows a feature fusion method based on self-supervised learning, which not only eliminates the need for annotation information, avoids the impact of incorrect annotation information on model training, and reduces the cost of model training; but also uses two independent network branches for training to prevent the network from deviating.

[0047] In the above embodiment, in step 20, based on the vehicle features and lane line features, a feature fusion method is adopted to calculate the vehicle-lane belonging relationship matrix and the lane-vehicle belonging relationship matrix respectively, which specifically includes:

[0048] Step 201: Based on vehicle feature F car , Lane line feature F lane, using the first activation function ReLU, respectively calculate the line-car feature array (Q1, K1, V1) and the vehicle-line feature array (Q2, K2, V2), wherein the first query feature Q1 in the line-car feature array is determined by the lane line feature, the first key feature K1 and the first true value feature V1 in the line-car feature array are determined by the vehicle feature, the second query feature Q2 in the vehicle-line feature array is determined by the vehicle feature, and the second key feature K2 and the second true value feature V2 in the vehicle-line feature array are determined by the lane line feature;

[0049] Specifically, before calculating the feature array, the above vehicle feature F car , Lane line feature F lane Input a fully connected neural network, perform dimension transformation, and obtain the transformed vehicle feature F′ car , Lane feature F′ lane ,in, d3 is the number of columns of the unified feature dimension.

[0050] In this embodiment, the feature fusion module is used to perform feature fusion, and then the correlation between the vehicle and the lane line is predicted. Considering that the features obtained by different fusion methods may be different, and this difference can be used for self-supervised training, this embodiment uses two different methods to fuse features:

[0051] 1. Integrate vehicle features into lane features;

[0052] 2. Integrate lane line features into vehicle features.

[0053] The inputs of these two methods correspond to the above-mentioned line-car feature array (Q1, K1, V1) and car-line feature array (Q2, K2, V2) respectively.

[0054] Taking the lane-vehicle feature array (Q1, K1, V1) as an example, the lane line feature is used as the query feature Q1, and the vehicle features are combined into a key-value pair: (K1, V1). The query feature Q1 is used to query the lane line features associated with each lane line, and an association score is calculated. Then, this score is used as the weight to perform weighted summation on the vehicle feature V1, so that the vehicle features associated with the current lane line can be integrated into the lane line features.

[0055] The calculation formulas corresponding to the line car feature array (Q1, K1, V1) and the car line feature array (Q2, K2, V2) are:

[0056]

[0057] Where W 11 ,…,W 23It is a built-in parameter of the neural network, obtained through training; ReLU() is an activation function commonly used in neural network training, which is used to add nonlinearity between matrix linear transformations and enhance the performance of the model.

[0058] The above calculations yield

[0059] Step 202: Based on the lane feature array (Q1, K1, V1) and the vehicle-lane feature array (Q2, K2, V2), the second activation function ReLU is used to calculate the lane line features F after feature fusion. lane_car , and the vehicle feature F after feature fusion car_lane ;

[0060] Specifically, the above two arrays (Q1, K1, V1) and (Q2, K2, V2) are respectively input into two feature fusion modules Feature fusion1 and Feature fusion2 to calculate the lane line feature F that integrates the vehicle features. lane_car And the vehicle feature F that integrates the lane line feature car_lane , the corresponding calculation formula is:

[0061]

[0062] Where W1, W2, b1, b2 are the built-in parameters of the neural network; QK T Used to calculate the correlation between lane lines and vehicles; the subsequent softmax() is used to increase the weights corresponding to more relevant information and weaken the weights corresponding to irrelevant information.

[0063] Through such calculation, the correlation between each vehicle and the lane line is obtained. According to the calculated correlation, the corresponding lane line / vehicle feature V is weighted and summed to obtain the features that need to be integrated into the vehicle / lane line: softmax(QK T )V; then add the features to be fused to the original vehicle / lane features, and introduce a nonlinear transformation to obtain the fused lane and vehicle features F lane_car 、F car_lane .

[0064] Step 203: Lane line feature F after feature fusion lane_car , and the vehicle feature F after feature fusion car_lane , using the third activation function sigmoid, calculate the car line belonging relationship matrix Mat car_lane , Line car ownership relationship matrix Mat lane_car .

[0065] Specifically, the corresponding calculation formula is:

[0066]

[0067] Among them, Mat lane_car ∈R n×m 、Mat car_lane ∈R m×n , sigmoid() is used to map the result to the range of [0,1], because the relationship we want to calculate is the probability that a certain vehicle has a relationship with a certain lane line; W lane_car、 W car_lane 、b lane_car 、b car_lane Built-in parameters for neural networks; Mat lane_car [i][j] is the probability that the i-th lane line and the j-th vehicle are related in the lane-vehicle relationship matrix, Mat car_lane [j][i] is the probability that the i-th lane line in the lane affiliation relationship matrix is ​​related to the j-th vehicle.

[0068] It should be noted that in the above two attribution relationship matrices, the probability that the i-th lane line is related to the j-th vehicle should be the same. Therefore, the difference between the two can be used as the basis for calculating the loss function.

[0069] In the above embodiment, considering that the correlation between the two features in the result obtained by the above feature fusion method is not high, taking the fusion of vehicle features into lane line features as an example, in the fused features, most of the information is still about the lane line, and only a small part of the information belongs to the vehicle. Therefore, in order to increase the proportion of vehicle features in the fused features, the method further includes: adopting an iterative method, based on the lane line feature F after feature fusion, lane_car , and the vehicle feature F after feature fusion car_lane , using the first activation function ReLU, recalculate the line-car feature array (Q1, K1, V1), the car-line feature array (Q2, K2, V2), and calculate the corresponding car-line attribution relationship matrix Mat car_lane , Line car ownership relationship matrix Mat lane_car .

[0070] Therefore, by adopting an iterative approach, the proportion of vehicle features (lane line features) in the fused features can be increased step by step, so that the similarity between the fused features and the original vehicle features increases, and the corresponding weights also increase accordingly. When a certain number of iterations is reached, the proportion of lane line features and the proportion of vehicle features can gradually tend to be balanced, which is also more conducive to achieving self-supervised training.

[0071] In the above embodiment, when it is determined that the sample image does not contain the attribution labels of the vehicle and the lane line, the calculation formula of the attribution loss function is:

[0072]

[0073] In the formula, Loss Mat is the attribution loss function, i=1,2,…n, i is the lane number, j=1,2,…n, j is the vehicle number, Mat lane_car is the line car ownership matrix, Mat car_lane is the car line attribution relationship matrix.

[0074] In the above embodiment, data annotation can also be performed in the sample database, and the attribution relationship between the vehicle and the lane line can be annotated in the sample image to obtain the corresponding label. Therefore, when it is determined that the sample image contains the attribution label of the vehicle and the lane line, the calculation formula of the attribution loss function is:

[0075]

[0076] In the formula, Loss Mat is the attribution loss function, i=1,2,…n, i is the lane number, j=1,2,…n, j is the vehicle number, Mat lane_car is the line car ownership matrix, Mat car_lane is the car line attribution matrix, Mat gt [i][j] is the probability that the i-th lane line is related to the j-th vehicle.

[0077] After obtaining the above-mentioned attribution loss function, the loss functions corresponding to the lane detection result and the vehicle detection result can be set as Loss car , Loss_lane, therefore, the corrected total loss function is:

[0078] Loss = αLoss car +βLoss lane +γLoss Mat

[0079] In the formula, α, β, and γ are artificially set hyperparameters.

[0080] In the above embodiment, considering that most of the existing 3D vehicle labels are generated by existing models, these labels are relatively inaccurate. Therefore, in order to prevent the model from overfitting the wrong labels and falling into the local optimum, this embodiment also introduces a perturbation function in the process of calculating the line vehicle feature array (Q1, K1, V1) and the vehicle line feature array (Q2, K2, V2). The method also includes:

[0081] Setting a random perturbation function and determining whether the value of the random perturbation function falls within a preset range;

[0082] If yes, the first query feature in the lane feature array is determined by the lane feature, and the first key feature and the first true value feature in the lane feature array are determined by the vehicle feature.

[0083] The second query feature in the vehicle-line feature array is determined by the vehicle feature, and the second key feature and the second true value feature in the lane-line feature array are determined by the lane-line feature;

[0084] If not, the first query feature in the lane feature array is determined by the vehicle feature, and the first key feature and the first true value feature in the lane feature array are determined by the lane feature.

[0085] The second query feature in the vehicle-line feature array is determined by the lane line feature, and the second key feature and the second true value feature in the line-vehicle feature array are determined by the vehicle feature.

[0086] It should be noted that the preset range can be an interval. For example, when the value range of the random perturbation function is [0, 1], the preset range can be set, for example: (0.4, 0.8), (0.5, 1), (0.6, 1].

[0087] Embodiment 2:

[0088] Based on the above embodiments, Figure 3 As shown, this embodiment also shows a lane line detection method, the method comprising:

[0089] Step 100: Obtain vehicle features and corresponding vehicle coordinates, lane line features and lane line coordinates in the current image;

[0090] Specifically, using the above network, target detection can be performed on the currently collected image, and the vehicle features and corresponding vehicle coordinates, lane line features and lane line coordinates contained in the current image can be output, that is, the vehicle features correspond to the vehicle coordinates one by one, and the lane line features correspond to the lane line coordinates one by one.

[0091] The current image is an image captured by the vehicle-mounted camera at the current moment. The vehicle-mounted camera may be a monocular camera, a binocular camera, or a surround-view camera.

[0092] Step 200: Calculate the correlation matrix between vehicle features and lane line features based on the correlation network;

[0093] Specifically, the association network selected in this embodiment is an association network obtained after training using the training method of the above embodiment. The association network can calculate the correlation between the spatial positions of the vehicle and the lane line in the image based on the input image, that is, the attribution relationship matrix in the training process.

[0094] It should be noted that the car line attribution relationship matrix Mat can be used car_lane , Line car ownership relationship matrix Mat lane_car As the final association matrix, any one of them can be used. For example, the line car ownership matrix Mat lane_car As a correlation matrix.

[0095] Step 300: Selecting associated vehicle coordinates from the vehicle coordinates based on the association matrix;

[0096] Specifically, after obtaining the correlation matrix, we can get the probability that the i-th lane line is related to the j-th vehicle. The value of this probability is (0,1). Therefore, we can set a threshold c (such as 0.7). When the correlation matrix Mat lane_car If an element in [i][j] is greater than or equal to the threshold c, we believe that the corresponding vehicle j is associated with the lane line i, then the vehicle coordinates of this vehicle are selected as the associated vehicle coordinates.

[0097] Step 400: Based on the associated vehicle coordinates and lane line coordinates, a fitted lane line is generated using a coordinate fitting method.

[0098] This embodiment makes full use of the correlation between the spatial positions of the vehicle and the lane line in the image, selects the vehicle coordinates associated with the lane line as the associated vehicle coordinates, and then uses the associated vehicle coordinates and the lane line coordinates together for lane line fitting to obtain the final lane line detection result, thereby improving the accuracy of lane line detection.

[0099] It should be noted that the fitted lane line in this embodiment is the lane center line, so the vehicle coordinates and lane line coordinates contained in the current image can be obtained through the neural network and the lane line detection head and vehicle detection head. Then, the selected associated vehicle coordinates and the corresponding lane line coordinates are used to fit the lane line based on the fitting algorithm, and the specific fitting process is not repeated here.

[0100] like Figure 4 As shown, considering that an embodiment shows a feature fusion method based on a self-supervised learning method, based on the vehicle feature F car , Lane line feature F lane, the lane-car feature array (Q1, K1, V1) and the vehicle-lane feature array (Q2, K2, V2) are calculated. Therefore, these two feature arrays can be used to train the lane detection head and the vehicle detection head respectively. The two feature fusion modules Feature rectify1 and Feature rectify2 are used to obtain the corrected lane features and the corrected vehicle features. By calculating the offset of the original prediction results, the training of the two heads is completed to realize the correction of the prediction results. The specific training process is not repeated here.

[0101] Therefore, in the above embodiment, in order to improve the accuracy of fitting lane lines, the vehicle coordinates are corrected, and the method further includes:

[0102] Step 301: Based on the vehicle features and lane features, the activation function ReLU is used to calculate the vehicle-lane feature array (Q4, K4, V4), wherein the fourth query feature Q4 in the vehicle-lane feature array is determined by the vehicle features, and the fourth key feature K4 and the fourth true value feature V4 in the lane-vehicle feature array are determined by the lane features; the corresponding calculation formula is:

[0103]

[0104] In the formula, F′ car is the vehicle feature, F′ lane is the lane line feature. The above two features are transformed in dimension. 21 ,…,W 23 、b 21 ,…,b 23 It is the built-in parameter of the neural network, obtained through training;

[0105] ReLU() is an activation function commonly used in neural network training. It is used to add nonlinearity between matrix linear transformations and enhance the performance of the model.

[0106] Step 302: Based on the vehicle lane feature array (Q4, K4, V4), use the dot product operation to calculate the vehicle correction feature F car_rec , and input the vehicle correction feature into the vehicle detection head to obtain the corrected vehicle coordinates; where the vehicle correction feature F car_rec The corresponding calculation formula is:

[0107] F car_rec =dot(V4,W′4concat(Q4,K4)b′4)+V4

[0108] In the formula, dot() represents the dot product operation, W′4 and b′4 are preset parameters, and the above operation converts the W′concat(Q,K)b′∈R on the right side mEach element in the vector is multiplied by each row of V; concat() represents the concatenation operation, which concatenates the original Q and K into a (nm×2d3) feature; the final calculated dot(V,W′concat(Q,K)b′) is the calculated feature offset, which, together with the original feature V, realizes the correction of the original feature.

[0109] Step 303: Based on the correlation matrix, select the associated vehicle coordinates from the corrected vehicle coordinates.

[0110] In the above embodiment, in order to improve the accuracy of fitting the lane line, the lane line coordinates are corrected, and the method further includes:

[0111] Step 401: Based on the vehicle features and lane line features, the activation function ReLU is used to calculate the lane-vehicle feature array (Q3, K3, V3), wherein the third query feature Q3 in the lane-vehicle feature array is determined by the lane line features, and the third key feature K3 and the third true value feature V3 in the lane-vehicle feature array are determined by the vehicle features; the corresponding calculation formula is:

[0112]

[0113] In the formula, F′ car is the vehicle feature, F′ lane is the lane line feature. The above two features are transformed in dimension. 11 ,…,W 13 、b 11 ,…,b 13 It is a parameter built into the neural network, obtained through training; RELU() is an activation function commonly used in neural network training, which is used to add nonlinearity between matrix linear transformations and enhance the performance of the model.

[0114] Step 402: Based on the lane feature array (Q3, K3, V3), use the dot product operation to calculate the lane correction feature F lane_rec , and input the lane line correction feature into the lane line detection head to obtain the corrected lane line coordinates; where the lane line correction feature F lane_rec The corresponding calculation formula is:

[0115] F lane_rec =dot(V3,W′3concat(Q3,K3)b′3)+V3

[0116] In the formula, dot() represents the dot product operation, W′3 and b′3 are preset parameters, and the above operation converts the W′concat(Q,K)b′∈R on the right side mEach element in the vector is multiplied by each row of V; concat() represents the concatenation operation, which concatenates the original Q and K into a (nm×2d3) feature; the final calculated dot(V,W′concat(Q,K)b′) is the calculated feature offset, which, together with the original feature V, realizes the correction of the original feature.

[0117] Step 403: Generate a fitted lane line based on the associated vehicle coordinates and the corrected lane line coordinates.

[0118] Embodiment three:

[0119] like Figure 5 As shown, this embodiment also shows a lane line detection system 1000, which includes:

[0120] A feature acquisition module 1001 is configured to acquire vehicle features and corresponding vehicle coordinates, lane line features and lane line coordinates in the current image;

[0121] The correlation matrix calculation module 1002 is configured to calculate the correlation matrix between the vehicle features and the lane line features based on the correlation network;

[0122] A coordinate extraction module 1003, the coordinate extraction module 1003 is configured to select associated vehicle coordinates from the vehicle coordinates based on the association matrix;

[0123] The fitting module 1004 is configured to generate a fitted lane line by using a coordinate fitting method based on the associated vehicle coordinates and the lane line coordinates.

[0124] In the above embodiment, the coordinate extraction module 1003 is further configured as follows:

[0125] Based on the vehicle features and lane line features, the activation function ReLU is used to calculate the vehicle line feature array;

[0126] Based on the vehicle line feature array, the vehicle correction features are calculated using the dot product operation, and the vehicle correction features are input into the vehicle detection head to obtain the corrected vehicle coordinates;

[0127] Based on the association matrix, the associated vehicle coordinates are selected from the corrected vehicle coordinates.

[0128] In the above embodiment, the fitting module 1004 is further configured to:

[0129] Based on the vehicle features and lane line features, the activation function ReLU is used to calculate the lane vehicle feature array;

[0130] Based on the lane-car feature array, the lane correction feature is calculated using the dot product operation, and the lane correction feature is input into the lane detection head to obtain the corrected lane coordinates;

[0131] Generate fitted lane lines based on the associated vehicle coordinates and the corrected lane line coordinates.

[0132] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0133] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present application.

[0134] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0135] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. A method for training a relevance network, characterized in that: The association network is used to calculate the association between vehicle features and lane line features. The method includes: Acquire the vehicle features and the lane line features in the sample image; Based on the lane line features and the vehicle features, a feature fusion method is used to calculate the line-vehicle attribution relationship matrix and the vehicle-lane attribution relationship matrix respectively; Based on the difference between the line-vehicle attribution relationship matrix and the vehicle-line attribution relationship matrix, an attribution loss function is calculated, and the total loss function of the association network is corrected using the attribution loss function; Based on the sample image, the relevance network is trained using the modified total loss function; The line-vehicle belonging relationship matrix is ​​a relationship matrix calculated after integrating the vehicle features into the lane line features, and the vehicle-line belonging relationship matrix is ​​a relationship matrix calculated after integrating the lane line features into the vehicle features. The method of calculating the vehicle-lane attribution relationship matrix and the lane-vehicle attribution relationship matrix respectively based on the vehicle features and the lane line features by adopting a feature fusion method specifically includes: Based on the vehicle features and the lane line features, using a first activation function ReLU, respectively calculating a lane-vehicle feature array and a vehicle-lane feature array; Based on the lane-vehicle feature array and the vehicle-lane feature array, a second activation function ReLU is used to adopt a feature fusion method to respectively calculate lane line features after feature fusion and vehicle features after feature fusion; Based on the lane line features after the feature fusion and the vehicle features after the feature fusion, using a third activation function sigmoid, respectively calculating the vehicle-lane belonging relationship matrix and the line-vehicle belonging relationship matrix; Among them, the calculation formula of the modified total loss function is: Loss=αLoss car +βLoss lane +γLoss Mat In the formula, α, β, and γ are artificially set hyperparameters, and Loss car is the loss function corresponding to the vehicle detection result, Loss lane is the loss function corresponding to the lane line detection result, Loss Mat is the attribution loss function.

2. The association network training method according to claim 1, characterized in that: The method further comprises: An iterative method is adopted, based on the lane line features after the feature fusion and the vehicle features after the feature fusion, the first activation function ReLU is used to recalculate the line-vehicle feature array and the vehicle-line feature array, and calculate the corresponding vehicle-line attribution relationship matrix and the line-vehicle attribution relationship matrix.

3. The association network training method according to any one of claims 1 to 2, characterized in that: When it is determined that the sample image does not contain the attribution labels of the vehicle and the lane line, the calculation formula of the attribution loss function is: In the formula, Loss Mat is the attribution loss function, i=1,2,…n, i is the lane number, j=1,2,…n, j is the vehicle number, Mat lane_car is the line car ownership matrix, Mat car_lane is the car line attribution relationship matrix.

4. The association network training method according to any one of claims 1 to 2, characterized in that: When it is determined that the sample image contains the attribution labels of the vehicle and the lane line, the calculation formula of the attribution loss function is: In the formula, Loss Mat is the attribution loss function, i=1,2,…n, i is the lane number, j=1,2,…n, j is the vehicle number, Mat lane_car is the line car ownership matrix, Mat car_lane is the car line attribution matrix, Mat gt [i][j] is the probability that the i-th lane line is related to the j-th vehicle.

5. The association network training method according to claim 1, characterized in that: The method further comprises: Setting a random perturbation function, and determining whether a value of the random perturbation function falls within a preset range; If yes, the first query feature in the line-vehicle feature array is determined by the lane line feature, and the first key feature and the first true value feature in the line-vehicle feature array are determined by the vehicle feature. The second query feature in the vehicle-line feature array is determined by the vehicle feature, and the second key feature and the second true value feature in the lane-line feature array are determined by the lane-line feature; If not, the first query feature in the lane feature array is determined by the vehicle feature, and the first key feature and the first true value feature in the lane feature array are determined by the lane feature. The second query feature in the vehicle-line feature array is determined by the lane line feature, and the second key feature and the second true value feature in the line-vehicle feature array are determined by the vehicle feature.

6. A lane line detection method, characterized in that: The method comprises: Obtain vehicle features and corresponding vehicle coordinates, lane line features and lane line coordinates in the current image; Based on the association network, the association matrix between the vehicle feature and the lane line feature is calculated; wherein the association network is a network trained by the association network training method according to any one of claims 1 to 5, and the association matrix is ​​any one of the vehicle-lane attribution relationship matrix and the lane-vehicle attribution relationship matrix; Based on the association matrix, selecting associated vehicle coordinates from the vehicle coordinates; Based on the associated vehicle coordinates and the lane line coordinates, a fitted lane line is generated by adopting a coordinate fitting method.

7. The lane line detection method according to claim 6, characterized in that: The method further comprises: Based on the vehicle features and the lane line features, a vehicle line feature array is calculated using an activation function ReLU; Based on the vehicle line feature array, a vehicle correction feature is calculated by using a dot product operation, and the vehicle correction feature is input into a vehicle detection head to obtain a corrected vehicle coordinate; Based on the association matrix, the associated vehicle coordinates are selected from the corrected vehicle coordinates.

8. The lane line detection method according to claim 6 or 7, characterized in that: The method further comprises: Based on the vehicle features and the lane line features, using the activation function ReLU, a lane-vehicle feature array is calculated; Based on the lane feature array, a lane correction feature is calculated by using a dot product operation, and the lane correction feature is input into a lane detection head to obtain a corrected lane coordinate; The fitted lane line is generated based on the associated vehicle coordinates and the corrected lane line coordinates.

9. A lane detection system, characterized in that: The system comprises: A feature acquisition module, wherein the feature acquisition module is configured to acquire vehicle features and corresponding vehicle coordinates, lane line features and lane line coordinates in a current image; An association matrix calculation module, wherein the association matrix calculation module is configured to calculate an association matrix between the vehicle features and the lane line features based on an association network; wherein the association network is a network trained using the association network training method according to any one of claims 1 to 5, and the association matrix is ​​any one of the vehicle-lane attribution relationship matrix and the lane-vehicle attribution relationship matrix; A coordinate extraction module, wherein the coordinate extraction module is configured to select associated vehicle coordinates from the vehicle coordinates based on the association matrix; A fitting module is configured to generate a fitting lane line by using a coordinate fitting method based on the associated vehicle coordinates and the lane line coordinates.

Images