Lane line recognition method and device, storage medium and electronic device

Abstract

The present invention discloses a lane line recognition method and device, a storage medium and an electronic device. The method comprises: performing perspective transformation processing on an original image to obtain a perspective transformation matrix, wherein the original image contains a target area where the lane line to be recognized is located; using the perspective transformation matrix to transform the original image into a perspective image, wherein the area ratio of the target area in the perspective image is greater than the area ratio of the target area in the original image; determining a parameter matrix of the perspective image, wherein the parameter matrix is ​​used to indicate the equation parameters of the lane line; based on the parameter matrix, recognizing the lane line in the target area. The present invention solves the technical problem of low lane line recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image information technology, and in particular to a lane line recognition method and device, a storage medium and an electronic device. Background Art

[0002] At present, lane line detection technology is not only used in the field of autonomous driving, but also in the field of road scene monitoring, and its application scenarios are very wide. If the application scenario covers a large area, it will cause the lane lines at a farther position to be unable to be recognized or the recognition accuracy will be low.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] Embodiments of the present invention provide a lane line recognition method and device, a storage medium, and an electronic device to at least solve the technical problem of low lane line recognition accuracy.

[0005] According to one aspect of an embodiment of the present invention, a lane line recognition method is provided, comprising: performing perspective transformation processing on an original image to obtain a perspective transformation matrix, wherein the original image contains a target area where a lane line to be recognized is located; using the perspective transformation matrix to transform the original image into a perspective image, wherein an area ratio of the target area in the perspective image is greater than an area ratio of the target area in the original image; determining a parameter matrix of the perspective image, wherein the parameter matrix is ​​used to indicate equation parameters of the lane line; and recognizing the lane line in the target area based on the parameter matrix.

[0006] According to another aspect of an embodiment of the present invention, a lane line recognition device is also provided, including: a perspective unit, used to perform perspective transformation processing on an original image to obtain a perspective transformation matrix, wherein the original image contains a target area where the lane line to be recognized is located; a transformation unit, used to transform the original image into a perspective image using the perspective transformation matrix, wherein the area ratio of the target area in the perspective image is greater than the area ratio of the target area in the original image; a determination unit, used to determine a parameter matrix of the perspective image, wherein the parameter matrix is ​​used to indicate equation parameters of the lane line; and an identification unit, used to identify the lane line in the target area based on the parameter matrix.

[0007] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above lane line recognition method when running.

[0008] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the lane line recognition method through the computer program.

[0009] In an embodiment of the present invention, a perspective transformation matrix is ​​obtained by performing perspective transformation processing on an original image, the original image is transformed according to the perspective transformation matrix to obtain a perspective image, a parameter matrix is ​​determined from the perspective image, and a curve for describing a lane line is obtained using the parameter matrix. The original image is transformed into a perspective image by obtaining the perspective transformation matrix, the area ratio of a target area where the lane line is located is magnified, and then the lane line is identified based on the perspective image and the determination of the parameter matrix, thereby achieving the purpose of accurately identifying the lane line in the target area with a smaller area ratio in the original image, thereby achieving the technical effect of accurately identifying the lane line contained in the image, and thus solving the technical problem of low lane line recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0011] Figure 1 is a schematic diagram of an application environment of an optional lane line recognition method according to an embodiment of the present invention;

[0012] Figure 2 is a schematic flow chart of an optional lane line recognition method according to an embodiment of the present invention;

[0013] Figure 3 is a schematic diagram of perspective transformation of an optional lane line recognition method according to an embodiment of the present invention;

[0014] Figure 4 is a schematic diagram of perspective transformation of an optional lane line recognition method according to an embodiment of the present invention;

[0015] Figure 5 is a schematic flow chart of an optional lane line recognition method according to an embodiment of the present invention;

[0016] Figure 6 is a schematic flow chart of an optional lane line recognition method according to an embodiment of the present invention;

[0017] Figure 7 is a schematic flow chart of an optional lane line recognition method according to an embodiment of the present invention;

[0018] Figure 8is a schematic flow chart of an optional lane line recognition method according to an embodiment of the present invention;

[0019] Fig. 9 is a schematic flow chart of an optional lane line recognition method according to an embodiment of the present invention;

[0020] Fig.10 is a schematic flow chart of an optional lane line recognition method according to an embodiment of the present invention;

[0021] Fig.11 is a schematic diagram of a model training of an optional lane line recognition method according to an embodiment of the present invention;

[0022] Fig.12 is a data flow diagram of an optional lane line recognition method according to an embodiment of the present invention;

[0023] Fig.13 is a schematic structural diagram of an optional lane line recognition device according to an embodiment of the present invention;

[0024] Fig.14 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] According to one aspect of an embodiment of the present invention, a lane line recognition method is provided. Optionally, the lane line recognition method can be applied to, but is not limited to, Figure 1 In the environment shown in FIG. 1 , the target device 102 sends the acquired original image to the server 112 through the network 110 to realize the recognition of the lane lines in the original image. The server 112 runs a database 114 and a processing engine 116. The database 114 is used for data storage, and the processing engine 116 is used for realizing the recognition of the lane lines in the original image based on the data stored in the database 114.

[0028] The method is not limited to sequentially executing S102 to S108 to realize lane line recognition. Obtain a perspective transformation matrix. Perform perspective transformation on the original image to obtain a perspective transformation matrix, wherein the original image contains a target area where the lane line to be recognized is located. Transform the original image into a perspective image. Transform the original image into a perspective image using the perspective transformation matrix, wherein the area ratio of the target area in the perspective image is greater than the area ratio of the target area in the original image. Obtain a parameter matrix. Determine a parameter matrix based on the perspective transformation matrix, wherein the parameter matrix is ​​used to indicate the equation parameters of the lane line. Identify the lane line. Identify the lane line in the target area based on the parameter matrix.

[0029] Optionally, in this embodiment, the target device is a terminal device configured with a target client, which may include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an IOS phone, etc.), a laptop, a tablet computer, a PDA, a MID (Mobile Internet Devices, mobile Internet devices), a PAD, a desktop computer, a smart TV, a camera terminal, etc. The target client may be a client with an image acquisition function, not limited to a camera client, an image client, a video client, an instant messaging client, a browser client, an education client, etc. The network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The server may be a single server, or a server cluster consisting of multiple servers, or a cloud server. The above is only an example, and no limitation is made to this in this embodiment.

[0030] As an optional implementation, Figure 2 As shown, the lane line recognition method includes:

[0031] S202, performing perspective transformation processing on the original image to obtain a perspective transformation matrix, wherein the original image contains a target area where the lane line to be identified is located;

[0032] S204, transforming the original image into a perspective image using a perspective transformation matrix, wherein the area ratio of the target area in the perspective image is greater than the area ratio of the target area in the original image;

[0033] S206, determining a parameter matrix of the perspective image, wherein the parameter matrix is ​​used to indicate equation parameters of the lane line;

[0034] S208: Identify lane lines in the target area based on the parameter matrix.

[0035] Optionally, the target area in the original image is the area of ​​the lane line to be identified. The target area is not limited to an area in the original image that is smaller or more difficult to identify the lane line. Converting the curve according to the type of the lane line is not limited to displaying the curve on the original image according to the type of the lane line to obtain a target image in which the lane line is identified.

[0036] Optionally, the perspective transformation process is not limited to Figure 3 As shown in . Assume that the initial image has lane line pixels at a and b, and the two pixels are adjacent. Figure 3 As shown in (1), the perspective transformation is equivalent to a bird's-eye view, and then transformed to Figure 3 (2) at a and b. Figure 3 (2) There is a pixel hole between a and b. By interpolation, Figure 3 The pixel attributes of the pixels at positions i and j in (2) are still lane line pixels. The pixel values ​​at positions i and j can be calculated in many ways, not limited to the average pixel value, etc., and are not limited here. Through perspective transformation, the number of lane line pixels in Figure 3 (1) is changed from Figure 3 (1) Two: a, b, increase to Figure 3 The four in (2): a, i, j, and b, increase the proportion of lane lines in the image, thereby facilitating the recognition of lane lines and improving the recognition success rate and accuracy of lane lines.

[0037] Taking the actual image as an example, lane line recognition is not limited to Figure 4 shown. Figure 4 (1) is the original image, Figure 4 (2) is the perspective image obtained by perspective transformation, Figure 4 (3) is the final image with lane lines identified. Figure 4 The D1 region in (1) is the target region, which is transformed into Figure 4 (2) In the perspective image, the D2 area increases the proportion of the area in the image. By determining the parameter matrix in the perspective image, the curve equation of the lane line is determined and displayed, and the obtained Figure 4(3) In the D3 area, the final image of the identified lane line is displayed. In the embodiment of the present application, the original image is input into the first model for perspective transformation processing to obtain a perspective transformation matrix, the original image is transformed according to the perspective transformation matrix to obtain a perspective image, the perspective image is input into the second model to obtain a parameter matrix, and the parameter matrix is ​​used to obtain a curve for describing the lane line. The perspective transformation matrix is ​​obtained by the first model to transform the original image into a perspective image, the area ratio of the target area where the lane line is located is enlarged, and then the parameter matrix is ​​determined according to the perspective image, and the curve corresponding to the lane line is constructed according to the parameter matrix to identify the lane line, so as to achieve the purpose of accurately identifying the lane line in the target area with a small area ratio in the original image, thereby achieving the technical effect of accurately identifying the lane line contained in the image, and thus solving the technical problem of low lane line recognition accuracy.

[0038] As an optional implementation, the identification of lane lines in the target area based on the parameter matrix includes:

[0039] Determine the type of lane line in the target area according to the type parameters in the parameter matrix, and construct the curve according to the coefficient parameters in the parameter matrix;

[0040] The curve is converted according to the type of lane line to obtain the recognized lane line.

[0041] Optionally, the equation parameters in the parameter matrix include a type parameter for indicating the type of the lane line and a coefficient parameter for indicating the curve equation of the lane line. The coefficient parameter is used to indicate the coefficient of the polynomial in the curve equation corresponding to the lane line, and the type parameter is used to indicate the type of the lane line.

[0042] As an optional implementation, Figure 5 As shown, the above-mentioned determining the type of lane line in the target area according to the type parameter in the parameter matrix, and constructing the curve according to the coefficient parameter in the parameter matrix includes:

[0043] S502, determining a parameter matrix of N rows and M columns, where N is the number of lane lines included in the target area, and M-1 is the order of the curve equation corresponding to the curve;

[0044] S504, determining the type of lane line according to the Mth element in each row;

[0045] S506, determining the first M-1 elements of each row as coefficients of the curve equation corresponding to each curve;

[0046] S508, constructing a curve based on the coefficients of the curve equation.

[0047] Optionally, for ease of representation, N=n, M=m+1 are set, and the parameter matrix L is not limited to being represented as:

[0048]

[0049] Among them, n is set to the number of identifiable lane lines, and m is the maximum order of the polynomial in the curve equation corresponding to each lane line.

[0050] Optionally, a corresponding value is pre-set for each lane line type. For example, a white solid line is set to 0, a white dashed line is set to 1, a yellow solid line is set to 2, and a yellow dashed line is set to 3. The type parameter in the last column of each row in the parameter matrix l is used to represent the type of the corresponding lane line.

[0051] In an embodiment of the present application, a parameter matrix is ​​directly used to display the coefficient parameters and type parameters of the curve corresponding to each lane line among multiple lane lines, so that the curve can be constructed directly according to the parameters in the parameter matrix, reducing the error in curve construction caused by parameter conversion, thereby improving the accuracy of lane line recognition.

[0052] As an optional implementation, Figure 6 As shown, the above curve constructed based on the coefficients of the curve equation includes:

[0053] S602, using the first M-1 elements of each row as coefficients of polynomials of descending order in the curve equation;

[0054] S604, combining the polynomials including the coefficients into a curve equation;

[0055] S606, constructing a curve based on the curve equation.

[0056] Optionally, the first M-1 elements of each row are used as coefficients of polynomials of decreasing order in the curve equation. Assuming that the curve equation is y=F(x), it is not limited to being expressed as:

[0057] y=F(x)=a 1,1 ·x m-1 +a 1,2 ·x m-2 +...+a 1,m-1 ·x+a 1,m

[0058] Where x is used to indicate the unknown number in the polynomial in the curve equation, and a 1,n Matrix number indicating the first row and nth column in the matrix L. Every point (x, y) on the curve satisfies the curve equation F(x).

[0059] Optionally, in the polynomials contained in the curve equation, the highest order is m-1.

[0060] As an optional implementation, Figure 7 As shown, the above-mentioned conversion of the curve according to the type of lane line to obtain the identified lane line includes:

[0061] S702, determining the line color and line style of the lane line type indication;

[0062] S704: Draw a curve according to the line style and line color to obtain a lane line.

[0063] Optionally, the type of lane line is determined according to the value of the type parameter in the parameter matrix, and the type of lane line includes line color and line style. The line color is not limited to white and yellow, and the line style is not limited to dashed line and solid line. When the line style indicates a dashed line, the curve is discontinuously converted according to a preset length, and when the line style indicates a solid line, the curve is converted according to a continuous line style. For example, when the value of the type parameter is 0, the type of lane line is determined to be a white solid line, and a curve is drawn on the original image in a white style to obtain a target image of the identified lane line.

[0064] In an embodiment of the present application, the type of lane line is indicated by the type parameter in the parameter matrix, so that a curve is directly drawn on the original image according to the line color and line style indicated by the type of lane line, thereby realizing lane line recognition.

[0065] As an optional implementation, performing perspective transformation processing on the original image to obtain a perspective transformation matrix includes: performing perspective transformation processing on the original image based on the first model obtained through training to obtain a perspective transformation matrix;

[0066] Determining the parameter matrix of the perspective image includes: inputting the perspective image into the trained second model to obtain the parameter matrix;

[0067] As an optional implementation, the first original model is trained using the first sample image, and the second original model is trained using the second sample image, wherein the second sample image is an image obtained by perspective transforming the first sample image using the sample perspective transformation matrix output by the first original model;

[0068] When the target loss is less than the loss threshold, the first original model is used as the first model and the second original model is used as the second model, wherein the target loss includes a first loss between the sample perspective transformation matrix and the reference perspective transformation matrix and a second loss between the sample parameter matrix output by the second original model and the reference parameter matrix.

[0069] Optionally, the main network structure of the first model and the second model can be but is not limited to any one of an attention neural network (Transformers) and a convolutional neural network (CNN).

[0070] Optionally, the first sample image is transformed by a sample perspective transformation matrix to obtain a second sample image, and the second model is trained using the second sample image.

[0071] In an embodiment of the present application, the comprehensive loss of the two models is used as a condition for model convergence, and the two models are comprehensively trained so that the trained first model and the second model can achieve better recognition effect, so as to improve the accuracy of lane line recognition through the accuracy of the output matrix.

[0072] As an optional implementation, Figure 8 As shown, the above-mentioned training of the first original model using the first sample image includes:

[0073] S802, using the first original model to identify the first sample image to obtain a sample perspective transformation matrix;

[0074] S804, extracting first marking data of a lane line marking in the first sample image;

[0075] S806, calculating and obtaining a reference perspective transformation matrix according to the first marking data;

[0076] S808, calculating the error between the sample perspective transformation matrix and the reference perspective transformation matrix to obtain a first loss.

[0077] Optionally, the position data of the target position on the lane line contained in the first sample image is extracted as the first marking data, and the reference perspective transformation matrix is ​​obtained using the first marking data.

[0078] Optionally, assuming that the perspective transformation matrix is ​​a 3*3 matrix, the perspective transformation matrix H can be expressed as:

[0079]

[0080] The calculation relationship between the pixel with coordinates (u, v) in the original image and the perspective transformation matrix can be expressed as: [x` y` ω`] = [uv ω]H, where x′, y′, and w′ are the coordinates and parameters after perspective transformation, and parameter ω = 1.

[0081] Then the calculation formula of the coordinates (x, y) corresponding to the coordinates (u, v) in the perspective image obtained after the perspective transformation can be expressed as:

[0082]

[0083]

[0084] Among them, a n is the value at the corresponding position in the matrix H.

[0085] Optionally, the calculation method of the first loss includes but is not limited to: mean square error loss (MSE), mean absolute error (MAE).

[0086] Optionally, the sample perspective transformation matrix and the reference perspective transformation matrix are denoted as H r and H g , when the first loss adopts mean square error, the first loss loss1 is expressed as:

[0087]

[0088] Among them, a ri is the perspective transformation matrix H r The matrix values ​​in a gi is the perspective transformation matrix H g The matrix values ​​in .

[0089] As an optional implementation, Fig. 9 As shown, the training of the second original model using the second sample image includes:

[0090] S902, performing image transformation on the first sample image according to the sample perspective transformation matrix to obtain a second sample image;

[0091] S904, using the second original model to identify the second sample image to obtain a sample parameter matrix;

[0092] S906, extracting second marking data of a marking on a lane line in a second sample image;

[0093] S908, calculating and obtaining a reference parameter matrix according to the second label data;

[0094] S910, calculating the error between the sample parameter matrix and the reference parameter matrix to obtain a second loss.

[0095] Optionally, the position data of the target position on the lane line included in the second sample image is extracted to serve as the second label data. The reference parameter matrix is ​​calculated using the second label data.

[0096] Optionally, the calculation method of the second loss includes but is not limited to: mean square error loss (MSE), mean absolute error (MAE).

[0097] Optionally, the sample parameter matrix and the reference parameter matrix are denoted as Lr and L g , then the second loss loss2 is expressed as: loss2 = F(L r ,L g ).

[0098] As an optional implementation, Fig.10 As shown, after obtaining the second loss, the method further includes:

[0099] S1002, obtaining loss weights corresponding to the first loss and the second loss;

[0100] S1004, performing a weighted summation of the first loss and the second loss to obtain a target loss;

[0101] S1006, when the target loss is greater than or equal to the loss threshold, optimizing the first original model and the second original model using the first loss and the second loss;

[0102] S1008, when the target loss is less than the loss threshold, the first sample model is used as the first model, and the second sample model is used as the second model.

[0103] Optionally, a loss weight corresponding to the first loss is determined, and a loss weight corresponding to the second loss is determined according to the loss weight of the first loss. The sum of the loss weight of the first loss and the loss weight of the second loss is 1.

[0104] Optionally, α is used to represent the first loss weight, and the target loss can be expressed as:

[0105] loss total =α·loss1+(1-α)·loss2

[0106] Among them, α∈[0,1].

[0107] Optionally, optimizing the first original model and the second original model using the first loss and the second loss is not limited to updating the parameter values ​​of the first original model and the second original model using the direction propagation of the first loss and the second loss. Wherein, the loss value loss`1 for updating the first original model by back propagation can be expressed as:

[0108] loss`1=α·loss total

[0109] The back-propagation loss value loss`2 used to update the second original model can be expressed as:

[0110] loss`2=(1-α)·loss total

[0111] Optionally, taking the case where both the first model and the second model are neural network models, the model training and acquisition process is not limited to the following: Fig.11 As shown, it includes a lane line recognition part and a perspective transformation matrix part. When the model training starts, the sample image is used as the input image and input into the neural network 1 to obtain the output predicted perspective transformation matrix, the perspective transformation matrix is ​​used to obtain the perspective transformation image, and the perspective transformation image is input into the neural network 2 to obtain the output predicted lane line parameter matrix.

[0112] At the same time, the first annotated image with annotations is used for calculation to obtain the real perspective transformation matrix. Based on the predicted perspective transformation matrix and the real perspective transformation matrix, the loss of the perspective transformation matrix is ​​calculated. The second annotated image with annotations is used for calculation to obtain the real lane line parameter matrix, and the lane line parameter matrix loss is calculated in combination with the predicted lane line parameter matrix. And the total loss is calculated. When the total loss is obtained, it is determined whether the total loss meets the conditions. The condition can be less than a preset threshold. If it is determined that the total loss meets the conditions, the model training is terminated, and the current neural network 1 is used as the first model and the neural network 2 is used as the second model.

[0113] If the judgment is no and the total loss does not meet the conditions, the loss is fed back to the neural network to optimize the neural network.

[0114] Optionally, the first model and the second model take a neural network model as an example, and the data flow of the lane line recognition process is not limited to Fig.12 As shown. The original image is input into the neural network 1. The input image is used to predict the 3×3 perspective transformation matrix. The input image is processed using the perspective transformation matrix. Specifically, but not limited to, the position of each pixel in the input image is obtained in turn, and the current pixel coordinate position is used to calculate the 3×3 perspective transformation matrix to obtain the new position of the current pixel after the perspective transformation. The image position without pixel value is approximated or blacked out to obtain the image after the perspective transformation, that is, the perspective image. The perspective image is input into the convolutional neural network 2, and a lane line parameter matrix of size (m+1)×n is predicted. Each lane line is identified as an m-order polynomial. The m-order polynomial is input into x to obtain the y value, and the lane line is drawn.

[0115] In an embodiment of the present application, when the target loss is greater than the loss threshold and there is no complete model training, the directional propagation of the loss value is used to update the parameter values ​​of the original model to train the model to a level with smaller overall error, so that the matrix output error between the first model and the second model is smaller, thereby improving the accuracy of lane line recognition.

[0116] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0117] According to another aspect of the embodiments of the present invention, a lane line recognition device for implementing the lane line recognition method is also provided. Fig.13 As shown, the device comprises:

[0118] A perspective unit 1302 is used to perform perspective transformation processing on the original image to obtain a perspective transformation matrix, wherein the original image contains a target area where the lane line to be identified is located;

[0119] A transformation unit 1304, configured to transform the original image into a perspective image using a perspective transformation matrix, wherein an area ratio of the target area in the perspective image is greater than an area ratio of the target area in the original image;

[0120] A determining unit 1306, configured to determine a parameter matrix of the perspective image, wherein the parameter matrix is ​​used to indicate equation parameters of the lane line;

[0121] The recognition unit 1308 is used to convert the curve according to the type of the lane line to obtain the recognized lane line and recognize the lane line in the target area based on the parameter matrix.

[0122] Optionally, the identification unit 1308 includes:

[0123] A construction module, used to determine the type of lane line in the target area according to the type parameters of the parameter matrix, and to construct a curve according to the coefficients in the reference matrix;

[0124] The conversion module is used to convert the curve according to the type of lane line to obtain the recognized lane line.

[0125] Optionally, the above building blocks include:

[0126] A first determination module is used to determine a parameter matrix of N rows and M columns, where N is the number of lane lines included in the target area, and M-1 is the order of the curve equation corresponding to the curve;

[0127] A second determination module is used to determine the type of lane line according to the Mth element in each row;

[0128] A third determination module is used to determine the first M-1 elements of each row as coefficients of the curve equation corresponding to each curve;

[0129] The first building module is used to build a curve based on the coefficients of the curve equation.

[0130] Optionally, the first building block includes:

[0131] The coefficient module is used to use the first M-1 elements of each row as the coefficients of the polynomials of the curve equation from high to low order;

[0132] A combination module, used to combine polynomials containing coefficients into curve equations;

[0133] Equation module, used to construct curves based on curve equations.

[0134] Optionally, the above conversion module includes:

[0135] A type module, used to determine the line color and line style of the lane line type indication;

[0136] The drawing module is used to draw curves according to line styles and line colors to obtain lane lines.

[0137] Optionally, the perspective unit further includes a first model module, which is used to perform perspective transformation processing on the original image based on the trained first model to obtain a perspective transformation matrix.

[0138] The determination unit further includes a second model module, which is used to input the perspective image into the trained second model to obtain a parameter matrix.

[0139] Optionally, the lane line recognition device further includes a training unit, including:

[0140] A training module, used to train the first original model using the first sample image, and to train the second original model using the second sample image, wherein the second sample image is an image obtained by perspective transforming the first sample image using the sample perspective transformation matrix output by the first original model;

[0141] A completion module is used to use the first original model as the first model and the second original model as the second model when the target loss is less than the loss threshold, wherein the target loss includes a first loss between the sample perspective transformation matrix and the reference perspective transformation matrix and a second loss between the sample parameter matrix output by the second original model and the reference parameter matrix.

[0142] Optionally, the training module includes:

[0143] A first acquisition module, used to recognize the first sample image using the first original model to obtain a sample perspective transformation matrix;

[0144] A first extraction module, used to extract first marking data of a lane line marking in a first sample image;

[0145] A first reference module, used for calculating a reference perspective transformation matrix according to the first marking data;

[0146] The first calculation module is used to calculate the error between the sample perspective transformation matrix and the reference perspective transformation matrix to obtain a first loss.

[0147] Optionally, the training module includes:

[0148] A transformation module, used for performing image transformation on the first sample image according to the sample perspective transformation matrix to obtain a second sample image;

[0149] A second acquisition module is used to identify the second sample image using the second original model to obtain a sample parameter matrix;

[0150] A second extraction module, used to extract second marking data of a lane line marking in a second sample image;

[0151] A second reference module, used for calculating a reference parameter matrix according to the second label data;

[0152] The second calculation module is used to calculate the error between the sample parameter matrix and the reference parameter matrix to obtain a second loss.

[0153] Optionally, the lane line recognition device further includes a loss unit, which is configured to, after obtaining the second loss, include:

[0154] A weight module, used to obtain loss weights corresponding to the first loss and the second loss;

[0155] A summation module, used for performing a weighted summation of the first loss and the second loss to obtain a target loss;

[0156] An optimization module, configured to optimize the first original model and the second original model using the first loss and the second loss when the target loss is greater than or equal to the loss threshold;

[0157] The end module is used to use the first sample model as the first model and the second sample model as the second model when the target loss is less than the loss threshold.

[0158] In an embodiment of the present application, a perspective transformation matrix is ​​obtained by performing perspective transformation processing on the original image, the original image is transformed according to the perspective transformation matrix to obtain a perspective image, a parameter matrix is ​​determined in the perspective image, and a curve for describing the lane line is obtained by using the parameter matrix. The original image is transformed into a perspective image by obtaining the perspective transformation matrix, the area ratio of the target area where the lane line is located is magnified, and then the lane line is identified based on the perspective image and the determination of the parameter matrix, thereby achieving the purpose of accurately identifying the lane line in the target area with a smaller area ratio in the original image, thereby achieving the technical effect of accurately identifying the lane line contained in the image, and thus solving the technical problem of low lane line recognition accuracy.

[0159] According to another aspect of the embodiments of the present invention, an electronic device for implementing the lane line recognition method is also provided. The electronic device may be Figure 1 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a server as an example. Fig.14 As shown, the electronic device includes a memory 1402 and a processor 1404. The memory 1402 stores a computer program, and the processor 1404 is configured to execute the steps in any of the above method embodiments through the computer program.

[0160] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0161] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0162] S1, performing perspective transformation processing on the original image to obtain a perspective transformation matrix, wherein the original image contains a target area where the lane line to be identified is located;

[0163] S2, transforming the original image into a perspective image using a perspective transformation matrix, wherein the area ratio of the target area in the perspective image is greater than the area ratio of the target area in the original image;

[0164] S3, determining a parameter matrix of the perspective image, wherein the parameter matrix is ​​used to indicate equation parameters of the lane line;

[0165] S4, based on the parameter matrix, the lane lines in the target area are identified.

[0166] Alternatively, a person skilled in the art may understand that: Fig.14The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an IOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.14 The structure of the electronic device is not limited. Fig.14 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig.14 Different configurations shown.

[0167] Among them, the memory 1402 can be used to store software programs and modules, such as the program instructions / modules corresponding to the lane line recognition method and device in the embodiment of the present invention. The processor 1404 executes various functional applications and data processing by running the software programs and modules stored in the memory 1402, that is, to implement the above-mentioned lane line recognition method. The memory 1402 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1402 may further include a memory remotely located relative to the processor 1404, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1402 can be specifically, but not limited to, used to store original images, perspective transformation matrices, perspective images, parameter matrices, curves and other information. As an example, such as Fig.14 As shown, the memory 1402 may include, but is not limited to, the perspective unit 1302, the transformation unit 1304, the determination unit 1306, and the recognition unit 1308 in the lane line recognition device. In addition, it may also include, but is not limited to, other module units in the lane line recognition device, which will not be repeated in this example.

[0168] Optionally, the transmission device 1406 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 1406 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1406 is a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0169] In addition, the electronic device further includes: a display 1408 for displaying the original image and the identified lane lines; and a connection bus 1410 for connecting the various module components in the electronic device.

[0170] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. Among them, the nodes may form a peer-to-peer (P2P, PeerToPeer) network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0171] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in the various optional implementations of the lane line recognition described above. The computer program is configured to execute the steps of any of the above method embodiments when it is run.

[0172] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0173] S1, performing perspective transformation processing on the original image to obtain a perspective transformation matrix, wherein the original image contains a target area where the lane line to be identified is located;

[0174] S2, transforming the original image into a perspective image using a perspective transformation matrix, wherein the area ratio of the target area in the perspective image is greater than the area ratio of the target area in the original image;

[0175] S3, determining a parameter matrix of the perspective image, wherein the parameter matrix is ​​used to indicate equation parameters of the lane line;

[0176] S4, based on the parameter matrix, the lane lines in the target area are identified.

[0177] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0178] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0179] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.

[0180] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0182] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0184] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A lane line recognition method, characterized in that: include: Performing perspective transformation processing on the original image to obtain a perspective transformation matrix, wherein the original image contains a target area where the lane line to be identified is located; The original image is transformed into a perspective image using the perspective transformation matrix, wherein the area ratio of the target area in the perspective image is greater than the area ratio of the target area in the original image; Determining a parameter matrix of the perspective image, wherein the parameter matrix is ​​used to indicate equation parameters of the lane line; Based on the parameter matrix, identifying the lane line in the target area; Performing perspective transformation processing on the original image to obtain a perspective transformation matrix includes: performing the perspective transformation processing on the original image based on a first model obtained through training to obtain the perspective transformation matrix, wherein the first model is obtained by the following steps: training the first original model using a first sample image, and training the second original model using a second sample image, wherein the second sample image is an image obtained by performing perspective transformation on the first sample image using a sample perspective transformation matrix output by the first original model; when a target loss is less than a loss threshold, using the first original model as the first model, and using the second original model as the second model, wherein the target loss includes a first loss between the sample perspective transformation matrix and a reference perspective transformation matrix, and a second loss between the sample parameter matrix output by the second original model and a reference parameter matrix; Based on the parameter matrix, identifying the lane line in the target area includes: determining the type of the lane line in the target area according to the type parameter in the parameter matrix, and constructing a curve according to the coefficient parameter in the parameter matrix, wherein the type parameter is used to indicate the type of the lane line, and the coefficient parameter is used to indicate the coefficient of the polynomial in the curve equation corresponding to the lane line; transforming the curve according to the type of the lane line to obtain the identified lane line.

2. The method according to claim 1, characterized in that Determining the type of the lane line in the target area according to the type parameters in the parameter matrix, and constructing a curve according to the coefficient parameters in the parameter matrix includes: Determine a parameter matrix of n rows and m columns, where n is the number of lane lines included in the target area, and m-1 is the order of the curve equation corresponding to the curve; Determine the type of the lane line according to the mth element in each row; Determine the first m-1 elements of each row as the coefficients of the curve equation corresponding to each curve; The curve is constructed based on the coefficients of the curve equation.

3. The method according to claim 2, characterized in that Constructing the curve based on the coefficients of the curve equation includes: The first m-1 elements of each row are used as coefficients of polynomials of descending order in the curve equation; Combining polynomials including the coefficients into the curve equation; The curve is constructed based on the curve equation.

4. The method according to claim 2, characterized in that: The curve is converted according to the type of the lane line, and the identified lane line includes: Determine the line color and line style of the lane line type indication; The curve is drawn according to the line style and line color to obtain the lane line.

5. The method according to claim 1, characterized in that: The determining the parameter matrix of the perspective image comprises: inputting the perspective image into the trained second model to obtain the parameter matrix.

6. The method according to claim 1, characterized in that The training of the first original model by using the first sample image comprises: Using the first original model to identify the first sample image, to obtain the sample perspective transformation matrix; Extracting first marking data of a lane line marking in the first sample image; Calculate the reference perspective transformation matrix according to the first marking data; An error between the sample perspective transformation matrix and the reference perspective transformation matrix is ​​calculated to obtain the first loss.

7. The method according to claim 6, characterized in that The training of the second original model by using the second sample image comprises: Performing image transformation on the first sample image according to the sample perspective transformation matrix to obtain the second sample image; Using the second original model to identify the second sample image, to obtain the sample parameter matrix; Extracting second marking data of a marking on the lane line in the second sample image; Calculate the reference parameter matrix according to the second label data; An error between the sample parameter matrix and the reference parameter matrix is ​​calculated to obtain the second loss.

8. The method according to claim 7, characterized in that After obtaining the second loss, the method further includes: Obtaining loss weights corresponding to the first loss and the second loss; Performing a weighted summation on the first loss and the second loss to obtain the target loss; When the target loss is greater than or equal to the loss threshold, optimizing the first original model and the second original model by using the first loss and the second loss; When the target loss is less than the loss threshold, the first original model is used as the first model, and the second original model is used as the second model.

9. A lane line recognition device, characterized in that: include: A perspective unit, used to perform perspective transformation processing on the original image to obtain a perspective transformation matrix, wherein the original image contains a target area where the lane line to be identified is located; a transformation unit, configured to transform the original image into a perspective image using the perspective transformation matrix, wherein an area ratio of the target area in the perspective image is greater than an area ratio of the target area in the original image; A determination unit, configured to determine a parameter matrix of the perspective image, wherein the parameter matrix is ​​used to indicate equation parameters of the lane line; an identification unit, configured to identify the lane line in the target area based on the parameter matrix; The perspective unit is further used to perform the perspective transformation processing on the original image based on the first model obtained through training to obtain the perspective transformation matrix, wherein the first model is obtained by the following steps: training the first original model using the first sample image, and training the second original model using the second sample image, wherein the second sample image is an image obtained by performing perspective transformation on the first sample image using the sample perspective transformation matrix output by the first original model; when the target loss is less than the loss threshold, using the first original model as the first model, and using the second original model as the second model, wherein the target loss includes a first loss between the sample perspective transformation matrix and the reference perspective transformation matrix and a second loss between the sample parameter matrix output by the second original model and the reference parameter matrix; The recognition unit is further used to determine the type of the lane line in the target area according to the type parameter in the parameter matrix, and construct a curve according to the coefficient parameter in the parameter matrix, wherein the type parameter is used to indicate the type of the lane line, and the coefficient parameter is used to indicate the coefficient of the polynomial in the curve equation corresponding to the lane line; the curve is transformed according to the type of the lane line to obtain the identified lane line.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, and the program executes the method described in any one of claims 1 to 8 when executed.

11. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

Images