Model training and lane line prediction methods, equipment, and autonomous driving vehicles

Abstract

This disclosure provides a model training and lane line prediction method, device, and autonomous driving vehicle, relating to the fields of artificial intelligence technology, particularly deep learning, image processing, and autonomous driving. The specific implementation scheme comprises: performing curve fitting based on a set of lane line pixels in a sample image to obtain polynomial annotation parameters corresponding to the lane lines in the sample image; obtaining annotation data for the sample image based on the polynomial annotation parameters; and training a lane line prediction model based on the sample image and its annotation data.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of deep learning, image processing, and autonomous driving. Background Art

[0002] Lane recognition is a key perception technology in autonomous driving scenarios. To apply the recognized lane information, it is often necessary to vectorize it into a curve represented by a polynomial. Current lane vectorization solutions typically involve two stages: image perception and recognition, and post-processing of the recognition results, resulting in a complex and cumbersome process. Summary of the Invention

[0003] The present disclosure provides a model training and lane line prediction method, device, and autonomous driving vehicle.

[0004] According to one aspect of the present disclosure, a model training method is provided, comprising:

[0005] Perform curve fitting based on the lane line pixel set in the sample image to obtain the polynomial annotation parameters corresponding to the lane line in the sample image;

[0006] Based on the polynomial annotation parameters, the annotation data of the sample image is obtained;

[0007] Based on the sample images and their annotated data, a lane line prediction model is trained.

[0008] According to another aspect of the present disclosure, a lane line prediction method is provided, comprising:

[0009] Obtain the image to be predicted;

[0010] The predicted image is processed based on the lane line prediction model to obtain predicted data;

[0011] The prediction data includes the parameters of the polynomial corresponding to the lane line in the image to be predicted; and the lane line prediction model is obtained by training based on the model training method provided in any embodiment of the present disclosure.

[0012] According to one aspect of the present disclosure, a model training device is provided, comprising:

[0013] A parameter acquisition module is used to perform curve fitting based on the lane line pixel set in the sample image to obtain the polynomial annotation parameters corresponding to the lane line in the sample image;

[0014] A data acquisition module, used to obtain the annotation data of the sample image based on the annotation parameters of the polynomial;

[0015] The model training module is used to train a lane line prediction model based on sample images and their annotated data.

[0016] According to another aspect of the present disclosure, a lane line prediction device is provided, comprising:

[0017] An image acquisition module, used to acquire an image to be predicted;

[0018] An image processing module is used to process the predicted image based on the lane line prediction model to obtain predicted data;

[0019] The prediction data includes the parameters of the polynomial corresponding to the lane line in the image to be predicted; and the lane line prediction model is obtained by training based on the model training device provided in any embodiment of the present disclosure.

[0020] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method in any embodiment of the present disclosure.

[0024] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method in any embodiment of the present disclosure.

[0025] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method in any embodiment of the present disclosure when executed by a processor.

[0026] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising the electronic device provided by any embodiment of the present disclosure.

[0027] According to the technology disclosed in the present invention, the trained lane line prediction model can directly output the parameters of the polynomial used to characterize the lane line based on the lane line image, thereby improving the efficiency of lane line vectorization.

[0028] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0030] Figure 1 is a schematic diagram of a model training method provided according to an embodiment of the present disclosure;

[0031] Figure 2 is a schematic diagram of a lane line prediction model according to an embodiment of the present disclosure;

[0032] Figure 3 is a schematic diagram of a lane line prediction method provided according to an embodiment of the present disclosure;

[0033] Figure 4 is a schematic diagram of a model training device provided according to one embodiment of the present disclosure;

[0034] Figure 5 is a schematic diagram of a model training device provided according to another embodiment of the present disclosure;

[0035] Figure 6 is a schematic diagram of a lane line prediction device provided according to one embodiment of the present disclosure;

[0036] Figure 7 is a schematic diagram of a lane line prediction device provided according to another embodiment of the present disclosure;

[0037] Figure 8 This is a block diagram of an electronic device used to implement the model training method or lane line prediction method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0039] Figure 1 FIG. 1 shows a schematic diagram of a model training method according to an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0040] Step S110, performing curve fitting based on the lane line pixel set in the sample image to obtain polynomial annotation parameters corresponding to the lane line in the sample image;

[0041] Step S120, obtaining annotation data of the sample image based on the annotation parameters of the polynomial;

[0042] Step S130: training a lane line prediction model based on the sample image and the labeled data of the sample image.

[0043] For example, in the embodiments of the present disclosure, a sample image may refer to an image used to train a lane line prediction model. The sample image may include a lane line image. The lane line pixel set in the sample image may include multiple pixels in the sample image. These pixels are the pixels where the lane lines are located. These pixels may be determined through manual annotation.

[0044] For example, each pixel point in the lane line pixel set can be fitted into a curve, such as a cubic polynomial curve. The cubic polynomial corresponding to the curve is, for example:

[0045] y=ax 3 +bx 2 +cx+d.

[0046] The parameters of the polynomial (also referred to as curve parameters) include a, b, c, and d. For example, the parameters of the polynomial can also be expressed as (a, b, c, d). These parameters or a polynomial determined based on these parameters can be used to represent lane lines in a sample image.

[0047] In the embodiment of the present disclosure, the annotation parameters may refer to the parameters of the curve obtained by fitting the lane line pixel set in the sample image, that is, the actual parameters of the polynomial corresponding to the lane line in the sample image.

[0048] For example, the polynomial annotation parameters can be used as the annotation data for the sample image. It can be understood that the lane prediction model trained based on the sample image and its annotation data can be used to output the polynomial parameters corresponding to the lane lines in the sample image. Therefore, this lane prediction model is an end-to-end model. By processing the image to be predicted based on this model, the lane vectorization result of the image to be predicted can be directly obtained, eliminating the need for curve fitting on the model output. This improves the efficiency of lane vectorization.

[0049] As an exemplary embodiment, step S130, training a lane prediction model based on the sample image and the labeled data of the sample image, may include:

[0050] Processing the sample image based on a preset model to obtain prediction data corresponding to the sample image; wherein the prediction data includes prediction parameters of the polynomial;

[0051] The first loss function is obtained based on the Euclidean distance between the labeled parameters and the predicted parameters of the polynomial;

[0052] According to the first loss function, the parameters of the preset model are adjusted, and when the first loss function meets the preset conditions, the preset model is used as the lane line prediction model.

[0053] Specifically, during the training process to accurately predict the polynomial parameters, the loss function is determined based on the Euclidean distance between the actual polynomial parameters corresponding to the lane lines in the sample image and the predicted parameters obtained by the model. Because the Euclidean distance intuitively reflects the gap between the predicted and actual parameters and is easy to calculate, deriving the loss function based on the Euclidean distance improves the effectiveness and speed of model parameter adjustment.

[0054] Exemplarily, the preset model can be a model designed based on a deep learning network architecture. For example, the preset model can include a convolutional network and a decoder. The convolutional network is used to extract feature information of the image input to the model, and the decoder is used to decode the feature information to obtain prediction parameters of the polynomial.

[0055] In actual applications, the polynomial corresponding to the lane line has multiple annotation parameters, for example, 4 annotation parameters. Accordingly, the preset model can predict multiple prediction parameters of the polynomial. Multiple annotation parameters correspond one to one to multiple prediction parameters. It should be noted that the Euclidean distance between the annotation parameters and the prediction parameters of the aforementioned polynomial includes the Euclidean distance between each annotation parameter in the multiple annotation parameters and its corresponding prediction parameter. For example, the polynomial corresponding to the lane line in the image has 4 annotation parameters including a, b, c, and d. Accordingly, the preset model predicts the 4 prediction parameters a′, b′, c′, and d′ of the polynomial. Then the aforementioned Euclidean distance includes the Euclidean distance between a and a′, the Euclidean distance between b and b′, the Euclidean distance between c and c′, and the Euclidean distance between d and d′.

[0056] Exemplarily, the above-mentioned preset model may include the loss function reaching a minimum, for example, the loss function no longer changes or the loss function is less than a preset threshold.

[0057] Optionally, obtaining annotation data of the sample image based on the annotation parameters of the polynomial includes:

[0058] The polynomial annotation parameters and the lane line annotation line type are used as the annotation data of the sample image.

[0059] Accordingly, the predicted data also includes a predicted line type; based on the Euclidean distance between the polynomial parameters and the predicted parameters, a first loss function is obtained, including:

[0060] The Euclidean distance between the polynomial's labeled parameters and predicted parameters is used as the second loss function;

[0061] Determine a first cross entropy based on the annotated line type and the predicted line type as the third loss function;

[0062] According to the second loss function and the third loss function, the first loss function is obtained.

[0063] In other words, during the model training annotation process, not only the polynomial parameters are annotated, but also the lane line type. Here, the line type can be determined based on various lane line attributes, including at least one of the line's solidity, color, and boundary attributes. Colors can be, for example, yellow or white. Boundary attributes are used to characterize whether a lane line is a road boundary or a non-road boundary. For example, the line type of a lane line can be a solid yellow line for a non-road boundary or a solid white line for a road boundary.

[0064] Accordingly, during the prediction process of model training, not only are the polynomial parameters corresponding to the lane lines predicted, but the lane shape is also predicted, denoted as the predicted lane shape. For lane shape, cross entropy is used as the loss function during model training. This loss function for lane shape and the loss function for polynomial parameters are then combined to produce the overall loss function.

[0065] According to the above optional method, the similarity between the labeled line type and the predicted line type is quantified by cross entropy, so that the line type prediction loss can be measured, which can enable the model to learn the ability to accurately predict the line type and improve the application effect of the model.

[0066] For example, a decoder corresponding to lane type can be set in the preset model. After the convolutional network extracts the image's feature information, the decoder is used to decode the lane type in the image. The parameters of the decoder are continuously updated during the model training process.

[0067] Optionally, the prediction data further includes confidence of the prediction parameters; and obtaining the first loss function according to the second loss function and the third loss function includes:

[0068] The second cross entropy is obtained based on the polynomial's labeled parameters, predicted parameters, and confidence level as the fourth loss function;

[0069] Based on the second loss function, the third loss function and the fourth loss function, a first loss function is obtained.

[0070] That is, during the prediction process of model training, the confidence level of the aforementioned prediction parameters is also predicted. During the loss function calculation, the polynomial's labeled and predicted parameters can be used to more accurately assess the actual confidence level of the prediction parameters. For example, the ratio between the predicted and labeled parameters can be used to represent the actual confidence level of the prediction parameters. Furthermore, the confidence level predicted by the model can represent the confidence level of the prediction. Based on the actual confidence level and the predicted confidence level, the confidence loss can be calculated.

[0071] In the above optional method, the similarity between the actual confidence level and the predicted confidence level is quantified by cross entropy, making the linear prediction loss measurable. This can be used to measure the model's ability to learn accurate prediction confidence levels, which is beneficial to improving the accuracy of lane line recognition.

[0072] Furthermore, by combining the loss of polynomial parameters, line shape loss, and confidence loss, the overall loss of the model is obtained, which is conducive to training an accurate lane line prediction model.

[0073] For example, a decoder corresponding to the confidence level can be set in the preset model. After the convolutional network extracts the feature information of the image, the decoder is used to decode and obtain the confidence level of the polynomial parameters. The parameters of the decoder are continuously updated during the model training process.

[0074] It can be seen that based on the above model training method, an end-to-end lane line prediction model can be trained. This end-to-end model can directly output vectorized parameters based on the image, thereby improving the efficiency of lane line vectorization. In a specific example, Figure 2 As shown, the model may include a convolutional network 200 and three decoders 210, 220, and 230. Convolutional network 200 may include one or more convolutional layers. Convolutional network 200 receives an input image and extracts feature information therefrom. Decoder 210 is configured to decode the feature information to obtain the parameters of the polynomial corresponding to the lane line. Decoder 220 is configured to decode the feature information to obtain the confidence level. Decoder 230 is configured to decode the feature information to obtain the lane line type.

[0075] The present disclosure also provides an application method of the lane line prediction model, that is, a lane line prediction method, such as Figure 3 As shown, the method includes:

[0076] Step S310, obtaining an image to be predicted;

[0077] Step S320: Process the image to be predicted based on the lane line prediction model to obtain prediction data; wherein the prediction data includes parameters of the polynomial corresponding to the lane line in the image to be predicted.

[0078] Optionally, the lane line prediction model is obtained by training based on the model training method of any embodiment of the present disclosure.

[0079] Optionally, the image to be predicted is processed based on the lane line prediction model to obtain prediction data, including:

[0080] Based on the convolutional network in the lane line prediction model, feature extraction is performed on the image to be predicted to obtain feature information of the image to be predicted;

[0081] The feature information is decoded based on the first decoder in the lane line prediction model to obtain the parameters of the polynomial corresponding to the lane line in the image to be predicted.

[0082] In other words, the lane line prediction model uses a convolutional network and a decoder to predict the parameters of the polynomial corresponding to the lane line. This prediction is based on deep learning and has high prediction accuracy.

[0083] Optionally, processing the image to be predicted based on the lane line prediction model to obtain prediction data further includes:

[0084] The feature information is decoded based on the second decoder in the lane line prediction model to obtain the line type of the lane line.

[0085] In other words, the lane line prediction model not only predicts the parameters of the polynomial corresponding to the lane line, but also predicts the line shape of the lane line, thereby improving the practicality of the lane line prediction model.

[0086] Optionally, processing the image to be predicted based on the lane line prediction model to obtain prediction data further includes:

[0087] The feature information is decoded based on the third decoder in the lane line prediction model to obtain the confidence of the parameters of the polynomial.

[0088] In other words, the lane line prediction model not only predicts the parameters of the polynomial corresponding to the lane line, but also predicts the confidence of the parameter. Therefore, the recognition accuracy of the lane line based on parameter representation can be evaluated based on the confidence, which improves the practicality of the lane line prediction model.

[0089] Optionally, the lane line prediction method further includes:

[0090] When the confidence level of the polynomial parameters is greater than or equal to a preset threshold, the lane line represented by the polynomial parameters is determined as a valid lane line.

[0091] Accordingly, when the confidence level of the polynomial parameters is less than a preset threshold, the lane line represented by the polynomial parameters is determined as an invalid lane line.

[0092] Through the above optional method, only lane lines with a confidence level that reaches a preset threshold are used as valid lane lines, thereby reducing misjudgment of lane lines and improving the accuracy of autonomous driving in autonomous driving scenarios.

[0093] Furthermore, in some embodiments, the lane line prediction model can be used to predict one or more lane lines, such as N lane lines, where N is an integer greater than or equal to 1. Since the model structure is fixed during the design phase, only a fixed number N of lane lines can be predicted. According to an embodiment of the present disclosure, the model outputs the parameters of a polynomial for N lane lines and outputs the confidence level of each polynomial parameter, and some invalid lane lines can be eliminated based on the confidence level. Therefore, even if the number of lane lines in the lane line image input to the model is less than N, a prediction result that matches the actual number of lane lines in the image can be output based on the model's confidence evaluation mechanism, thereby expanding the scope of application of the model.

[0094] As an implementation of the above methods, the present disclosure also provides a model training device, such as Figure 4 As shown, the device includes:

[0095] A parameter acquisition module 410 is configured to perform curve fitting based on a set of lane line pixels in a sample image to obtain polynomial annotation parameters corresponding to the lane lines in the sample image;

[0096] A data acquisition module 420 is configured to obtain annotation data of a sample image based on the annotation parameters of the polynomial;

[0097] The model training module 430 is used to train a lane line prediction model based on sample images and labeled data of the sample images.

[0098] Among them, Figure 5 As shown, the model training module 430 includes:

[0099] The image processing unit 431 is configured to process the sample image based on a preset model to obtain prediction data corresponding to the sample image; wherein the prediction data includes prediction parameters of the polynomial;

[0100] a loss function obtaining unit 432 for obtaining a first loss function based on a Euclidean distance between the polynomial labeled parameters and the predicted parameters;

[0101] The parameter adjustment unit 433 is used to adjust the parameters of the preset model according to the first loss function, and use the preset model as the lane line prediction model when the first loss function meets the preset conditions.

[0102] Among them, Figure 5 As shown, the data acquisition module 420 includes:

[0103] The annotation data acquisition unit 421 is used to use the annotation parameters of the polynomial and the annotation line type of the lane line as the annotation data of the sample image;

[0104] Accordingly, the prediction data also includes a prediction line type; the loss function acquisition unit 432 is specifically used to:

[0105] The Euclidean distance between the polynomial's labeled parameters and predicted parameters is used as the second loss function;

[0106] Determine a first cross entropy based on the annotated line type and the predicted line type as the third loss function;

[0107] According to the second loss function and the third loss function, the first loss function is obtained.

[0108] Exemplarily, the prediction data also includes the confidence of the prediction parameters; the loss function acquisition unit 432 is specifically used to:

[0109] The second cross entropy is obtained based on the polynomial's labeled parameters, predicted parameters, and confidence level as the fourth loss function;

[0110] Based on the second loss function, the third loss function and the fourth loss function, a first loss function is obtained.

[0111] As an implementation of the above methods, the present disclosure also provides a lane line prediction device, such as Figure 6 As shown, the device includes:

[0112] An image acquisition module 610 is used to acquire an image to be predicted;

[0113] An image processing module 620 is used to process the image to be predicted based on the lane line prediction model to obtain prediction data;

[0114] The prediction data includes the parameters of the polynomial corresponding to the lane line in the image to be predicted; the lane line prediction model is obtained by training based on the above-mentioned model training device.

[0115] For example, Figure 7 As shown, the image processing module 620 includes:

[0116] A feature extraction unit 621 is configured to extract features of the image to be predicted based on the convolutional network in the lane line prediction model to obtain feature information of the image to be predicted;

[0117] The parameter acquisition unit 622 is used to decode the feature information based on the first decoder in the lane line prediction model to obtain the parameters of the polynomial corresponding to the lane line in the image to be predicted.

[0118] Exemplarily, the image processing module 620 further includes:

[0119] The line type acquisition unit 623 is used to decode the feature information based on the second decoder in the lane line prediction model to obtain the line type of the lane line.

[0120] Exemplarily, the image processing module 620 further includes:

[0121] The confidence acquisition unit 624 is configured to decode the feature information based on the third decoder in the lane line prediction model to obtain the confidence of the parameters of the polynomial.

[0122] The functions of each unit, module or sub-module in each device of the embodiments of the present disclosure can be found in the corresponding description in the above method embodiments, and will not be repeated here.

[0123] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product. Optionally, the electronic device can be applied to or deployed in an autonomous vehicle. Exemplarily, the present disclosure further provides an autonomous vehicle including the electronic device.

[0124] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0125] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0126] Multiple components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0127] The computing unit 801 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the model training method and the lane line prediction method. For example, in some embodiments, the model training method and the lane line prediction method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the model training method and the lane line prediction method described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the model training method and the lane line prediction method in any other appropriate manner (e.g., by means of firmware).

[0128] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0130] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0133] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0134] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0135] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A model training method, comprising: Performing curve fitting based on a lane line pixel set in a sample image to obtain annotation parameters of a polynomial corresponding to the lane line in the sample image; Obtaining annotation data of the sample image based on the annotation parameters of the polynomial; Processing the sample image based on a preset model to obtain prediction data corresponding to the sample image; wherein the prediction data includes prediction parameters of the polynomial; Obtaining a first loss function based on a Euclidean distance between the labeled parameters of the polynomial and the predicted parameters; According to the first loss function, the parameters of the preset model are adjusted, and when the first loss function meets the preset conditions, the preset model is used as the lane line prediction model.

2. The method according to claim 1, wherein Obtaining annotation data of the sample image based on the annotation parameters of the polynomial, including: Using the polynomial annotation parameters and the lane line annotation type as the annotation data of the sample image; Correspondingly, the predicted data also includes a predicted line type; and obtaining a first loss function based on the Euclidean distance between the parameters of the polynomial and the predicted parameters includes: Using the Euclidean distance between the labeled parameters of the polynomial and the predicted parameters as a second loss function; Determine a first cross entropy based on the marked line type and the predicted line type as a third loss function; The first loss function is obtained according to the second loss function and the third loss function.

3. The method according to claim 2, wherein: The prediction data also includes the confidence of the prediction parameter; and obtaining the first loss function according to the second loss function and the third loss function includes: Obtaining a second cross entropy based on the labeled parameters of the polynomial, the prediction parameters, and the confidence level as a fourth loss function; The first loss function is obtained based on the second loss function, the third loss function and the fourth loss function.

4. A lane line prediction method, comprising: Obtain the image to be predicted; Processing the image to be predicted based on the lane line prediction model to obtain prediction data; Wherein, the prediction data includes parameters of a polynomial corresponding to the lane line in the image to be predicted; wherein, the lane line prediction model is trained based on the method described in any one of claims 1-3.

5. The method according to claim 4, wherein The processing of the image to be predicted based on the lane line prediction model to obtain prediction data includes: Performing feature extraction on the image to be predicted based on the convolutional network in the lane line prediction model to obtain feature information of the image to be predicted; The feature information is decoded based on a first decoder in the lane line prediction model to obtain parameters of a polynomial corresponding to the lane line in the image to be predicted.

6. The method according to claim 5, wherein: The processing of the image to be predicted based on the lane line prediction model to obtain prediction data further includes: The feature information is decoded based on a second decoder in the lane line prediction model to obtain the line type of the lane line.

7. The method according to claim 5 or 6, wherein: The processing of the image to be predicted based on the lane line prediction model to obtain prediction data further includes: The feature information is decoded based on a third decoder in the lane line prediction model to obtain confidence levels of the parameters of the polynomial.

8. The method according to claim 7, further comprising: When the confidence level of the parameters of the polynomial is greater than or equal to a preset threshold, the lane line represented by the parameters of the polynomial is determined as a valid lane line.

9. A model training device comprising: A parameter acquisition module, configured to perform curve fitting based on a set of lane line pixels in a sample image to obtain polynomial annotation parameters corresponding to the lane lines in the sample image; A data acquisition module, configured to obtain the labeled data of the sample image based on the labeled parameters of the polynomial; A model training module, configured to train a lane line prediction model based on the sample image and the labeled data of the sample image; The model training module includes: An image processing unit, configured to process the sample image based on a preset model to obtain prediction data corresponding to the sample image; wherein the prediction data includes prediction parameters of the polynomial; a loss function obtaining unit, configured to obtain a first loss function based on a Euclidean distance between the labeled parameters of the polynomial and the predicted parameters; A parameter adjustment unit is used to adjust the parameters of the preset model according to the first loss function, and use the preset model as the lane line prediction model when the first loss function meets the preset conditions.

10. The device according to claim 9, wherein The data acquisition module includes: a labeling data acquiring unit, configured to use the labeling parameters of the polynomial and the labeling line type of the lane line as labeling data of the sample image; Correspondingly, the prediction data also includes a prediction line type; the loss function acquisition unit is specifically used to: Using the Euclidean distance between the labeled parameters of the polynomial and the predicted parameters as a second loss function; Determine a first cross entropy based on the marked line type and the predicted line type as a third loss function; The first loss function is obtained according to the second loss function and the third loss function.

11. The device according to claim 10, wherein The prediction data also includes the confidence of the prediction parameter; the loss function acquisition unit is specifically used to: Obtaining a second cross entropy based on the labeled parameters of the polynomial, the prediction parameters, and the confidence level as a fourth loss function; The first loss function is obtained based on the second loss function, the third loss function and the fourth loss function.

12. A lane line prediction device, comprising: An image acquisition module, used to acquire an image to be predicted; An image processing module is used to process the image to be predicted based on the lane line prediction model to obtain prediction data; Wherein, the prediction data includes parameters of the polynomial corresponding to the lane line in the image to be predicted; wherein, the lane line prediction model is trained based on the device described in any one of claims 9-11.

13. The device according to claim 12, wherein The image processing module includes: a feature extraction unit, configured to extract features of the image to be predicted based on a convolutional network in the lane line prediction model to obtain feature information of the image to be predicted; A parameter acquisition unit is used to decode the feature information based on the first decoder in the lane line prediction model to obtain parameters of the polynomial corresponding to the lane line in the image to be predicted.

14. The device according to claim 13, wherein The image processing module also includes: A line type acquisition unit is used to decode the feature information based on the second decoder in the lane line prediction model to obtain the line type of the lane line.

15. The device according to claim 13 or 14, wherein The image processing module also includes: A confidence acquisition unit is used to decode the feature information based on the third decoder in the lane line prediction model to obtain the confidence of the parameters of the polynomial.

16. The apparatus according to claim 15, further comprising: The validity determination module is configured to determine the lane line represented by the parameters of the polynomial as a valid lane line if the confidence level of the parameters of the polynomial is greater than or equal to a preset threshold.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are configured to cause a computer to execute the method according to any one of claims 1 to 8.

19. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.

20. An autonomous driving vehicle comprising the electronic device according to claim 17.

Images