A lane line detection method and device, electronic equipment and storage medium

Abstract

This application relates to a lane line detection method, apparatus, electronic device, and storage medium, belonging to the field of autonomous driving technology. The lane line detection method includes: acquiring an image to be detected; determining an initial set of points in the image, wherein the initial set of points is located on the lane lines of the image; extracting positional features of at least one point in the initial set of points; processing the positional features of the at least one point using a pre-trained first network model to obtain the trend information of the corresponding lane line; and generating a target lane line containing the initial set of points based on the trend information. This application determines an initial set of points on the lane lines in a road image, processes the positional features of the initial points using a network model, thereby obtaining the trend information of the lane lines, and then can quickly generate the complete lane lines of the road image based on this trend information.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving technology, specifically relating to a lane line detection method, device, electronic device, and storage medium. Background Technology

[0002] With the development of automobiles, more and more artificial intelligence technologies are being applied to autonomous driving technology, and lane detection is a crucial component of this technology. However, in road images captured by sensors, the direction in which lane lines extend may be obstructed, making it impossible to determine whether there are lane lines beneath the obstruction. Therefore, it is necessary to estimate the position of obstructed lane lines to obtain the complete lane line.

[0003] Currently, lane detection in occluded conditions mostly involves processing multiple images or combining depth maps and camera poses for prediction. Processing multiple images results in large data volumes, long processing times, and poor real-time performance. While combining depth maps and camera poses increases computational complexity and cost due to their reliance on these parameters. Summary of the Invention

[0004] Therefore, the purpose of this application is to provide a lane line detection method, apparatus, electronic device and storage medium to improve the problem that existing lane line detection methods cannot quickly and accurately detect complete lane lines.

[0005] The embodiments of this application are implemented as follows:

[0006] In a first aspect, embodiments of this application provide a lane line detection method, comprising: acquiring an image to be detected; determining an initial point set in the image, wherein the initial point set is on a lane line in the image; extracting positional features of at least one point in the initial point set; processing the positional features of the at least one point using a pre-trained first network model to obtain trend information of the corresponding lane line; and generating a target lane line containing the initial point set based on the trend information.

[0007] In this embodiment, an initial set of points on the lane lines in an image is obtained, and the positional features of at least one point are extracted from it. A pre-trained first network model is used to process the extracted positional features of at least one point to obtain the trend information of the corresponding lane lines. Based on the trend information, a target lane line containing the initial set of points is generated. By predicting the trend information of the lanes, the complete lane lines can be detected quickly, which solves the problem that existing lane line detection methods cannot detect complete lane lines quickly and accurately under conditions such as occlusion.

[0008] In one possible implementation of the first aspect embodiment, each lane line is represented by points longitudinally spaced at predetermined intervals, wherein the positional features of each point include the position coordinates of that point and an offset feature; the offset feature is a high-dimensional vector characterizing the positional relationship between that point and the next point, and is extracted from the intermediate layer of a pre-trained second network model based on the position coordinates of each point.

[0009] In one possible implementation of the first aspect embodiment, when the first network model processes the positional features of each point: the input of the first network model is the position coordinates of the current point, the offset features, and the trend information learned by the first network model in the previous iteration; the output of the first network model is the coordinate offset from the current point to the next point, and the trend information learned in the current iteration.

[0010] In one possible implementation of the first aspect embodiment, the trend information is a high-dimensional vector, and the trend information input to the first network model during the first prediction is a preset initial value.

[0011] In one possible implementation of the first aspect embodiment, a pre-trained first network model is used to process the positional features of the at least one point to obtain the trend information of the corresponding lane line, and a target lane line containing the initial point set is generated based on the trend information. This includes: processing the position coordinates, offset features, and trend information previously learned by the first network model using the first network model, outputting the coordinate offset from the current point to the next point, and the trend information learned in the current iteration; determining the position coordinates of the next point based on the coordinate offset and the position coordinates of the current point; using the offset features, position coordinates, and trend information learned in the current iteration of the first network model as input to the model, outputting the offset coordinates from the next point to the point after that, and the latest trend information, and so on, until a preset stopping position is reached.

[0012] In one possible implementation of the first aspect embodiment, determining the position coordinates of the next point based on the coordinate offset and the position coordinates of the current point includes: if the horizontal distance between the position coordinates of the next point and a certain point at the same height in the initial point set is less than or equal to a predetermined pixel, then the position coordinates of the next point are corrected based on that same point at the same height.

[0013] In one possible implementation of the first aspect embodiment, the first network model is a recurrent neural network model with shared parameters at each step, and the position coordinates input to the first network model are normalized coordinates.

[0014] In one possible implementation of the first aspect embodiment, the image is divided into m*n image grids, and the coordinate offset from the current point to the next point is the coordinate offset from the center point of the grid where the current point is located to the midpoint of the lane line in the next row of grids.

[0015] In one possible implementation of the first aspect embodiment, the first network model includes a first network branch and a second network branch; the first network branch is used to predict the trend information of the corresponding lane line from a first direction; the second network branch is used to predict the trend information of the corresponding lane line from a second direction; wherein the first direction and the second direction are opposite.

[0016] In one possible implementation of the first aspect embodiment, the target lane line includes a lane line whose endpoints on both sides of the line segment reach the image critical line; the preset stop position includes the position when a predetermined number of iterations is reached or the image critical line; the image critical line includes at least one of the upper critical line, lower critical line, left critical line and right critical line.

[0017] In one possible implementation of the first aspect embodiment, a pre-trained first network model is used to process the positional features of the at least one point to obtain the trend information of the corresponding lane line, and a target lane line containing the initial point set is generated based on the trend information. This includes: processing the positional features of the at least one point using the pre-trained first network model to obtain the trend information of the corresponding lane line along a first direction, and generating multiple points based on the trend information in the first direction until a preset stopping position in the first direction is reached; starting from the preset stopping position in the first direction, the first network model is used to predict the trend information of the corresponding lane line along a second direction, and multiple points are generated based on the trend information in the second direction until a preset stopping position in the second direction is reached, wherein the first direction and the second direction are opposite.

[0018] In one possible implementation of the first aspect embodiment, a pre-trained first network model is used to process the positional features of the at least one point to obtain the trend information of the corresponding lane line, including: processing the positional features of at least one point in the initial point set in a first direction using the pre-trained first network model to obtain the trend information of the corresponding lane line along the first direction; and processing the positional features of at least one point in the initial point set in a second direction using the pre-trained first network model to obtain the trend information of the corresponding lane line along the second direction, wherein the first direction and the second direction are opposite.

[0019] In one possible implementation of the first aspect embodiment, the method further includes: obtaining a coordinate sequence of multiple lane lines;

[0020] Extract the coordinate offset and offset features from each point in each lane line to the next point to generate the first training set; and use the first training set to train the initial first network model to obtain the trained model.

[0021] In one possible implementation of the first aspect embodiment, the method further includes: acquiring a second training set, wherein the training images in the second training set are labeled with multiple lane lines and the coordinate offset of each point in each lane line to the next point; and using the second training set to train an initial second network model to obtain a trained model.

[0022] In one possible implementation of the first aspect embodiment, when the initial point set includes multiple points, the multiple points are located on the same lane line, or the multiple points are distributed on multiple lane lines.

[0023] Secondly, embodiments of this application also provide a lane line detection device, comprising: an acquisition module and a processing module; the acquisition module is used to acquire an image including lane lines; the processing module is used to determine an initial point set in the image, the initial point set being on the lane lines of the image; extract positional features of at least one point in the initial point set; process the positional features of the at least one point using a pre-trained first network model to obtain trend information of the corresponding lane line, and generate a target lane line containing the initial point set based on the trend information.

[0024] Thirdly, embodiments of this application also provide an electronic device, including: a memory and a processor, the processor being connected to the memory; the memory being used to store a program; the processor being used to invoke the program stored in the memory to execute the method provided by any possible implementation of the first aspect embodiment and / or in combination with the first aspect embodiment.

[0025] Fourthly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when run by a processor, performs the method provided by any possible implementation of the first aspect embodiments and / or in combination with the first aspect embodiments.

[0026] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The above and other objects, features, and advantages of this application will become clearer through the drawings. The same reference numerals indicate the same parts in all the drawings. The drawings are not intentionally drawn to scale to actual size; the focus is on illustrating the main points of this application.

[0028] Figure 1 A schematic diagram of the architecture of a vehicle provided in an embodiment of this application is shown;

[0029] Figure 2 A schematic flowchart of a lane line detection method provided in an embodiment of this application is shown;

[0030] Figure 3 This illustration shows a schematic diagram of the principle of lane line completion provided in an embodiment of this application;

[0031] Figure 4 This illustration shows a schematic diagram of a road image provided in an embodiment of this application;

[0032] Figure 5 This illustration shows a schematic diagram of the initial lane lines identified from a road image according to an embodiment of this application;

[0033] Figure 6 A schematic diagram of the generated target lane lines provided in an embodiment of this application is shown;

[0034] Figure 7 This application shows a module block diagram of a lane line detection device according to an embodiment of the present application; and

[0035] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0036] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0037] It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, relational terms such as "first," "second," etc., in the description of this application are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0038] Furthermore, the term "and / or" in this application is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0039] Given that existing lane detection methods cannot accurately and quickly detect complete lane lines, this application provides a lane detection method that uses a neural network model to predict the initial set of points on the lane lines in a road image and extracts the positional features of at least one point from it. Then, a pre-trained first network model is used to process the positional features of at least one point to obtain the trend information of the corresponding lane line. Based on the trend information, the target lane line containing the initial set of points can be quickly generated, thereby solving the problem that existing lane detection methods cannot accurately and quickly detect complete lane lines.

[0040] The lane detection method provided in this application can be applied to vehicles. The following will combine... Figure 1 This application describes the vehicles involved. Specifically, Figure 1This is a schematic diagram of a vehicle 100 in which the various technologies disclosed herein can be implemented. Vehicle 100 can be a car, truck, motorcycle, bus, boat, airplane, helicopter, lawnmower, excavator, snowmobile, aircraft, recreational vehicle, amusement park vehicle, farm equipment, construction equipment, tram, golf cart, train, trolleybus, or other vehicle. Vehicle 100 can operate fully or partially in an autonomous driving mode. In autonomous driving mode, vehicle 100 can control itself; for example, vehicle 100 can determine the current state of the vehicle and the current state of the environment in which the vehicle is located, determine the predicted behavior of at least one other vehicle in the environment, determine the trust level corresponding to the probability that the at least one other vehicle will perform the predicted behavior, and control vehicle 100 itself based on the determined information. In autonomous driving mode, vehicle 100 can operate without human interaction.

[0041] Vehicle 100 may include various vehicle systems, such as drive system 142, sensor system 144, control system 146, user interface system 148, computing system 150, and communication system 152. Vehicle 100 may include more or fewer systems, and each system may include multiple units. Furthermore, each system and unit of vehicle 100 may be interconnected. For example, computing system 150 is capable of data communication with one or more of drive system 142, sensor system 144, control system 146, user interface system 148, and communication system 152. Thus, one or more of the described functions of vehicle 100 may be divided into additional functional components or physical components, or combined into a smaller number of functional components or physical components. In a further example, additional functional components or physical components may be increased to, for example... Figure 1 In the example shown, the drive system 142 may include a plurality of operable components (or units) that provide kinetic energy to the vehicle 100. In one embodiment, the drive system 142 may include an engine or electric motor, wheels, a transmission, electronic systems, and a power source (or power source). The engine or electric motor may be any combination of an internal combustion engine, an electric motor, a steam engine, a fuel cell engine, a propane engine, or other forms of engine or electric motor. In some embodiments, the engine may convert a power source into mechanical energy. In some embodiments, the drive system 142 may include multiple engines or electric motors. For example, a hybrid vehicle may include a gasoline engine and an electric motor, or other configurations may be included.

[0042] The wheels of vehicle 100 can be standard wheels. The wheels of vehicle 100 can be of various forms, including single-wheel, two-wheel, three-wheel, or four-wheeled, such as the four wheels on a car or truck. Other numbers of wheels are also possible, such as six or more wheels. One or more wheels of vehicle 100 can be operated to rotate in a different direction than the other wheels. A wheel can be at least one wheel fixedly connected to a transmission. The wheel can include a combination of metal and rubber, or other materials. The transmission can include units operable to transmit mechanical power from the engine to the wheels. For this purpose, the transmission can include a gearbox, clutch, differential gears, and driveshaft. The transmission can also include other units. The driveshaft can include one or more axles that match the wheels. The electronic system can include units for transmitting or controlling electronic signals of vehicle 100. These electronic signals can be used to activate multiple lights, multiple servo mechanisms, multiple electric motors, and other electronic drives or controls in vehicle 100. The power source can be an energy source that provides power to the engine or electric motor, either wholly or partially. That is, the engine or electric motor is capable of converting the power source into mechanical energy. For example, the power source may include gasoline, petroleum, petroleum-based fuels, propane, other compressed gaseous fuels, ethanol, fuel cells, solar panels, batteries, and other electrical energy sources. The power source may optionally include any combination of a fuel tank, battery, capacitor, or flywheel. The power source may also provide energy to other systems of vehicle 100.

[0043] Sensor system 144 may include multiple sensors for sensing information about the environment and conditions of vehicle 100. For example, sensor system 144 may include an inertial measurement unit (IMU), a GNSS (Global Navigation Satellite System) transceiver (e.g., a Global Positioning System (GPS) transceiver), radar (RADAR), laser rangefinder / LIDAR (or other distance measurement device), acoustic sensors, ultrasonic sensors, and cameras or image capture devices. Sensor system 144 may include multiple sensors for monitoring vehicle 100 (e.g., oxygen (O2) monitor, fuel gauge sensor, engine oil pressure sensor, and temperature, humidity, pressure sensors, etc.). Other sensors may also be configured. One or more sensors included in sensor system 144 may be driven individually or collectively to update the position, orientation, or both of the sensors.

[0044] The IMU may include a combination of sensors (e.g., accelerometers and gyroscopes) for sensing changes in the position and orientation of vehicle 100 based on inertial acceleration. The GPS transceiver may be any sensor used to estimate the geographic location of vehicle 100. For this purpose, the GPS transceiver may include a receiver / transmitter to provide position information of vehicle 100 relative to the Earth. It should be noted that GPS is an example of a Global Navigation Satellite System; therefore, in some embodiments, the GPS transceiver may be replaced with a BeiDou Navigation Satellite System transceiver or a Galileo Navigation Satellite System transceiver. The radar unit may use radio signals to sense objects in the environment in which vehicle 100 is located. In some embodiments, in addition to sensing objects, the radar unit may also be used to sense the speed and direction of travel of objects approaching vehicle 100. The laser rangefinder or LIDAR unit (or other distance measurement device) may be any sensor that uses lasers to sense objects in the environment in which vehicle 100 is located. In one embodiment, the laser rangefinder / LIDAR unit may include a laser source, a laser scanner, and a detector. The laser rangefinder / LIDAR unit is used to operate in continuous (e.g., using heterodyne detection) or discontinuous detection modes. The camera may include means for capturing multiple images of the environment in which the vehicle 100 is located. The camera may be a still image camera or a video camera.

[0045] The control system 146 is used to control the operation of the vehicle 100 and its components (or units). Accordingly, the control system 146 may include various units, such as a steering unit, a power control unit, a braking unit, and a navigation unit.

[0046] The steering unit may be a combination of mechanisms for adjusting the forward direction of vehicle 100. A power control unit (e.g., a throttle) may be used to control the engine speed, thereby controlling the speed of vehicle 100. The braking unit may include a combination of mechanisms for decelerating vehicle 100. The braking unit may utilize friction to decelerate the vehicle in a standard manner. In other embodiments, the braking unit may convert the kinetic energy of the wheels into electrical current. The braking unit may also take other forms. The navigation unit may be any system that determines a driving path or route for vehicle 100. The navigation unit may also dynamically update the driving path as vehicle 100 travels. The control system 146 may also additionally or optionally include other components (or units) not shown or described.

[0047] User interface system 148 can be used to allow vehicle 100 to interact with external sensors, other vehicles, other computer systems, and / or the user of vehicle 100. For example, user interface system 148 may include standard visual display devices (e.g., plasma displays, liquid crystal displays (LCDs), touchscreen displays, head-mounted displays, or other similar displays), speakers or other audio output devices, microphones or other audio input devices. For example, user interface system 148 may also include navigation interfaces and interfaces for controlling the internal environment of vehicle 100 (e.g., temperature, fan, etc.).

[0048] Communication system 152 can provide vehicle 100 with a means of communicating with one or more devices or other vehicles in the vicinity. In an exemplary embodiment, communication system 152 can communicate with one or more devices directly or through a communication network. Communication system 152 can be, for example, a wireless communication system. For example, the communication system can use 3G cellular communication (e.g., CDMA, EVDO, GSM / GPRS) or 4G cellular communication (e.g., WiMAX or LTE), and can also use 5G cellular communication. Optionally, the communication system can communicate with a wireless local area network (WLAN) (e.g., using...). In some embodiments, the communication system 152 can communicate directly with one or more devices or other vehicles in the vicinity, for example, using infrared light. Or ZigBee. Other wireless protocols, such as various vehicular communication systems, are also within the scope of this application. For example, the communication system may include one or more Dedicated Short Range Communication (DSRC) devices, V2V devices, or V2X devices that conduct public or private data communication with vehicles and / or roadside stations.

[0049] The computing system 150 can control some or all of the functions of the vehicle 100. The autonomous driving control unit in the computing system 150 can be used to identify, assess, and avoid or traverse potential obstacles in the environment in which the vehicle 100 is located. Typically, the autonomous driving control unit can be used to control the vehicle 100 without a driver or to assist a driver in controlling the vehicle. In some embodiments, the autonomous driving control unit is used to combine data from a GPS transceiver, radar data, LiDAR data, camera data, and data from other vehicle systems to determine the driving path or trajectory of the vehicle 100. The autonomous driving control unit can be activated to enable the vehicle 100 to be driven in autonomous driving mode.

[0050] The computing system 150 may include at least one processor (which may include at least one microprocessor) that executes processing instructions (i.e., machine-executable instructions) stored in a non-volatile computer-readable medium (e.g., a data storage device or memory). The computing system 150 may also be a plurality of computing devices that distribute and control components or systems of the vehicle 100. In some embodiments, the memory may contain processing instructions (e.g., program logic) executed by the processor to implement various functions of the vehicle 100. In one embodiment, the computing system 150 is capable of data communication with a drive system 142, a sensor system 144, a control system 146, a user interface system 148, and / or a communication system 152. Interfaces within the computing system facilitate data communication between the computing system 150 and the drive system 142, sensor system 144, control system 146, user interface system 148, and communication system 152.

[0051] The memory may also include other instructions, including instructions for sending data, instructions for receiving data, instructions for interaction, or instructions for controlling the drive system 142, sensor system 144, or control system 146 or user interface system 148.

[0052] In addition to storing processing instructions, the memory can store various types of information or data, such as image processing parameters, road maps, and route information. This information can be used by the vehicle 100 and the computing system 150 while the vehicle 100 is operating in automatic, semi-automatic, and / or manual mode.

[0053] Although the autonomous driving control unit is shown as separate from the processor and memory, it should be understood that in some embodiments, some or all of the functions of the autonomous driving control unit may be implemented using program code instructions residing in one or more memories (or data storage devices) and executed by one or more processors, and in some cases, the autonomous driving control unit may be implemented using the same processor and / or memory (or data storage device). In some embodiments, the autonomous driving control unit may be implemented at least in part using various special-purpose circuit logics, various processors, various field-programmable gate arrays (“FPGAs”), various application-specific integrated circuits (“ASICs”), various real-time controllers, and hardware.

[0054] The computing system 150 can control the functions of the vehicle 100 based on inputs received from various vehicle systems (e.g., drive system 142, sensor system 144, and control system 146) or from the user interface system 148. For example, the computing system 150 can use inputs from the control system 146 to control the steering unit to avoid obstacles detected by the sensor system 144. In one embodiment, the computing system 150 can be used to control multiple aspects of the vehicle 100 and its systems.

[0055] Although Figure 1 The diagram shows various components (or units) integrated into vehicle 100, one or more of which may be mounted on or separately associated with vehicle 100. For example, a computing system may exist partially or entirely independent of vehicle 100. Thus, vehicle 100 can exist as separate or integrated device units. The device units constituting vehicle 100 can communicate with each other via wired or wireless communication. In some embodiments, additional components or units may be added to or removed from various systems (e.g., ...). Figure 1 (as shown by LIDAR or radar).

[0056] After introducing the vehicle architecture involved in this application, the following will combine... Figure 2 The lane line detection method provided in the embodiments of this application will be described.

[0057] Step S101: Obtain the image to be detected.

[0058] The image to be detected can be obtained by capturing real-time images of the road as the vehicle is in motion using a camera installed on the vehicle.

[0059] Step S102: Determine an initial set of points in the image, the initial set of points being on the lane lines of the image.

[0060] After acquiring the image to be detected, the image is processed to determine the initial set of points located on the lane lines. All points in the initial set lie along the entire length of the lane lines in the image. Assuming the lane line segment displayed in the image is segment AB, AB is extended to the boundary line of the image following its trend to obtain the complete segment CD. Then, any one or more points within segment CD can be selected as points in the initial set. One objective of this application is to randomly determine some initial point sets and generate complete lane lines containing these point sets, for example, by selecting any one or more points from segment CD to generate the complete lane line of segment CD.

[0061] There are various methods for determining the initial point set located on the lane lines in the image. This application is not limited to any specific implementation method, as long as it ensures that all points in the determined initial point set are along the lane lines in the image. For example, lane line pixels in the image can be identified, and some initial points can be determined from these lane line pixels to form the initial point set. Lane line pixel identification can employ semantic segmentation methods, such as using a semantic segmentation model to identify whether each pixel is a lane line pixel, and selecting one or more initial points from the lane line pixels to form the initial point set. Alternatively, the initial lane lines in the image can be directly identified using a semantic segmentation model, and the initial point set can be selected from the initial lane lines. Another approach is to use image processing methods to determine the initial point set, such as using specific image edge detection or line detection methods based on the image features of the lane lines to detect the lane lines, and extracting points from these lines as initial points.

[0062] In addition, an enumeration method can be used to determine the initial point set. For example, some pixel sets in the image can be randomly selected as candidate point sets. For each selected point, the positional features of the point (including its position coordinates and offset features) are input into a pre-trained trend prediction model, i.e., the first network model discussed below. If the prediction result indicates that the next point is still in the current pixel or the current grid (for the latter, when performing lane line detection, the image can be divided into m*n image grids, where n and m are positive integers, such as dividing a 576x1024 pixel image into 72x128 grids, and the specific number of divisions is not limited to this), then the point is considered not to be the initial point. Conversely, if prediction can continue, then the point can be considered to be the initial point. It should be understood that if, after predicting all points in the candidate point set, it is found that none of these candidate points are initial points, it means that there are no lane lines in the current frame image. Therefore, the lane line completion step for the current frame image can be skipped, and the prediction processing for the next frame image can continue.

[0063] The initial point set includes at least one point. When the initial point set includes multiple points, these points can be located on the same lane line or distributed across multiple lane lines. The initial point set can include points located on multiple lane lines, and for the same lane line, these points can be continuously distributed along that lane line or discretely distributed along that lane line.

[0064] Since each lane line is represented by points spaced at predetermined intervals along the longitudinal direction, when the initial point set includes multiple points, the determined initial point set can be used to represent the initial lane lines. Accordingly, step S102 can be understood as: processing the image to obtain the initial lane lines in the image. Specifically, a pre-trained semantic segmentation model can be used to process the image to obtain the initial lane lines.

[0065] When training a semantic segmentation model, an initial semantic segmentation model can be trained using training images to obtain a trained model. The training images include labels indicating whether each pixel or the cell containing each pixel belongs to a lane line. It should be noted that any blank area between any two adjacent dashed line segments in the training image is labeled as a lane line, so that the initial lane lines obtained by the semantic segmentation model include the blank areas between any two adjacent dashed line segments within the dashed lane lines.

[0066] Here, when labeling data, the blank areas between dashed lane lines are directly labeled as lane lines. Therefore, during model prediction, these blank areas will also be predicted as lane line pixels. This application uses a semantic segmentation model to accurately connect the dashed lane lines, achieving high efficiency and accuracy. If the blank areas between dashed lane lines are not labeled as lane lines, after generating the dashed lane lines, a lane line completion method is needed, significantly increasing computational complexity and reducing the accuracy of lane line completion.

[0067] In addition, the positional attributes of each lane line in the image can be marked during lane line labeling. For example, the lane line can be designated as the left lane line, middle lane line, or right lane line, or as the first lane line, second lane line, ..., nth lane line; or as left lane 1, left lane 2, right lane 1, right lane 2, etc. Those skilled in the art can set the specific labeling method as needed, and this application does not impose any restrictions on this. In this way, when predicting the initial lane lines, the positional attributes of each lane line can be obtained, making it easier to connect and complete lane lines with the same positional attributes later. It can also be used to judge and verify the lane line connection results. The specific training process is well known to those skilled in the art and will not be described here.

[0068] Step S103: Extract the positional features of at least one point in the initial point set.

[0069] After obtaining the initial set of points located on the lane lines in the image, the positional features of at least one point in the initial set are extracted so that the positional features of the extracted at least one point can be processed by a pre-trained first network model to obtain the trend information of the corresponding lane lines.

[0070] If the initial point set includes points located on multiple lane lines, then in subsequent lane line prediction, multiple lane lines can be predicted simultaneously in parallel, such as extracting the position features of at least one point from the initial point set located on each lane line at the same time; or the prediction can be performed in a serial manner, such as extracting the position features of at least one point from the initial point set located on a certain lane line for prediction, generating the complete line of that lane line, and then extracting the position features of at least one point from the initial point set located on the next lane line for prediction, and so on.

[0071] Furthermore, since each lane line is represented by points spaced at predetermined intervals along the longitudinal direction, when the initial point set includes multiple points, the determined initial point set can be used to represent the initial lane line. Accordingly, this step can also be understood as: extracting the positional features of at least one point from the initial lane line. Further, it can be understood as: identifying the initial lane lines in the image, determining the initial point set from the initial lane lines, and extracting the positional features of at least one point from the initial point set.

[0072] If the initial lane line is already a complete lane line (e.g., a lane line that reaches the image's four edge lines), then no completion is needed. Therefore, to save time, after obtaining the initial lane line, the lane line to be completed can be identified from the initial lane line, and then the positional features of at least one point can be extracted from the lane line to be completed. Here, the lane line to be completed can be identified by determining whether the initial lane line reaches the image's edge lines. For example, if it simultaneously reaches the upper and lower edge lines of the image, or simultaneously reaches the upper left edge line of the image, it can be considered a complete lane line and does not need to be completed. If one end of the initial lane line does not reach the image's edge line, then it can be considered a lane line to be completed. In this case, the initial point set can include the two endpoints of the initial lane line or the lane line to be completed, but it is not limited to this.

[0073] Considering that each initial lane line or the lane line to be completed may have multiple points at the same level, this application, after obtaining the initial lane line or the lane line to be completed, can first transform the thick lane line into a thin lane line, that is, to reduce the points at the same horizontal level in the lane line, such as retaining only the points on the same axis of the lane line, for example, the points on the center line of the lane line. Alternatively, the coordinates of multiple points at the same horizontal level can be averaged to obtain a new point coordinate, and this new point can replace the multiple points at the same horizontal level. Of course, this application is not limited to specific implementation methods; all methods that can transform a thick lane line into a thin lane line are within the scope of protection of this application. Then, processing can be performed based on the center point of the thin line, such as determining the initial point set, extracting features of at least one point, and correcting the coordinates predicted by the model based on the coordinates of the thin line's center point. If there are multiple lane lines to be completed, the lane line with the longest distance can be selected first, and the positional features of at least one point in the lane line with the longest distance can be extracted, and the completion process can begin with the lane line with the longest distance. Optionally, after generating the complete lane line containing the longest lane line to be filled, the longest lane line can be extracted from the remaining lane lines to be filled to start filling, which can improve the accuracy of prediction.

[0074] The term "to-be-completed full lane line" refers to lane lines whose endpoints on both sides do not reach the image critical line. If even one endpoint fails to reach the image critical line, it is considered a lane line to be completed. The target lane line is the complete lane line, including lane lines whose endpoints on both sides reach the image critical line. The image critical line includes at least one of an upper critical line, a lower critical line, a left critical line, and a right critical line. The upper critical line is at a first predetermined distance from the upper edge of the image, the lower critical line is at a second predetermined distance from the lower edge of the image, the left critical line is at a third predetermined distance from the upper edge of the image, the right critical line is at a second predetermined distance from the right edge of the image, and so on. These four predetermined distances can be the same or different values, and those skilled in the art can set them according to experience and needs; this application imposes such limitations. Preferably, considering that the upper part of the image is generally sky or ocean, the first predetermined distance can be the length ratio of the sky or ocean in the image. In this case, the upper critical line can be considered as the lower edge of the sky or ocean in the image. Of course, all four predetermined distances can also be 0, for example, the lower left and right critical lines are the lower left and right edges of the image, respectively. The positional features of each point include its position coordinates and offset features. The offset features are high-dimensional vectors representing the positional relationship between that point and the next point. The offset features of each point can be extracted from the intermediate layers of a pre-trained second network model based on the position coordinates of each point. This second network model can be a semantic segmentation model or a neural network model specifically designed for extracting offset features.

[0075] The second network model is a pre-trained network model. Its training process can be as follows: obtain a second training set, which includes multiple labeled training images; train the initial second network model using the second training set to obtain the trained model, i.e., the second network model. It should be noted that the training images in the second training set are labeled with multiple lane lines and the coordinate offsets from each point in each lane line to the next point. The specific training process is well-known to those skilled in the art and will not be described in detail here.

[0076] In this embodiment of the application, the offset features of each point are extracted by using a second network model trained to extract offset features that characterize the positional relationship between the point and the next point, thereby improving the detection accuracy.

[0077] Step S104: Use a pre-trained first network model to process the position features of the at least one point to obtain the trend information of the corresponding lane line, and generate a target lane line containing the initial point set based on the trend information.

[0078] After extracting the positional features of at least one point, a pre-trained first network model is used to process the positional features of the at least one point to obtain the trend information of the corresponding lane lines. Based on the trend information, target lane lines containing an initial set of points are generated. There can be one or more target lane lines, and the initial set of points is distributed on these target lane lines. Each target lane line is a complete lane line or a lane line predicted to reach a preset stopping position. That is, after acquiring a road image, the lane lines in the image itself may not be fully displayed due to various reasons such as dirt wear, shadow coverage, pixel blur, or occlusion by objects. After using the lane line detection method of this application, the final displayed lane lines are generally complete lane lines that reach the edge of the image, improving the accuracy of lane line recognition. Based on the distance between these lane lines and the road boundary lines, more accurate positioning can be achieved.

[0079] In this embodiment, when processing the positional features of each point, the first network model receives the following inputs: the current point's position coordinates, offset features, and the trend information learned in the previous iteration. The output of the first network model is the coordinate offset from the current point to the next point, and the trend information learned in the current iteration. The trend information is a high-dimensional vector, and the trend information input to the first network model during the first prediction is a preset initial value. In this embodiment, in addition to inputting the current point's position coordinates and offset features into the network model, the trend information learned in the previous iteration is also input. By continuously learning the trend information between points, the model can quickly and accurately complete the lane lines, obtaining a complete lane line.

[0080] In one implementation, step S104 may be as follows: using the first network model to process the position coordinates, offset features, and trend information learned by the first network model in the previous iteration, outputting the coordinate offset from the current point to the next point, and the trend information learned in this iteration; determining the position coordinates of the next point based on the coordinate offset and the position coordinates of the current point; using the offset features, position coordinates, and trend information learned by the first network model in this iteration as input to the model, outputting the offset coordinates from the next point to the point after that, and the latest trend information, and so on, until a preset stopping position is reached.

[0081] To facilitate understanding, an example is provided. In the first prediction, the coordinates of the starting point (the first point), its offset features, and the preset initial trend information are input into the first network model. The output is the coordinate offset from the current point (the starting point) to the next point, along with the learned trend information. In the second prediction, based on the coordinate offset obtained from the first prediction and the coordinates of the first point, the coordinates of the next point (the second point) are obtained. Then, the coordinates of the next point (the second point), its offset features, and the learned trend information are input into the first network model. The output is the offset coordinates from the second point (the second point) to the next point after that (the third point), along with the latest trend information. In the third prediction, based on the coordinate offset obtained from the second prediction and the coordinates of the second point, the coordinates of the second point are obtained. Then, the coordinates of the third point, its offset features, and the learned trend information are input into the first network model. The output is the offset coordinates from the third point to the fourth point, along with the latest trend information. This process continues until a preset stopping position is reached. It should be noted that the offset features of each point can be extracted from the intermediate layers of the pre-trained second network model based on the coordinates of each point.

[0082] The process of determining the preset stop position includes: when the predetermined number of iterations, such as 100 iterations, or when the lane line to be filled reaches the image critical line, the preset stop position is determined.

[0083] In one implementation, the process of determining the position coordinates of the next point based on the coordinate offset and the position coordinates of the current point can be as follows: if the horizontal distance between the position coordinates of the next point and a certain point at the same height in the initial point set is less than or equal to a predetermined pixel (e.g., 1 pixel), then the position coordinates of the next point are corrected based on that same height point. By correcting the position coordinates of the next point, errors can be reduced, thereby improving the accuracy of detection.

[0084] Preferably, if an initial lane line has been identified, when predicting the lane line trend information based on points in the initial lane line, the process of determining the position coordinates of the next point based on the coordinate offset and the current point's position coordinates can be as follows: if the horizontal distance between the next point's position coordinates and the initial lane line is less than or equal to a predetermined pixel (e.g., 1 pixel), then the position coordinates of the next point are corrected based on the same height point on the initial lane line. Further, if a complete lane line to be supplemented is identified from the initial lane line, when predicting the lane line trend information based on points in the complete lane line to be supplemented, the process of determining the position coordinates of the next point based on the coordinate offset and the current point's position coordinates can be as follows: if the horizontal distance between the next point's position coordinates and the complete lane line to be supplemented is less than or equal to a predetermined pixel (e.g., 1 pixel), then the position coordinates of the next point are corrected based on the same height point on the complete lane line to be supplemented.

[0085] In one implementation, the image can be divided into m*n image grids. In this case, the coordinate offset from the current point to the next point is the coordinate offset from the center point of the current point's grid to the midpoint of the lane line in the next row of grids. By dividing the image into m*n image grids, the position coordinates of each point and the coordinate offset from each point to the next point can be determined more quickly.

[0086] In one optional implementation, the input position coordinates of the first network model are normalized coordinates, which can reduce prediction errors. Of course, the position coordinates of the first network model can also be standardized based on other upper and lower limits. Those skilled in the art can set the maximum value of standardization as needed, and this application does not limit this.

[0087] In one implementation, to improve the accuracy of lane line prediction, step S104 can be as follows: A pre-trained first network model is used to process the positional features of at least one point to obtain the trend information of the corresponding lane line along a first direction. Multiple points are generated based on the trend information in the first direction until a preset stopping position is reached in the first direction. Starting from the preset stopping position in the first direction, the first network model is used to predict the trend information of the corresponding lane line along a second direction. Multiple points are generated based on the trend information in the second direction until a preset stopping position is reached in the second direction. In this implementation, firstly, along the first direction, the positional features of at least one point are processed to obtain the trend information of the corresponding lane line along the first direction. Multiple points are generated based on the trend information in the first direction until a preset stopping position is reached in the first direction. Then, along the opposite direction, the trend information of the lane line along the second direction is predicted, and multiple points are generated based on the trend information in the second direction until a preset stopping position is reached in the second direction. By predicting the same lane line from both directions, since more feature points are input into the first network model, the model can learn more accurate trend information, thereby improving the detection accuracy.

[0088] The process of predicting the trend information of lane lines in each direction is the same as the process described above. For example, the first network model is used to process the position coordinates, offset features, and trend information learned by the first network model in the previous time. The output is the coordinate offset from the current point to the next point, as well as the trend information learned in this time. The position coordinates of the next point are determined based on the coordinate offset and the position coordinates of the current point. The offset features, position coordinates, and trend information learned by the first network model in this time are used as the input of the model. The output is the offset coordinates from the next point to the next point after that, as well as the latest trend information. This process continues until the preset stopping position is reached.

[0089] For ease of understanding, Figure 3 Using the illustrated diagram as an example, assuming the initial point set is located in line segment 1 of the diagram, the lane line's trajectory can be predicted along either the first or second direction. For instance, predicting the lane line's trajectory in the first direction generates multiple points based on this trajectory information, continuing until a preset stop position is reached in the first direction. Then, predicting the lane line's trajectory in the opposite direction, i.e., the second direction, generates multiple points based on this trajectory information, continuing until a preset stop position is reached in the second direction. It should be noted that... Figure 3 The first and second directions can be interchanged; that is, the upward direction can be the second direction, and the downward direction can be the first direction.

[0090] In one optional implementation, the process of processing the positional features of at least one point using a pre-trained first network model to obtain the trend information of the corresponding lane line can be as follows: The pre-trained first network model processes the positional features of at least one point in the initial point set along a first direction to obtain the trend information of the corresponding lane line along the first direction; the pre-trained first network model then processes the positional features of at least one point in the initial point set along a second direction to obtain the trend information of the corresponding lane line along the second direction. By simultaneously processing the positional features of at least one point along two different directions, the trend information of the corresponding lane line along two directions is obtained, thereby enabling the completion of the lane line.

[0091] For ease of understanding, Figure 3 Using the diagram shown as an example, assuming the initial point set is located in line segment 1 of the diagram, the same lane line can be predicted along both the first and second directions. For example, starting from point A, the trend of the lane line in the first direction can be predicted along the first direction; and starting from point B, the trend of the lane line in the second direction can be predicted along the second direction. This allows for more initial input points, thereby improving prediction accuracy.

[0092] If the initial point set includes multiple points, it can be represented as an initial lane line. If the initial lane line is a lane line to be filled, the process of using a pre-trained first network model to process the positional features of at least one point to obtain the trend information of the corresponding lane line can be understood as follows: using a pre-trained first network model to process the positional features of multiple points of the lane line to be filled in the first direction to obtain the trend information of the lane line to be filled in the first direction; using a pre-trained first network model to process the positional features of multiple points of the lane line to be filled in the second direction to obtain the trend information of the lane line to be filled in the second direction; wherein, there is a common point among the multiple points of the lane line to be filled in the first direction and the multiple points in the second direction.

[0093] Furthermore, if there are multiple lane lines to be filled, the longest lane line can be selected for completion. After completing the longest lane line, lane lines can be selected from the remaining lane lines to be filled. This can be done by filling from the first direction to a preset stop position, and then starting from the preset stop position along the second direction to fill to the preset stop position in the second direction; alternatively, the longest lane line can be filled simultaneously from both the first and second directions.

[0094] For example, the positional features of at least one point extracted from the longest lane line to be filled include a first positional feature at the first endpoint and a second positional feature at the second endpoint. The process of simultaneously completing the lane line from the first direction and the second direction for the longest lane line to be filled can be: processing the first positional features using a pre-trained first network model to predict the trend information of the lane line to be filled from the first endpoint to the second endpoint; and processing the second positional features using a pre-trained first network model to predict the trend information of the lane line to be filled from the second endpoint to the first endpoint.

[0095] by Figure 3 Taking the illustrated diagram as an example, assume the initial lane line includes two parts: line segment 1 and line segment 2. Since both line segments 1 and 2 are lane lines to be completed, and line segment 1 is longer than line segment 2, line segment 1 can be selected for lane line completion. For example, multiple points are generated along the first direction to obtain line segment 3, and multiple points are generated along the second direction to obtain line segment 4, thus completing the lane line. In one implementation, the first network model includes a first network branch and a second network branch. The first network branch is used to predict the trend information of the corresponding lane line from the first direction; the second network branch is used to predict the trend information of the corresponding lane line from the second direction; wherein the first direction and the second direction are opposite. In this implementation, when predicting the trend information of the lane line along the first direction, the first network branch is used for prediction; when predicting the trend information of the lane line along the second direction, the second network branch is used for prediction. By using different network branches to predict the trend information in different directions, the accuracy of lane line detection can be improved. At the same time, it allows two network branches to be used simultaneously to complete the lane line from different directions, which can improve the speed of obtaining the complete lane line. The first network model is a pre-trained trend prediction model, primarily used to predict lane line trends. It can be a recurrent neural network (RNN) model, such as one where parameters are shared at each step. Normalization of the position coordinates prevents large differences in the magnitude of data involved in the calculation, which could affect training accuracy. The RNN model can be a Long Short-Term Memory (LSTM) network model or a Gate Recurrent Unit (GRU) network model, etc.

[0096] The training process can be as follows: obtaining the coordinate sequences of multiple lane lines; extracting the coordinate offsets and offset features from each point in each lane line to the next point to generate a first training set; and training the initial first network model using the first training set to obtain the trained model. The initial first network model is trained using the extracted coordinate offsets and offset features from each point in each lane line to the next point, enabling the first network model to learn the positional relationships between each point in the lane line and the next point, thereby learning the trend information of the lane lines. This allows for rapid detection of lane lines based on the network model. The specific training process is well-known to those skilled in the art and will not be described in detail here.

[0097] To facilitate a comprehensive understanding of this solution, the following will combine... Figures 4-6 The schematic diagram shown illustrates an example of lane line detection using the lane line detection method described in the embodiments of this application.

[0098] Figure 4 This is a schematic diagram of a road image from which the initial lane lines identified are shown. Figure 5 As shown in the diagram. The leftmost lane line is obscured by a vehicle, so the initial displayed lane line only shows the unobscured portion. The middle lane line is a dashed line; although it reaches the upper boundary of the image, it has not reached the lower boundary, so it needs to be completed. Furthermore, the blank areas between the dashed lines are also identified as initial lane lines, improving the efficiency of subsequent lane line completion. The rightmost lane line is a solid line; this solid line has already reached both the upper and lower boundary of the image, therefore it is a complete lane line and does not require completion. After applying the lane line detection method shown in this embodiment, the lane line detection... Figure 5 After completing the lane lines shown, we get... Figure 6 The results shown depict each lane line as a complete lane line, improving the user experience and facilitating subsequent algorithms such as vehicle positioning based on these complete lane lines.

[0099] Based on the same inventive concept, this application also provides a lane line detection device 200, such as... Figure 7 As shown. The lane line detection device 200 includes: an acquisition module 210 and a processing module 220.

[0100] The acquisition module 210 is used to acquire an image including lane lines.

[0101] Processing module 220 is configured to determine an initial set of points in the image, the initial set of points being located on lane lines in the image; extract positional features of at least one point in the initial set of points; process the positional features of the at least one point using a pre-trained first network model to obtain trend information of the corresponding lane lines; and generate a target lane line containing the initial set of points based on the trend information. The target lane line includes lane lines whose endpoints on both sides reach the image boundary line.

[0102] Each lane line is represented by points spaced at predetermined intervals along the longitudinal direction. The positional features of each point include its position coordinates and offset features. The offset features are high-dimensional vectors that characterize the positional relationship between the point and the next point, and are extracted from the intermediate layer of a pre-trained second network model based on the position coordinates of each point.

[0103] When the initial set of points includes multiple points, the multiple points are located on the same lane line, or the multiple points are distributed on multiple lane lines.

[0104] When the first network model processes the positional features of each point: the input of the first network model is the position coordinates of the current point, the offset features, and the trend information learned by the first network model in the previous time; the output of the first network model is the coordinate offset from the current point to the next point, and the trend information learned in this time.

[0105] The trend information is a high-dimensional vector, and the trend information input to the first network model during the first prediction is a preset initial value.

[0106] In one implementation, the processing module 220 is specifically configured to: process the position coordinates and offset features of the current point using the first network model, as well as the trend information learned by the first network model in the previous iteration, and output the coordinate offset from the current point to the next point, and the trend information learned in the current iteration; determine the position coordinates of the next point based on the coordinate offset and the position coordinates of the current point; use the offset features and position coordinates of the next point, as well as the trend information learned by the first network model in the current iteration, as input to the model, and output the offset coordinates from the next point to the point after that, and the latest trend information, and so on, until a preset stopping position is reached. The preset stopping position includes the position when a predetermined number of iterations is reached or an image critical line. The image critical line includes at least one of an upper critical line, a lower critical line, a left critical line, and a right critical line.

[0107] In one implementation, the processing module 220 is specifically used to correct the position coordinates of the next point based on the same height point if the horizontal distance between the position coordinates of the next point and a certain height point in the initial point set is less than or equal to a predetermined number of pixels.

[0108] In one implementation, the first network model is a recurrent neural network model with shared parameters at each step, and the position coordinates input to the first network model are normalized coordinates.

[0109] In one implementation, the image is divided into m*n image grids, and the coordinate offset from the current point to the next point is the coordinate offset from the center point of the grid where the current point is located to the midpoint of the lane line in the next row of grids.

[0110] In one implementation, the first network model includes a first network branch and a second network branch; the first network branch is used to predict the trend information of the corresponding lane line from a first direction; the second network branch is used to predict the trend information of the corresponding lane line from a second direction; wherein the first direction and the second direction are opposite.

[0111] In one implementation, the processing module 220 is specifically used to: process the positional features of the at least one point using a pre-trained first network model to obtain the trend information of the corresponding lane line along a first direction, and generate multiple points based on the trend information in the first direction until a preset stopping position is reached in the first direction; starting from the preset stopping position in the first direction, use the first network model to predict the trend information of the corresponding lane line along a second direction, and generate multiple points based on the trend information in the second direction until a preset stopping position is reached in the second direction, wherein the first direction and the second direction are opposite.

[0112] In one implementation, the processing module 220 is specifically used to: process the positional features of at least one point in the initial point set in a first direction using a pre-trained first network model to obtain the trend information of the corresponding lane line along the first direction; and process the positional features of at least one point in the initial point set in a second direction using a pre-trained second network model to obtain the trend information of the corresponding lane line along the second direction, wherein the first direction and the second direction are opposite.

[0113] Optionally, the acquisition module 210 is further configured to acquire the coordinate sequence of multiple lane lines; the processing module 220 is further configured to: extract the coordinate offset and offset features of each point in each lane line to the next point, generate a first training set; and use the first training set to train the initial first network model to obtain the trained model.

[0114] Optionally, the acquisition module 210 is further configured to acquire a second training set, wherein the training images in the second training set are labeled with multiple lane lines and the coordinate offset of each point in each lane line to the next point; the processing module 220 is further configured to: train the initial second network model using the second training set to obtain the trained model.

[0115] The lane line detection device 200 provided in this application embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0116] Based on the same inventive concept, such as Figure 8 As shown in the figure, this application embodiment also provides an electronic device 300 for performing the above-described lane line detection method. The electronic device 300 includes: a communication interface 310, a memory 320, a communication bus 330, and a processor 340.

[0117] The communication interface 310, the memory 320, and the processor 340 are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 330 or signal lines. The communication interface 310 is used to implement one or more communication protocols (LTE, Wi-Fi, etc.). The memory 320 is used to store computer programs, such as... Figure 7 The software functional module shown is the lane line detection device 200. The lane line detection device 200 includes at least one software functional module that can be stored in the memory 320 in the form of software or firmware, or embedded in the operating system (OS) of the electronic device 300. The processor 340 is used to execute executable modules stored in the memory 320, such as the software functional module or computer program included in the lane line detection device 200. For example, the processor 340 is used to acquire an image to be detected; determine an initial set of points in the image, the initial set of points being on the lane lines of the image; extract positional features of at least one point in the initial set of points; process the positional features of the at least one point using a pre-trained first network model to obtain the trend information of the corresponding lane line; and generate a target lane line containing the initial set of points based on the trend information.

[0118] The memory 320 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0119] Processor 340 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor, or processor 340 can be any conventional processor.

[0120] Among them, the aforementioned electronic devices 300 include, but are not limited to, computers, servers, and vehicle-mounted computers.

[0121] This application embodiment also provides a non-volatile computer-readable storage medium (hereinafter referred to as the storage medium) storing a computer program, which is executed by a computer such as the electronic device 300 described above to perform the lane line detection method described above.

[0122] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0124] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0125] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, laptop, server, or electronic device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A lane line detection method characterized by, include: Acquire the image to be detected; Determine an initial set of points in the image, the initial set of points being located on the lane lines of the image; Extract the positional features of at least one point in the initial point set; The positional features of the at least one point are processed using a pre-trained first network model to obtain the trend information of the corresponding lane lines; as well as Generate a target lane line containing the initial point set based on the trend information; The first network model includes a first network branch and a second network branch; The first network branch is used to predict the trend information of the corresponding lane line from the first direction; The second network branch is used to predict the trend information of the corresponding lane line from the second direction; The first direction is opposite to the second direction.

2. The method according to claim 1, characterized in that, Each lane line is represented by points spaced longitudinally at predetermined intervals, wherein the positional features of each point include the position coordinates and offset features of that point; The offset feature is a high-dimensional vector representing the positional relationship between the point and the next point, and is extracted from the intermediate layer of the pre-trained second network model based on the position coordinates of each point.

3. The method of claim 2, wherein, When the first network model processes the positional features of each point: The input to the first network model is the current point's position coordinates, offset features, and the trend information learned by the first network model in the previous iteration; The output of the first network model is the coordinate offset from the current point to the next point and the trend information learned in this study.

4. The method of claim 3, wherein, The trend information is a high-dimensional vector, and the trend information input to the first network model during the first prediction is a preset initial value.

5. The method of claim 2, wherein, The location features of at least one point are processed using a pre-trained first network model to obtain the trend information of the corresponding lane line, and a target lane line containing the initial point set is generated based on the trend information, including: The first network model is used to process the position coordinates and offset features of the current point, as well as the trend information learned by the first network model in the previous time, and outputs the coordinate offset from the current point to the next point, and the trend information learned in this time. The position coordinates of the next point are determined based on the coordinate offset and the current point's position coordinates. The offset features and position coordinates of the next point, as well as the trend information learned by the first network model, are used as inputs to the model. The output is the offset coordinates from the next point to the next point after that, as well as the latest trend information. This process continues until a preset stopping position is reached.

6. The method of claim 5, wherein, Determining the position coordinates of the next point based on the coordinate offset and the current point's position coordinates includes: If the horizontal distance between the position coordinates of the next point and a certain point at the same height in the initial point set is less than or equal to a predetermined number of pixels, then the position coordinates of the next point are corrected based on that point at the same height.

7. The method of claim 2, wherein, The first network model is a recurrent neural network model with shared parameters at each step, and the position coordinates input to the first network model are normalized coordinates.

8. The method of claim 3, wherein, The image is divided into m*n image grids, and the coordinate offset from the current point to the next point is the coordinate offset from the center point of the grid where the current point is located to the midpoint of the lane line in the next row of grids.

9. The method according to claim 5, characterized in that, The target lane line includes lane lines whose endpoints on both sides of the line segment reach the image boundary line; The preset stop position includes the position when a predetermined number of iterations is reached or the image critical line; The image critical line includes at least one of the following: upper critical line, lower critical line, left critical line, and right critical line.

10. The method of claim 1, wherein, The location features of at least one point are processed using a pre-trained first network model to obtain the trend information of the corresponding lane line, and a target lane line containing the initial point set is generated based on the trend information, including: The positional features of the at least one point are processed using a pre-trained first network model to obtain the trend information of the corresponding lane line along the first direction, and multiple points are generated based on the trend information in the first direction until a preset stopping position in the first direction is reached. Starting from a preset stop position in the first direction, the first network model is used to predict the trend information of the corresponding lane line along the second direction, and multiple points are generated based on the trend information in the second direction until the preset stop position in the second direction is reached, wherein the first direction and the second direction are opposite.

11. The method of claim 1, wherein, The positional features of the at least one point are processed using a pre-trained first network model to obtain the trend information of the corresponding lane lines, including: The positional features of at least one point in the initial point set in the first direction are processed using a pre-trained first network model to obtain the trend information of the corresponding lane line along the first direction. A pre-trained first network model is used to process the positional features of at least one point in the initial point set in the second direction to obtain the trend information of the corresponding lane line along the second direction, wherein the first direction is opposite to the second direction.

12. The method of claim 1, wherein, The method further includes: Obtain the coordinate sequence of multiple lane lines; Extract the coordinate offset and offset features from each point in each lane line to the next point to generate the first training set; and The initial first network model is trained using the first training set to obtain the trained model.

13. The method of claim 2, wherein, The method further includes: Obtain a second training set, in which the training images are labeled with multiple lane lines and the coordinate offset of each point in each lane line to the next point; The initial second network model is trained using the second training set to obtain the trained model.

14. The method of claim 1, wherein, When the initial set of points includes multiple points, the multiple points are located on the same lane line, or the multiple points are distributed on multiple lane lines.

15. A lane line detection device characterized by comprising: include: The acquisition module is used to acquire images including lane lines; A processing module is used to determine an initial set of points in the image, the initial set of points being located on lane lines in the image; Extract the positional features of at least one point in the initial point set; The positional features of the at least one point are processed using a pre-trained first network model to obtain the trend information of the corresponding lane line, and a target lane line containing the initial point set is generated based on the trend information. The first network model includes a first network branch and a second network branch; The first network branch is used to predict the trend information of the corresponding lane line from the first direction; The second network branch is used to predict the trend information of the corresponding lane line from the second direction; The first direction is opposite to the second direction.

16. An electronic device, comprising: include: A memory and a processor, wherein the processor is connected to the memory; The memory is used to store programs; The processor is configured to invoke a program stored in the memory to execute the method as described in any one of claims 1-14.

17. A storage medium, characterized by It stores a computer program, which, when executed by a processor, performs the method as described in any one of claims 1-14.

Images