A lane line corner point detection method and device, electronic equipment and storage medium

Abstract

The application relates to a lane line corner point detection method and device, electronic equipment and a storage medium, and belongs to the technical field of automatic driving. The method comprises the following steps: acquiring a road image to be detected; inputting the road image into a pre-trained neural network model for processing to obtain a heat map prediction result corresponding to the road image, wherein the heat map prediction result comprises the probability of each pixel point belonging to a specific corner point category; determining a plurality of corner points of different categories in the road image according to the probability of each pixel belonging to the specific corner point category in the heat map prediction result; and grouping the plurality of corner points of different categories according to a predetermined rule to obtain a plurality of corner point groups. The heat map corresponding to the lane line in the road image is predicted by using the neural network model, so that the probability of each pixel belonging to the specific corner point category in the heat map prediction result can be quickly obtained, and then the plurality of corner point groups can be quickly obtained, thereby solving the problem that the lane line corner points cannot be accurately and quickly detected.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving, and particularly relates to a lane line corner point detection method and device, an electronic device and a storage medium. BACKGROUND

[0002] With the development of automobiles, more and more artificial intelligence technologies are applied to automatic driving technology, and lane line detection is an important link in automatic driving technology.

[0003] The current lane line detection algorithm can detect the edges of the lane line by using a Canny edge detection algorithm, then detect potential straight lines in the image by using a Hough straight line transformation, and then find the lane line by using some geometric restrictions of the lane. In addition, there are lane line detection methods based on deep learning, such as using a semantic segmentation method to identify the lane line. However, the current methods can only identify the lane line pixels in the image, and cannot accurately and quickly detect the lane line corner points. SUMMARY

[0004] In view of this, the purpose of the present application is to provide a lane line corner point detection method and device, an electronic device and a storage medium to solve the problem that the current lane line detection method cannot accurately and quickly detect the lane line corner points.

[0005] Embodiments of the present application are implemented as follows:

[0006] In a first aspect, the embodiments of the present application provide a lane line corner point detection method, comprising: acquiring a road image to be detected; inputting the road image into a pre-trained neural network model for processing to obtain a heat map prediction result corresponding to the road image, the heat map prediction result including probabilities of each pixel belonging to a specific corner point category; determining a plurality of different category corner points in the road image according to the probabilities of each pixel belonging to the specific corner point category in the heat map prediction result; and grouping the plurality of different category corner points according to a predetermined rule to obtain a plurality of corner point groups. In the embodiments of the present application, the heat map corresponding to the lane line in the road image is predicted by using the neural network model, so that the probabilities of each pixel belonging to the specific corner point category in the heat map prediction result can be quickly obtained, and then the plurality of different category corner points in the road image are determined according to the probabilities of each pixel belonging to the specific corner point category in the heat map prediction result, and the plurality of different category corner points are grouped, so that the plurality of corner point groups can be quickly obtained, thereby solving the problem that the current lane line detection method cannot accurately and quickly detect the lane line corner points.

[0007] With reference to a possible implementation of the first aspect, the neural network model comprises a first network branch and a second network branch, the first network branch is configured to output a semantic segmentation result of whether each pixel in the road image belongs to a lane line, and the second network branch is configured to output a heat map prediction result of the road image. In the embodiments of the present application, the neural network model comprising two network branches is adopted, so that only one network can output the semantic segmentation result of whether each pixel in the road image belongs to the lane line and the heat map prediction result of the road image, so as to remove the corner points in the heat map in the area without lane line based on the semantic segmentation result, thereby improving the accuracy of the corner point detection, and in addition, only one neural network model is involved, thereby saving the use cost.

[0008] With reference to a possible implementation of the first aspect, the predicted value of each pixel in the semantic segmentation result is 0 or 1, or the predicted value of each pixel in the semantic segmentation result is a probability value in the interval [0, 1]; and the predicted value of each pixel in the heat map prediction result is a probability value in the interval [0, 1].

[0009] With reference to a possible implementation of the first aspect, the method further comprises: multiplying the heat map prediction result and the predicted value of each pixel in the semantic segmentation result, so as to reduce the predicted probability value of the corner point in the area without lane line in the heat map, and obtain an updated heat map. In this way, the accuracy of the corner point detection can be improved.

[0010] With reference to a possible implementation of the first aspect, the multiplying the heat map prediction result and the predicted value of each pixel in the semantic segmentation result comprises: expanding the pixel range representing the lane line in the semantic segmentation result; and multiplying the heat map prediction result and the predicted value of each pixel in the semantic segmentation result after the range expansion. In the embodiments of the present application, the pixel range representing the lane line in the semantic segmentation result is expanded, and then the heat map prediction result is multiplied by the predicted value of each pixel in the semantic segmentation result after the range expansion, so as to prevent missing the corner points which are close to the lane line but have small predicted values in the semantic segmentation result in the corner point screening process.

[0011] With reference to a possible implementation of the first aspect, the heat map prediction result comprises four heat maps, each heat map corresponds to one corner point category, and comprises the probability that each pixel in the heat map belongs to the corresponding corner point category. In the embodiments of the present application, when predicting the corner point category to which each pixel in the heat map belongs by using the neural network model, the model classifies each corner point type, and directly outputs four heat maps of the corresponding four corner point categories after the classification, thereby improving the subsequent corner point matching accuracy.

[0012] In a possible implementation of the first aspect, the method further includes: determining the corner points of the different categories of the road image according to the probabilities of each pixel belonging to a specific corner point category in the heat maps, including: determining the corner points of the corresponding category of the heat map according to the probabilities of each pixel belonging to a specific corner point category in each heat map, and then obtaining the four categories of corner points contained in the four heat maps. In the embodiments of the present application, the corner points of the corresponding category of the heat map are determined according to the probabilities of each pixel belonging to a specific corner point category in the heat map, so that the four categories of corner points contained in the four heat maps can be quickly determined.

[0013] In a possible implementation of the first aspect, the method further includes: determining the corner points of the corresponding category of the heat map according to the probabilities of each pixel belonging to a specific corner point category in each heat map, including: performing a sliding window operation on the heat map by using a preset sliding window region, and if the probability value of the center point of the sliding window region meets a predetermined condition, marking the center point of the sliding window region as a corner point of the corresponding category. In the embodiments of the present application, the sliding window operation is performed on the heat map by using a preset sliding window region, and the center point of the sliding window region whose probability value meets a predetermined condition is marked as a corner point of the corresponding category. Through the sliding window operation, the speed of confirming the corner points can be accelerated, and at the same time, since the center point of the sliding window region whose probability value meets a predetermined condition is marked as a corner point, the detection accuracy of the corner points can also be improved.

[0014] In a possible implementation of the first aspect, the predetermined condition is that the probability value of the center point of the sliding window region is greater than or equal to a preset probability threshold and is a local maximum value of the sliding window region. In the embodiments of the present application, only when the probability value of the center point of the sliding window region is greater than or equal to a preset probability threshold and is a local maximum value of the sliding window region, the center point is marked as a corner point, so that the detection accuracy of the corner points is ensured.

[0015] In a possible implementation of the first aspect, the categories of the corner points include at least one of a left-up category, a right-up category, a left-down category, and a right-down category.

[0016] In one possible implementation of the first aspect embodiment, multiple corner points of different categories are grouped according to predetermined rules to obtain multiple corner point groups, including: matching the nearest upper left and upper right category corner points to obtain upper corner point pairs; matching the nearest lower left and lower right category corner points to obtain lower corner point pairs; and matching the nearest upper and lower corner point pairs to obtain multiple corner point groups. In this embodiment, when grouping corner points of different categories, upper corner point pairs are obtained by matching the nearest upper left and upper right category corner points, lower corner point pairs are obtained by matching the nearest lower left and lower right category corner points, and finally, the nearest upper and lower corner point pairs are matched. This allows for the rapid acquisition of multiple corner point groups, and because the matching is based on proximity, it ensures to a certain extent that each corner point group contains all four categories of corner points belonging to the same lane line.

[0017] In one possible implementation of the first aspect embodiment, multiple corner points of different categories are grouped according to predetermined rules to obtain multiple corner point groups, including: matching the nearest upper-left and lower-left category corner points to obtain a left corner point pair; matching the nearest upper-right and lower-right category corner points to obtain a right corner point pair; and matching the nearest left and right corner point pairs to obtain multiple corner point groups. In this embodiment, when grouping corner points of different categories, by matching the nearest upper-left and lower-left category corner points to obtain a left corner point pair, matching the nearest upper-right and lower-right category corner points to obtain a right corner point pair, and then matching the nearest left and right corner point pairs, multiple corner point groups can be quickly obtained. Furthermore, since the matching is based on proximity, it ensures to a certain extent that each corner point group contains all four categories of corner points belonging to the same lane line.

[0018] In one possible implementation of the first aspect embodiment, after obtaining multiple corner point groups, the method further includes: removing abnormal corner point groups from the multiple corner point groups. In this embodiment, by removing abnormal corner point groups from the multiple corner point groups and retaining only normal corner point groups, the accuracy of corner point detection is improved.

[0019] In one possible implementation of the first aspect embodiment, removing abnormal corner groups from multiple corner groups includes: removing corner groups where the difference in side length between two pairs of corner points is greater than a predetermined threshold. In this embodiment, by comparing the difference in side length between two pairs of corner points in a corner group with a predetermined threshold, abnormal corner groups can be quickly removed.

[0020] In a possible implementation of the first aspect, the method further includes: obtaining a plurality of training images containing dashed lane lines, the training images being labeled with corner points of the dashed lane lines and probabilities that each pixel near the corner points belongs to a specific corner point category; and training the initial neural network model using the training images to obtain the trained neural network model. In the embodiment, the initial neural network model is trained using the training images labeled with the corner points of the dashed lane lines and the probabilities that each pixel near the corner points belongs to a specific corner point category, so that a heat map capable of predicting the probabilities that each pixel in a lane line belongs to a specific corner point category can be obtained, thereby providing support for subsequent corner point detection.

[0021] In a possible implementation of the first aspect, the probabilities that each pixel near the corner points belongs to a specific corner point category are Gaussian distributed, and each corner point group constitutes a lane line instance. In the embodiment, when labeling, the probabilities that each pixel near the corner points belongs to a specific corner point category are labeled according to the Gaussian distribution feature, thereby ensuring the accuracy of subsequent corner point detection.

[0022] In a possible implementation of the first aspect, the probabilities that each pixel near the corner points belongs to a specific corner point category are Gaussian distributed, and each corner point group constitutes a lane line instance. In the embodiment, when labeling, the probabilities that each pixel near the corner points belongs to a specific corner point category are labeled according to the Gaussian distribution feature, thereby ensuring the accuracy of subsequent corner point detection.

[0023] In a possible implementation of the first aspect, the probabilities that each pixel near the corner points belongs to a specific corner point category are Gaussian distributed, and each corner point group constitutes a lane line instance. In the embodiment, when labeling, the probabilities that each pixel near the corner points belongs to a specific corner point category are labeled according to the Gaussian distribution feature, thereby ensuring the accuracy of subsequent corner point detection.

[0024] In a possible implementation of the first aspect, the probabilities that each pixel near the corner points belongs to a specific corner point category are Gaussian distributed, and each corner point group constitutes a lane line instance. In the embodiment, when labeling, the probabilities that each pixel near the corner points belongs to a specific corner point category are labeled according to the Gaussian distribution feature, thereby ensuring the accuracy of subsequent corner point detection.

[0025] Other features and advantages of the present application will be set forth in the descriptions that follow and in part will be apparent from the description or can be learned by practice of the application. The purposes and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and claims thereof. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor. The above and other objects, features and advantages of the present application will be more apparent through the drawings shown. The same reference numerals in all the drawings indicate the same parts. The drawings are not necessarily drawn to scale, and the emphasis is on illustrating the principles of the present application.

[0027] Figure 1 A schematic diagram of an architecture of a vehicle is shown.

[0028] Figure 2 A schematic diagram of a lane line corner point detection method is shown.

[0029] Figure 3 A schematic diagram of a lane line corner point detection method is shown.

[0030] Figure 4 A module block diagram of a lane line corner point detection device is shown.

[0031] Figure 5 A schematic diagram of a structure of an electronic device is shown. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application.

[0033] It should be noted that similar reference numerals and letters refer to similar items throughout the accompanying drawings, and once an item is defined in one drawing, that item does not need to be further defined and explained in subsequent drawings. Moreover, the use of the terms first, second, etc. do not imply any physical or logical relationship between the items or operations. Furthermore, the terms "comprise", "contain", and "include", or any variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0034] In addition, the term "and / or" in the present application is only used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent three cases of A alone, A and B together, and B alone.

[0035] In view of the problem that the existing lane line detection method cannot accurately and quickly detect lane line corner points, the embodiment of the present application provides a lane line corner point detection method, which predicts a heat map corresponding to lane lines in a road image by using a neural network model, so that the probability that each pixel in the heat map prediction result belongs to a specific corner point category can be quickly obtained, and then the probabilities that each pixel in the heat map prediction result belongs to a specific corner point category are used to determine a plurality of corner points of different categories in the road image, and then the plurality of corner points of different categories are grouped, so that a plurality of corner point groups can be obtained, thereby solving the problem that the existing lane line detection method cannot accurately and quickly detect lane line corner points.

[0036] The lane line corner point detection method provided by the embodiment of the present application can be applied to a vehicle, and the following will be combined with the Figure 1 The vehicle related to the present application is described. Among them, Figure 1is a schematic diagram of a vehicle 100 in which various techniques disclosed herein can be implemented. The vehicle 100 can be a car, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, a lawnmower, an excavator, a snowmobile, an aircraft, a recreational vehicle, an amusement park vehicle, a farm device, a construction device, a tram, a golf cart, a train, a trolley, or other vehicle. The vehicle 100 can operate entirely or partially in an autonomous driving mode. The vehicle 100 in the autonomous driving mode can control itself, e.g., the vehicle 100 can determine a current state of the vehicle and a current state of an environment in which the vehicle is located, determine a predicted behavior of at least one other vehicle in the environment, determine a trust level corresponding to a likelihood that the at least one other vehicle performs the predicted behavior, and control the vehicle 100 itself based on the determined information. While in the autonomous driving mode, the vehicle 100 can operate without human interaction.

[0037] The vehicle 100 can include various vehicle systems, such as a drive system 142, a sensor system 144, a control system 146, a user interface system 148, a computing system 150, and a communication system 152. The vehicle 100 can include more or fewer systems, and each system can include multiple units. Further, each system and unit of the vehicle 100 can be interconnected. For example, the computing system 150 can be in data communication with one or more of the drive system 142, the sensor system 144, the control system 146, the user interface system 148, and the communication system 152. Thus, one or more of the described functionalities of the vehicle 100 can be divided among additional functional components or physical components, or combined into fewer functional components or physical components. In still further examples, additional functional components or physical components can be added to the example shown. Figure 1 The drive system 142 can include multiple operable components (or units) that provide motive power for the vehicle 100. In one embodiment, the drive system 142 can include an engine or motor, wheels, a transmission, electronics, and a power source (or power source). The engine or motor can be any combination of an internal combustion engine, an electric motor, a steam engine, a fuel cell engine, a propane engine, or other form of engine or motor. In some embodiments, the engine can convert a power source into mechanical energy. In some embodiments, the drive system 142 can include multiple engines or motors. For example, a hybrid vehicle can include a gasoline engine and an electric motor, among other possibilities.

[0038] The wheels of the vehicle 100 can be standard wheels. The wheels of the vehicle 100 can be in a variety of forms, including unicycle, bicycle, tricycle, or four-wheel forms, such as those found on cars or trucks. Other numbers of wheels are possible, such as six or more wheels. One or more of the wheels of the vehicle 100 can be operated to rotate in a direction different from the other wheels. The wheels can be at least one wheel that is fixedly connected to the transmission. The wheels can include a combination of metal and rubber, or other combinations of materials. The transmission can include units that are operable to transmit mechanical power from the engine to the wheels. For this purpose, the transmission can include a gear box, a clutch, a differential gear, and a drive shaft. The transmission can also include other units. The drive shaft can include one or more axles that are adapted to the wheels. The electronic system can include units for transmitting or controlling electronic signals of the vehicle 100. These electronic signals can be used to activate a variety of lights, servos, electric motors, and other electronic driving or control devices in the vehicle 100. The power source can be an energy source that provides power, in whole or in part, to the engine or electric motor. That is, the engine or electric motor is capable of converting the power source into mechanical energy. Illustratively, the power source can include gasoline, petroleum, petroleum-based fuels, propane, other compressed gas fuels, ethanol, fuel cells, solar panels, batteries, and other sources of electrical energy. The power source can additionally or alternatively include any combination of fuel tanks, batteries, capacitors, or flywheels. The power source can also provide energy to other systems of the vehicle 100.

[0039] The sensor system 144 can include a plurality of sensors for sensing information about the environment and conditions of the vehicle 100. For example, the sensor system 144 can include an inertial measurement unit (IMU), a GNSS (global navigation satellite system) transceiver (e.g., a global positioning system (GPS) transceiver), a RADAR, a laser rangefinder / LIDAR (or other distance measuring device), an acoustic sensor, an ultrasonic sensor, and a camera or image capture device. The sensor system 144 can include a plurality of sensors for monitoring the vehicle 100 (e.g., an oxygen (O2) monitor, an oil gauge sensor, an engine oil pressure sensor, and temperature, humidity, pressure sensors, etc.). Other sensors can also be configured. One or more of the sensors included in the sensor system 144 can be individually or collectively actuated to update the position, orientation, or both of the one or more sensors.

[0040] The IMU can include a combination of sensors (e.g., accelerometers and gyroscopes) for sensing changes in position and orientation of the vehicle 100 based on inertial acceleration. The GPS transceiver can be any sensor for estimating the geographical position of the vehicle 100. For this purpose, the GPS transceiver can include a receiver / transmitter to provide position information of the vehicle 100 relative to the earth. It is noted that GPS is an example of a global navigation satellite system, and thus, in some embodiments, the GPS transceiver can be replaced by a Beidou satellite navigation system transceiver or a Galileo satellite navigation system transceiver. The radar unit can use radio signals to sense objects in the environment in which the vehicle 100 is located. In some embodiments, in addition to sensing objects, the radar unit can also be used to sense the speed and heading of objects approaching the vehicle 100. The laser rangefinder or LIDAR unit (or other distance measuring device) can be any sensor that uses laser light to sense objects in the environment in which the vehicle 100 is located. In one embodiment, the laser rangefinder / LIDAR unit can include a laser source, a laser scanner, and a detector. The laser rangefinder / LIDAR unit can be operated in a continuous (e.g., using heterodyne detection) or discontinuous detection mode. The camera can include a device for capturing a plurality of images of the environment in which the vehicle 100 is located. The camera can be a still image camera or a dynamic video camera.

[0041] The control system 146 is configured to control the operation of the vehicle 100 and its components (or units). Accordingly, the control system 146 can include various units, such as a steering unit, a power control unit, a braking unit, and a navigation unit.

[0042] The steering unit can be a combination of mechanisms that adjust the heading of the vehicle 100. The power control unit (e.g., which can be a throttle) can be used to control the speed of the engine, and thus the speed of the vehicle 100. The braking unit can include a combination of mechanisms for decelerating the vehicle 100. The braking unit can utilize friction in a standard manner to decelerate the vehicle. In other embodiments, the braking unit can convert the kinetic energy of the wheels into electrical current. The braking unit can also take other forms. The navigation unit can be any system that determines a driving path or route for the vehicle 100. The navigation unit can also dynamically update the driving path as the vehicle 100 travels. The control system 146 can additionally or alternatively include other components (or units) that are not shown or described.

[0043] User interface system 148 can be used to allow vehicle 100 to interact with external sensors, other vehicles, other computer systems, and / or the user of vehicle 100. For example, user interface system 148 may include standard visual display devices (e.g., plasma displays, liquid crystal displays (LCDs), touchscreen displays, head-mounted displays, or other similar displays), speakers or other audio output devices, microphones or other audio input devices. For example, user interface system 148 may also include navigation interfaces and interfaces for controlling the internal environment of vehicle 100 (e.g., temperature, fan, etc.).

[0044] Communication system 152 can provide vehicle 100 with a means of communicating with one or more devices or other vehicles in the vicinity. In an exemplary embodiment, communication system 152 can communicate with one or more devices directly or through a communication network. Communication system 152 can be, for example, a wireless communication system. For example, the communication system can use 3G cellular communication (e.g., CDMA, EVDO, GSM / GPRS) or 4G cellular communication (e.g., WiMAX or LTE), and can also use 5G cellular communication. Optionally, the communication system can communicate with a wireless local area network (WLAN) (e.g., using...). In some embodiments, the communication system 152 can communicate directly with one or more devices or other vehicles in the vicinity, for example, using infrared light. Or ZigBee. Other wireless protocols, such as various vehicular communication systems, are also within the scope of this application. For example, the communication system may include one or more Dedicated Short Range Communication (DSRC) devices, V2V devices, or V2X devices that conduct public or private data communication with vehicles and / or roadside stations.

[0045] The computing system 150 can control some or all of the functions of the vehicle 100. The autonomous driving control unit in the computing system 150 can be used to identify, assess, and avoid or traverse potential obstacles in the environment in which the vehicle 100 is located. Typically, the autonomous driving control unit can be used to control the vehicle 100 without a driver or to assist a driver in controlling the vehicle. In some embodiments, the autonomous driving control unit is used to combine data from a GPS transceiver, radar data, LiDAR data, camera data, and data from other vehicle systems to determine the driving path or trajectory of the vehicle 100. The autonomous driving control unit can be activated to enable the vehicle 100 to be driven in autonomous driving mode.

[0046] The computing system 150 can include at least one processor (which can include at least one microprocessor) that executes processing instructions (i.e., machine executable instructions) stored in a non-transitory computer readable medium (e.g., a data storage device or memory). The computing system 150 can also be a plurality of computing devices that are distributed to control components or systems of the vehicle 100 in a distributed manner. In some embodiments, the memory can contain processing instructions (e.g., program logic) that are executed by the processor to implement various functions of the vehicle 100. In one embodiment, the computing system 150 is capable of data communication with the drive system 142, the sensor system 144, the control system 146, the user interface system 148, and / or the communication system 152. Interfaces in the computing system are used to facilitate data communication between the computing system 150 and the drive system 142, the sensor system 144, the control system 146, the user interface system 148, and the communication system 152.

[0047] The memory can also include other instructions, including instructions for data transmission, instructions for data reception, instructions for interaction, or instructions for controlling the drive system 142, the sensor system 144, or the control system 146 or the user interface system 148.

[0048] In addition to storing processing instructions, the memory can store various information or data, such as image processing parameters, road maps, and path information. These information can be used by the vehicle 100 and the computing system 150 during operation of the vehicle 100 in an autonomous manner, a semi-autonomous manner, and / or a manual mode.

[0049] Although the autonomous driving control unit is shown as being separate from the processor and the memory, it should be understood that in some implementations, some or all of the functions of the autonomous driving control unit can be implemented with program code instructions that reside in one or more memories (or data storage devices) and are executed by one or more processors, and the autonomous driving control unit can in some cases be implemented using the same processor and / or memory (or data storage device). In some implementations, the autonomous driving control unit can be implemented at least partially using various specialized circuit logic, various processors, various field programmable gate arrays (“FPGAs”), various application specific integrated circuits (“ASICs”), various real-time controllers, and hardware.

[0050] The computing system 150 can control functions of the vehicle 100 based on inputs received from various vehicle systems (e.g., the drive system 142, the sensor system 144, and the control system 146), or inputs received from the user interface system 148. For example, the computing system 150 can use inputs from the control system 146 to control a steering unit to avoid an obstacle detected by the sensor system 144. In one embodiment, the computing system 150 can be used to control multiple aspects of the vehicle 100 and its systems.

[0051] Although Figure 1 The various components (or units) shown in FIG. 1 can be integrated into the vehicle 100 or one or more of the components (or units) can be attached to the vehicle 100 or otherwise associated with the vehicle 100. For example, the computing system can exist partially or entirely outside of the vehicle 100. Thus, the vehicle 100 can exist in the form of a separate or integrated device unit. The device units that make up the vehicle 100 can communicate with each other in a wired or wireless manner. In some embodiments, additional components or units can be added to or removed from the various systems (e.g., the LiDAR or radar shown). Figure 1

[0052] Having introduced the vehicle architecture to which the present application relates, the following will be described in conjunction with Figure 2 A lane line corner point detection method provided by an embodiment of the present application will be described.

[0053] Step S101: Obtain a road image to be detected.

[0054] The road image to be detected can be obtained by a camera installed on the vehicle to collect road images in real time during automatic driving of the vehicle. The camera can be a front camera installed in front of the vehicle, a side camera installed on the side of the vehicle, or a hybrid mode of a front camera and a side camera.

[0055] In the road image collected by the front camera, the small squares in the dashed lane line are in a vertical oblong shape, and in the road image collected by the side camera, the small squares in the dashed lane line are in a horizontal oblong shape.

[0056] Step S102: Input the road image into a pre-trained neural network model for processing to obtain a heat map prediction result corresponding to the road image, the heat map prediction result including probabilities of each pixel point belonging to a specific corner point category.

[0057] ​The road image to be detected is input into the pre-trained neural network model for processing, and a heat map prediction result corresponding to the road image is obtained, wherein the heat map prediction result includes the probability of each pixel belonging to a specific corner point category (which can be the upper left category, the upper right category, the lower left category, or the lower right category), and different pixels belong to different specific corner point categories.

[0058] The neural network model is a pre-trained neural network model, and the training process can be: first, a plurality of (greater than or equal to 2) training images containing dashed lane lines are obtained, and then the initial neural network model is trained using the training images to obtain the trained neural network model. In the training image, the corner points of the dashed lane lines and the probabilities of each pixel near the corner points belonging to a specific corner point category are labeled, that is, after obtaining the road image containing the dashed lane lines, the corner points of the dashed lane lines in the road image and the probabilities of each pixel near the corner points belonging to a specific corner point category are labeled using a labeling algorithm. The probability of the pixel located at the corner point is 1, and the probabilities of each pixel near the corner point belonging to a specific corner point category are regularly distributed, that is, the probabilities conform to a specific function distribution, such as a Gaussian distribution, but are not limited thereto. Generally, the probability value of a pixel farther away from the corner point is relatively small. Different corner point categories correspond to different labels, and the corner point categories include the upper left category, the upper right category, the lower left category, and the lower right category. Four types of corner points belonging to the same dashed lane line form a corner point group, and each corner point group forms a lane line instance. The neural network model can be a convolutional neural network (CNN).

[0059] In an optional implementation, the neural network model can include a first network branch and a second network branch, the first network branch is used to output a semantic segmentation result of whether each pixel in the road image belongs to a lane line, and the second network branch is used to output a heat map prediction result of the road image. In this implementation, the road image is input into the neural network model, and the semantic segmentation result of whether each pixel in the road image belongs to a lane line and the heat map prediction result of the road image are output, and the principle diagram is as shown in Figure 3 The predicted value of each pixel in the semantic segmentation result is 0 or 1, or the predicted value of each pixel in the semantic segmentation result is a probability value in the interval [0, 1]. The predicted value of each pixel in the heat map prediction result is a probability value in the interval [0, 1]. Of course, a person skilled in the art can also set other numerical ranges according to needs, as long as the probability difference between pixels can be represented, and the present application does not limit this.

[0060] Wherein, during training, commonly used neural network optimization algorithms such as Stochastic Gradient Descen (SGD), Adam, etc. can be used for optimization training. When selecting the loss function, for the first network branch, commonly used segmentation loss functions such as normalization (Softmax), cross entropy (Cross Entropy) can be used. For the second network branch, commonly used loss functions for training Heatmap (heat map) such as focal loss, MSE (abbreviation of Mean Square Error) can be used. The optimization algorithm and loss function of the above example are well known to those skilled in the art, and will not be introduced here.

[0061] It should be understood that the lane line detection method can be executed during the running of the real vehicle, at which time the photographed image may or may not contain a road. Therefore, when processing the image using the model, it can be determined whether there is a lane line in the image (for example, the semantic segmentation result shows that each pixel point is a background point), and if so, the subsequent steps such as the prediction of the heat map prediction result are executed; otherwise, the lane line corner detection of the current image is stopped, and the lane line detection of the next image is continued. Here, the current image can also be determined whether there is a lane line according to some prior knowledge (for example, according to the target tracking result of the previous frame). In order to improve the accuracy of the lane line corner detection, in one embodiment, the heat map prediction result can be multiplied by the prediction value of each pixel in the semantic segmentation result to reduce the prediction value of the corner point in the heat map area without lane lines, and an updated heat map is obtained (see Figure 3 ). Wherein, the prediction value of each pixel in the semantic segmentation result is 0 or 1, or the prediction value of each pixel in the semantic segmentation result is a probability value in the interval [0, 1]. The prediction value of each pixel in the heat map prediction result is a probability value in the interval [0, 1]. The probability value of the pixel point representing the foreground (lane line) in the semantic segmentation result is 1, and the probability value of the pixel point representing the background (not lane line) is 0. By multiplying the heat map prediction result with the prediction value of each pixel in the semantic segmentation result, the prediction probability value of the corner point in the heat map area without lane lines can be effectively reduced, and the accuracy of the corner detection can be improved. For example, if there is a lane line at a certain position, then multiply 1 on the Heatmap, if not, multiply 0.

[0062] In multiplying the heat map prediction result with the prediction value of each pixel in the semantic segmentation result, in one implementation, the heat map prediction result can be directly multiplied with the prediction value of each pixel in the semantic segmentation result. In another alternative implementation, the pixel range representing the lane line in the semantic segmentation result can be first expanded (e.g., by three pixels, but not limited thereto), and then the heat map prediction result can be multiplied with the prediction value of each pixel in the range-expanded semantic segmentation result. In this way, it can be ensured that in filtering the corners in the heat map without lane lines, the corners in the lane line area will not be filtered out.

[0063] The pixel range representing the lane line in the semantic segmentation result is expanded in the present application, mainly to prevent missing the corners close to the lane line but with small semantic segmentation prediction results (the predicted corner result is outside the edge) in the corner screening process. It should be noted that when the semantic segmentation result is represented by binary classification (0 or 1), the pixel range representing the lane line in the semantic segmentation result can be directly expanded. When the semantic segmentation result is represented by a probability value in the interval [0, 1], the semantic segmentation result can not be expanded, or a certain threshold (e.g., 0.5) can be set to change the probability value in the interval [0, 1] to a binary classification result, and then the pixel range representing the lane line can be expanded.

[0064] In one alternative implementation, the heat map prediction result includes four heat maps, each heat map corresponding to one corner category and including the probability of each pixel in the heat map belonging to the corresponding corner category. In this implementation, the heat map prediction result includes a heat map representing the left-up category, a heat map representing the right-up category, a heat map representing the left-down category, and a heat map representing the right-down category. At this time, each pixel in the same heat map belongs to the same corner category. The probability value of each corner area conforms to a Gaussian distribution, and the probability value at the position of the corner is maximum 1. As the distance from the corner increases, the probability value also decreases accordingly.

[0065] In the present application, when predicting the corner category to which each pixel in the heat map belongs by using the neural network model, the model classifies each corner type and directly outputs four heat maps of the corresponding four corner categories after classification, thereby improving the subsequent corner matching accuracy. Here, the classification of the corner can reduce the error matching. For example, the left-up corner will only match the right-up corner. If the right-up corner is missed, there is no suitable right-up corner to match, so no error matching will occur.

[0066] In one alternative, the four heat maps described above can be integrated on the same map.

[0067] Step S103: determining the multiple different categories of corner points in the road image according to the probability of each pixel belonging to a specific corner point category in the heat map prediction result.

[0068] After obtaining the heat map prediction result, the multiple different categories of corner points in the road image can be determined according to the probability of each pixel belonging to a specific corner point category in the heat map prediction result, that is, the corner points belonging to the top-left category, the top-right category, the bottom-left category and the bottom-right category in the road image are determined.

[0069] In an optional implementation, the heat map prediction result includes four heat maps, each heat map corresponds to a category of corner points, and includes the probability of each pixel belonging to the corresponding category of corner points in the heat map. At this time, the process of determining the multiple different categories of corner points in the road image according to the probability of each pixel belonging to a specific category of corner points in the heat map prediction result can be: determining the corner points of the corresponding category in each heat map according to the probability of each pixel belonging to a specific category of corner points in the heat map, and then obtaining the four categories of corner points included in the four heat maps. For example, for the probability value of each pixel in the heat map belonging to the top-left category, the corner points belonging to the top-left category are determined, such as selecting the pixel point with the maximum probability value in each sliding window region as the corner point.

[0070] In an optional implementation, the heat map prediction result includes four heat maps, each heat map corresponds to a category of corner points, and includes the probability of each pixel belonging to the corresponding category of corner points in the heat map. At this time, the process of determining the multiple different categories of corner points in the road image according to the probability of each pixel belonging to a specific category of corner points in the heat map prediction result can be: determining the corner points of the corresponding category in each heat map according to the probability of each pixel belonging to a specific category of corner points in the heat map, and then obtaining the four categories of corner points included in the four heat maps. For example, for the probability value of each pixel in the heat map belonging to the top-left category, the corner points belonging to the top-left category are determined, such as selecting the pixel point with the maximum probability value in each sliding window region as the corner point.

[0071] Step S104: grouping the multiple different categories of corner points according to a predetermined rule to obtain multiple groups of corner points.

[0072] After determining the multiple different categories of corner points in the road image, the multiple different categories of corner points can be grouped according to a predetermined rule to obtain multiple groups of corner points, wherein each group of corner points includes four categories (top-left category, top-right category, bottom-left category and bottom-right category) of corner points belonging to the same lane line.

[0073] In grouping the corner points of the plurality of different categories according to the predetermined rule, in an alternative implementation, the corner points of the nearest top-left category and the top-right category can be matched to obtain a top corner point pair, the corner points of the nearest bottom-left category and the bottom-right category can be matched to obtain a bottom corner point pair, and then the nearest top corner point pair and the bottom corner point pair can be matched, so as to obtain the plurality of corner point groups. In the matching, the corner point of the top-left category can be matched with the corner point of the nearest top-right category, or the corner point of the top-right category can be matched with the corner point of the nearest top-left category. Similarly, in the matching of the corner points of the nearest bottom-left category and the bottom-right category, the corner point of the bottom-left category can be matched with the corner point of the nearest bottom-right category, or the corner point of the bottom-right category can be matched with the corner point of the nearest bottom-left category. Similarly, in the matching of the nearest top corner point pair and the bottom corner point pair, the top corner point pair can be matched with the nearest bottom corner point pair, or the bottom corner point pair can be matched with the nearest top corner point pair.

[0074] In another alternative implementation, the corner points of the nearest top-left category and the bottom-left category can be matched to obtain a left corner point pair, the corner points of the nearest top-right category and the bottom-right category can be matched to obtain a right corner point pair, and then the nearest left corner point pair and the right corner point pair can be matched, so as to obtain the plurality of corner point groups. In the matching, the corner point of the top-left category can be matched with the corner point of the nearest bottom-left category, or the corner point of the bottom-left category can be matched with the corner point of the nearest top-left category. Similarly, in the matching of the corner points of the nearest top-right category and the bottom-right category, the corner point of the top-right category can be matched with the corner point of the nearest bottom-right category, or the corner point of the bottom-right category can be matched with the corner point of the nearest top-right category. Similarly, in the matching of the nearest left corner point pair and the right corner point pair, the left corner point pair can be matched with the nearest right corner point pair, or the right corner point pair can be matched with the nearest left corner point pair.

[0075] In the matching, for the road image collected by the front camera (the small squares in the dashed lane line are in a vertical oblong shape), the nearest left upper category and right upper category corner points are matched to obtain an upper corner point pair, the nearest left lower category and right lower category corner points are matched to obtain a lower corner point pair, and then the nearest upper corner point pair and lower corner point pair are matched. For the road image collected by the side camera (the small squares in the dashed lane line are in a horizontal oblong shape), in the matching, the nearest left upper category and left lower category corner points are matched to obtain a left corner point pair, the nearest right upper category and right lower category corner point are matched to obtain a right corner point pair, and then the nearest left corner point pair and right corner point pair are matched.

[0076] To improve the accuracy of the corner point detection, as an optional implementation, after the plurality of corner point groups are obtained, the method further includes: eliminating an abnormal corner point group from the plurality of corner point groups. By eliminating the obviously abnormal corner point group from the plurality of obtained corner point groups, the accuracy of the corner point detection is improved.

[0077] In the corner point matching, the nearest distance matching is used, which to some extent ensures that each corner point group contains four types of corner points belonging to the same lane line. Through the subsequent elimination of the abnormal corner point group, it is further ensured that each corner point group contains four types of corner points belonging to the same lane line, and each corner point group constitutes a lane line instance.

[0078] In the elimination of the abnormal corner point group from the plurality of corner point groups, the corner point group in which the edge length difference of the two corner point pairs is greater than a predetermined threshold can be eliminated. That is, for each corner point group (including four corner points of left upper, left lower, right upper and right lower), if the edge length difference between the left upper / right upper corner point pair and the left lower / right lower corner point pair is greater than a predetermined threshold, or the edge length difference between the right upper / right lower corner point pair and the left upper / left lower corner point pair is greater than a predetermined threshold, the corner point group is an abnormal corner point group. The predetermined threshold can be set as needed.

[0079] In addition, the corner point group in which the intersection of the left upper and right lower lines and the intersection of the left lower and right upper lines does not have a lane line in the semantic segmentation result corresponding to the diagonal line region can also be eliminated. That is, for each corner point group (including four corner points of left upper, left lower, right upper and right lower), if the intersection of the left upper and right lower lines and the intersection of the left lower and right upper lines does not have a lane line in the semantic segmentation result corresponding to the diagonal line region, it can be eliminated.

[0080] The embodiment of the present application also provides a lane line corner point detection device 200, as shown in Figure 4The lane line corner point detection apparatus 200 includes an image acquisition module 210, a model prediction module 220, a corner point determination module 230, and a corner point group determination module 240.

[0081] The image acquisition module 210 is configured to acquire a road image to be detected.

[0082] The model prediction module 220 is configured to input the road image into a pre-trained neural network model for processing to obtain a heat map prediction result corresponding to the road image, the heat map prediction result including probabilities of each pixel belonging to a specific corner point category.

[0083] The corner point determination module 230 is configured to determine a plurality of corner points of different categories in the road image according to the probabilities of each pixel belonging to a specific corner point category in the heat map prediction result.

[0084] The corner point group determination module 240 is configured to group the plurality of corner points of different categories according to a predetermined rule.

[0085] Optionally, the neural network model includes a first network branch and a second network branch, the first network branch being configured to output a semantic segmentation result of whether each pixel in the road image belongs to a lane line, and the second network branch being configured to output a heat map prediction result of the road image. The prediction value of each pixel in the semantic segmentation result is 0 or 1, or the prediction value of each pixel in the semantic segmentation result is a probability value in the interval [0, 1]; the prediction value of each pixel in the heat map prediction result is a probability value in the interval [0, 1]. Correspondingly, the lane line corner point detection apparatus 200 further includes a heat map updating module configured to multiply the heat map prediction result and the prediction value of each pixel in the semantic segmentation result to reduce the prediction value of a corner point in a region without a lane line in the heat map, and obtain an updated heat map. Optionally, the heat map updating module is specifically configured to expand a pixel range representing a lane line in the semantic segmentation result; and multiply the heat map prediction result and the prediction value of each pixel in the range-expanded semantic segmentation result.

[0086] Optionally, the heat map prediction result includes four heat maps, each heat map corresponding to a category of corner points and including a probability of each pixel in the heat map belonging to the corresponding category of corner points. The corner point determination module 230 is specifically configured to determine a corner point of the corresponding category in each heat map according to the probability of each pixel in the heat map belonging to a specific category of corner points, and thus obtain four categories of corner points included in the four heat maps. The corner point determination module 230 is specifically configured to perform a sliding window operation on the heat map using a preset sliding window region, and if the probability value of the center point of the sliding window region satisfies a predetermined condition, mark the center point of the sliding window region as a corner point of the corresponding category. Optionally, the predetermined condition is that the probability value of the center point of the sliding window region is greater than or equal to a preset probability threshold and is a local maximum value of the sliding window region.

[0087] The corner point category includes at least one of the following: top left, top right, bottom left, and bottom right.

[0088] Optionally, the corner point group determination module 240 is specifically used to: match the corner points of the nearest upper left and upper right categories to obtain upper corner point pairs; match the corner points of the nearest lower left and lower right categories to obtain lower corner point pairs; and match the nearest upper corner point pairs and lower corner point pairs to obtain multiple corner point groups.

[0089] The corner point group determination module 240 is specifically used to: match the corner points of the nearest upper left category and lower left category to obtain a left corner point pair; match the corner points of the nearest upper right category and lower right category to obtain a right corner point pair; and match the nearest left corner point pair and right corner point pair to obtain multiple corner point groups.

[0090] Optionally, the lane corner detection device 200 further includes a corner group elimination module for eliminating abnormal corner groups from multiple corner groups. Specifically, the corner group elimination module is used to eliminate corner groups from multiple corner groups where the difference in side length between two corner pairs is greater than a predetermined threshold, and also to eliminate corner groups from multiple four-corner groups where the semantic segmentation result corresponding to the diagonal region does not contain lane lines.

[0091] The lane corner detection device 200 further includes a training module for: acquiring multiple training images containing dashed lane lines, wherein the training images are labeled with the corners of the dashed lane lines and the probability of each pixel near the corner belonging to a specific corner category; and training an initial neural network model using the training images to obtain a trained neural network model. Optionally, the probability of each pixel near the corner belonging to a specific corner category follows a Gaussian distribution, and each group of corners constitutes a lane line instance.

[0092] The lane line corner detection device 200 provided in this application embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0093] like Figure 5 As shown, Figure 5 This diagram illustrates a structural block diagram of an electronic device 300 according to an embodiment of this application. The electronic device 300 includes: a communication interface 310, a memory 320, a communication bus 330, and a processor 340.

[0094] The communication interface 310, memory 320, and processor 340 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 330 or signal lines. The communication interface 310 is used to implement one or more communication protocols (LTE, Wi-Fi, etc.). The memory 320 is used to store computer programs, such as... Figure 4 The software functional module shown is the lane corner detection device 200. The lane corner detection device 200 includes at least one software functional module that can be stored as software or firmware in the memory 320 or embedded in the operating system (OS) of the electronic device 300. The processor 340 is used to execute executable modules stored in the memory 320, such as the software functional module or computer program included in the lane corner detection device 200. For example, the processor 340 is used to acquire a road image to be detected; input the road image into a pre-trained neural network model for processing to obtain a heatmap prediction result corresponding to the road image, the heatmap prediction result including the probability of each pixel belonging to a specific corner category; determine multiple different categories of corners in the road image based on the probability of each pixel belonging to a specific corner category in the heatmap prediction result; and group the multiple different categories of corners according to a predetermined rule to obtain multiple corner groups.

[0095] The memory 320 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0096] The processor 340 can be an integrated circuit chip with processing capability. The above processor can be a general purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; also can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general purpose processor can be a microprocessor or the processor 340 can also be any conventional processor or the like.

[0097] The electronic device 300 described above includes but is not limited to a computer, a server, and a vehicle-mounted computer in a vehicle, etc.

[0098] The embodiments of the present application also provide a non-volatile computer readable storage medium (hereinafter referred to as storage medium) having a computer program stored thereon, wherein the computer program is run by a computer such as the electronic device 300 described above, and executes the lane line corner detection method shown above.

[0099] It should be noted that each of the embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts between the embodiments can be referred to each other.

[0100] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus embodiments described above are merely illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders from those described in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0101] In addition, the functional modules in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0102] If the present application is implemented in the form of software functional modules and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that makes a contribution to the prior art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a notebook computer, a server, or an electronic device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0103] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting lane line corner points, characterized in that, include: Acquire the image of the road to be detected; The road image is input into a pre-trained neural network model for processing to obtain a heat map prediction result corresponding to the road image. The heat map prediction result includes the probability that each pixel belongs to a specific corner point category. Based on the probability of each pixel belonging to a specific corner point category in the heatmap prediction results, multiple different categories of corner points in the road image are determined; as well as The corner points of different categories are grouped according to predetermined rules to obtain multiple corner point groups; Each group of corner points constitutes a lane line instance, and the corner point categories include top-left, top-right, bottom-left, and bottom-right categories. The method further includes: Remove corner point groups from the plurality of corner point groups where the difference in side length between two corner point pairs is greater than a predetermined threshold.

2. The method according to claim 1, characterized in that, The neural network model includes a first network branch and a second network branch. The first network branch is used to output the semantic segmentation result of whether each pixel in the road image belongs to a lane line, and the second network branch is used to output the heat map prediction result of the road image.

3. The method according to claim 2, characterized in that, The predicted value of each pixel in the semantic segmentation result is 0 or 1, or the predicted value of each pixel in the semantic segmentation result is a probability value in the interval [0,1]. The predicted value of each pixel in the heatmap prediction result is a probability value within the interval [0,1].

4. The method according to claim 2, further comprising: The predicted value of each pixel in the heatmap is multiplied by the predicted value of each pixel in the semantic segmentation result to remove points in the heatmap without lane lines, thus obtaining an updated heatmap.

5. The method according to claim 4, characterized in that, Multiplying the heatmap prediction result with the predicted value of each pixel in the semantic segmentation result includes: Expand the range of pixels representing lane lines in the semantic segmentation results; The heatmap prediction result is multiplied by the predicted value of each pixel in the semantic segmentation result after the range is expanded.

6. The method according to claim 1, characterized in that, The heatmap prediction result includes four heatmaps, each corresponding to a corner point category, and includes the probability that each pixel in the heatmap belongs to the corresponding corner point category.

7. The method according to claim 6, characterized in that, Based on the probability of each pixel belonging to a specific corner point category in the heatmap prediction results, multiple different categories of corner points in the road image are determined, including: Based on the probability of each pixel in each heatmap belonging to a specific corner point category, the corner point of the corresponding category in that heatmap is determined, thus obtaining the four categories of corner points contained in the four heatmaps.

8. The method according to claim 7, characterized in that, Based on the probability that each pixel in each heatmap belongs to a specific corner point category, the corner points of the corresponding category in that heatmap are determined, including: A sliding window operation is performed on the heat map using a preset sliding window area. If the probability value of the center point of the sliding window area meets a predetermined condition, the center point of the sliding window area is marked as a corner point of a corresponding category.

9. The method according to claim 8, characterized in that, The predetermined conditions are as follows: The probability value of the center point of the sliding window area is greater than or equal to a preset probability threshold and is a local maximum of the sliding window area.

10. The method according to claim 1, characterized in that, The corner points of different categories are grouped according to predetermined rules to obtain multiple corner point groups, including: Match the corner points of the nearest top-left and top-right categories to obtain the top corner point pairs; Match the corner points of the nearest bottom left and bottom right categories to obtain bottom corner point pairs; Match the nearest top corner pair and bottom corner pair to obtain multiple corner pair groups.

11. The method according to claim 1, characterized in that, The corner points of different categories are grouped according to predetermined rules to obtain multiple corner point groups, including: Match the corner points of the nearest top-left and bottom-left categories to obtain left corner point pairs; Match the corner points of the nearest top-right and bottom-right categories to obtain right corner point pairs; Match the nearest left and right corner point pairs to obtain multiple corner point groups.

12. The method according to claim 1, characterized in that, Also includes: Acquire multiple training images containing dashed lane lines, wherein the training images are labeled with the corner points of the dashed lane lines and the probability that each pixel near the corner point belongs to a specific corner point category; The initial neural network model is trained using the training images to obtain a trained neural network model.

13. The method according to claim 12, characterized in that, The probability that each pixel near the corner point belongs to a specific corner point category follows a Gaussian distribution.

14. A lane line corner detection device, characterized in that, include: The image acquisition module is used to acquire images of the road to be detected. The model prediction module is used to input the road image into a pre-trained neural network model for processing to obtain the heat map prediction result corresponding to the road image. The heat map prediction result includes the probability that each pixel belongs to a specific corner point category. The corner point determination module is used to determine multiple different categories of corner points in the road image based on the probability that each pixel belongs to a specific corner point category in the heat map prediction results. The corner point group determination module is used to group the multiple different categories of corner points according to a predetermined rule to obtain multiple corner point groups. Each corner point group constitutes a lane line instance. The corner point categories include the upper left category, the upper right category, the lower left category, and the lower right category. as well as The corner group elimination module is used to eliminate corner groups in which the difference in side length between two pairs of corner points is greater than a predetermined threshold.

15. An electronic device, characterized in that, include: A memory and a processor, wherein the processor is connected to the memory; The memory is used to store programs; The processor is configured to invoke a program stored in the memory to execute the method as described in any one of claims 1-13.

16. A storage medium, characterized in that, It stores a computer program, which, when executed by a processor, performs the method as described in any one of claims 1-13.

Images