Image-based lane line departure early warning detection method

Abstract

The invention relates to an image-based lane line departure early warning detection method and device, electronic equipment and a storage medium. The method comprises the steps that a video image of a preset angle in front of a vehicle is collected based on a vehicle-mounted camera arranged at a preset position of the vehicle, and lane line detection is completed with the video image as input based on a lane line detection model; converting the 2D coordinates of the detected lane line into 3D coordinates based on the internal reference information and the external reference information of the vehicle-mounted camera; and based on the position change between the vehicle and the lane in the preset frame number of the video image, calculating the approaching speed of the vehicle to the lane so as to complete lane line departure early warning detection. According to the invention, a lane line departure early warning function can be realized only by adopting a single visible light camera, so that the cost is low, the robustness is high, and the precision is relatively high; the network is simple, installation and deployment are convenient, and detection real-time performance is high.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent assistance systems for automotive safety, and more specifically, to an image-based lane departure warning detection method, device, electronic device, and computer-readable storage medium. Background Technology

[0002] Lane departure warning detection is a technology used in vehicle driver assistance systems. It aims to detect whether a vehicle is deviating from its lane in real time by analyzing image data of the road ahead and issuing a warning signal when necessary. Existing lane departure warning detection methods are mainly based on traditional computer vision techniques and machine learning algorithms. These methods typically require image preprocessing, such as edge detection and filtering, followed by feature extraction and classifiers to identify lane lines and determine whether the vehicle is deviating from its lane. However, these traditional methods are limited in performance under complex scenarios, such as changes in lighting, poor weather conditions, and missing road signs.

[0003] Therefore, one or more methods are needed to solve the above problems.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide an image-based lane departure warning detection method, apparatus, electronic device, and computer-readable storage medium, thereby overcoming, at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0006] According to one aspect of this disclosure, an image-based lane departure warning detection method is provided, comprising:

[0007] The vehicle-mounted camera, positioned at a preset location on the vehicle, captures video images from a preset angle in front of the vehicle. Based on a lane detection model, the video images are used as input to complete lane detection.

[0008] Based on the intrinsic and extrinsic information of the vehicle-mounted camera, the 2D coordinates of the detected lane lines are converted into 3D coordinates.

[0009] Based on the positional changes between the vehicle and the lane between preset frames of the video image, the approach speed of the vehicle to the lane is calculated, thereby completing the lane departure warning detection.

[0010] In one exemplary embodiment of this disclosure, the lane detection model in the method further includes:

[0011] The training dataset for lane line detection is based on preset lane line data;

[0012] Establish a lane line detection model that includes a backbone network, a multi-scale fusion module, and a detection module;

[0013] The lane detection model is trained based on the training dataset to complete the construction of the lane detection model.

[0014] In one exemplary embodiment of this disclosure, the method further includes:

[0015] The training dataset for lane line detection is based on preset lane line data, and the data in the training dataset is augmented and the lane line labels are processed into the format of the network output.

[0016] In one exemplary embodiment of this disclosure, the method further includes:

[0017] The backbone network is based on the ResNet-18 model, with an image input size of 3x288x800, and outputs feature maps downsampled by 8x, 16x, and 32x.

[0018] The multi-scale fusion module is based on the multi-scale feature pyramid (FPN) structure, which fuses features downsampled by 8 times, 16 times, and 32 times, and outputs a 256x36x100 feature map.

[0019] The detection module is based on parallel connection of instance detection and structure detection, which performs lane line instance detection and lane line structure detection respectively.

[0020] In one exemplary embodiment of this disclosure, the method further includes:

[0021] Using the training dataset as input, based on the focal-loss loss function and the cross-entropy loss function of the structure detection module, the maximum number of training epochs is set to 100, the optimizer is SGD, the initial learning rate is 0.01, and the batch size is 24. The weight parameter values ​​of each layer are updated through backpropagation and learning rate to complete the model training.

[0022] In one exemplary embodiment of this disclosure, the method further includes:

[0023] Based on the intrinsic parameter information of the vehicle-mounted camera, the camera coordinate system of the video image is transformed into the vehicle coordinate system;

[0024] Based on the extrinsic information of the vehicle-mounted camera, calculate the true depth value of the acquisition plane of the vehicle-mounted camera;

[0025] Based on the video image and actual depth value of the vehicle coordinate system, the 2D coordinates of the detected lane lines are converted into 3D coordinates.

[0026] In one exemplary embodiment of this disclosure, the method further includes:

[0027] Based on the positional changes between the vehicle and the lane at 5-frame intervals in the video image, the approach speed of the vehicle to the lane is calculated.

[0028] Based on the vehicle's preset three-dimensional coordinate position and vehicle width information, calculate the time the vehicle deviates from the lane;

[0029] The time the vehicle deviates from its lane is compared with a preset threshold. If the time the vehicle deviates from its lane is less than the preset threshold, a vehicle deviation warning is issued.

[0030] In one aspect of this disclosure, an image-based lane departure warning detection device is provided, comprising:

[0031] The lane detection module is used to acquire video images of a preset angle in front of the vehicle based on an onboard camera placed at a preset position on the vehicle, and to complete lane detection based on the lane detection model and the video images as input.

[0032] The 2D to 3D module is used to convert the 2D coordinates of the detected lane lines into 3D coordinates based on the intrinsic and extrinsic information of the vehicle-mounted camera.

[0033] The lane departure warning module is used to calculate the approach speed of the vehicle to the lane based on the positional changes between the vehicle and the lane between preset frames of the video image, thereby completing the lane departure warning detection.

[0034] In one aspect of this disclosure, an electronic device is provided, comprising:

[0035] Processor; and

[0036] A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of the preceding claims.

[0037] In one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to any one of the preceding claims.

[0038] An exemplary embodiment of this disclosure provides an image-based lane departure warning detection method. The method includes: acquiring a video image at a preset angle in front of the vehicle using an onboard camera positioned at a preset location on the vehicle; using the video image as input and a lane line detection model to detect lane lines; converting the 2D coordinates of the detected lane lines into 3D coordinates based on the intrinsic and extrinsic parameters of the onboard camera; and calculating the approach speed of the vehicle to the lane based on the positional changes between the vehicle and the lane over a preset number of frames in the video image, thereby completing the lane departure warning detection. This disclosure requires only a single visible light camera to achieve lane departure warning functionality, offering low cost, strong robustness, and high accuracy; it also simplifies network requirements, facilitates installation and deployment, and provides high real-time detection performance.

[0039] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0040] The above and other features and advantages of this disclosure will become more apparent from the detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0041] Figure 1 A flowchart of an image-based lane departure warning detection method according to an exemplary embodiment of the present disclosure is shown;

[0042] Figure 2 A system flowchart of an image-based lane departure warning detection method according to an exemplary embodiment of the present disclosure is shown;

[0043] Figure 3 A lane detection flowchart of an image-based lane departure warning detection method according to an exemplary embodiment of the present disclosure is shown;

[0044] Figure 4 A schematic diagram of a CAM fusion module for an image-based lane departure warning detection method according to an exemplary embodiment of the present disclosure is shown.

[0045] Figure 5 A schematic diagram of a residual module of an image-based lane departure warning detection method according to an exemplary embodiment of the present disclosure is shown.

[0046] Figure 6 A structural block diagram of an image-based lane departure warning detection device according to an exemplary embodiment of the present disclosure is shown;

[0047] Figure 7 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown schematically;

[0048] Figure 8The illustration shows a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0049] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0050] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details described, or other methods, components, materials, apparatuses, steps, etc., can be employed. In other instances, well-known structures, methods, apparatuses, implementations, materials, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0051] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.

[0052] In this example embodiment, an image-based lane departure warning detection method is first provided; refer to Figure 1 As shown, this image-based lane departure warning detection method may include the following steps:

[0053] Step S110: Based on the video image acquired by the vehicle-mounted camera placed at a preset position on the vehicle at a preset angle in front of the vehicle, and based on the lane line detection model, the lane line detection is completed with the video image as input.

[0054] Step S120: Based on the intrinsic and extrinsic information of the vehicle-mounted camera, the 2D coordinates of the detected lane lines are converted into 3D coordinates.

[0055] Step S130: Based on the positional changes between the vehicle and the lane between preset frames of the video image, calculate the approach speed of the vehicle to the lane, and then complete the lane departure warning detection.

[0056] An exemplary embodiment of this disclosure provides an image-based lane departure warning detection method. The method includes: acquiring a video image at a preset angle in front of the vehicle using an onboard camera positioned at a preset location on the vehicle; using the video image as input and a lane line detection model to detect lane lines; converting the 2D coordinates of the detected lane lines into 3D coordinates based on the intrinsic and extrinsic parameters of the onboard camera; and calculating the approach speed of the vehicle to the lane based on the positional changes between the vehicle and the lane over a preset number of frames in the video image, thereby completing the lane departure warning detection. This disclosure requires only a single visible light camera to achieve lane departure warning functionality, offering low cost, strong robustness, and high accuracy; it also simplifies network requirements, facilitates installation and deployment, and provides high real-time detection performance.

[0057] The following will further explain an image-based lane departure warning detection method in this example embodiment.

[0058] Example 1:

[0059] In this example embodiment, this disclosure addresses the limitation of traditional lane departure warning systems in complex scenarios by providing a lane line detection method and lane departure warning system applicable to complex scenarios, requiring only a single camera. This invention utilizes deep learning algorithms to analyze and process road images acquired by an onboard camera, combining the camera's intrinsic and extrinsic parameters to transform the image information into a vehicle coordinate system space, further extracting lane line information in three-dimensional space. Finally, by comprehensively analyzing the lane line information in three-dimensional space and the vehicle's motion state, the relative distance and relative speed between the vehicle and the lane line are used to calculate the time of imminent lane departure, and a lane departure warning is issued in advance based on a pre-set time threshold.

[0060] In step S110, a video image at a preset angle in front of the vehicle can be acquired based on an onboard camera placed at a preset position on the vehicle. The lane line detection model is used as input to complete the lane line detection.

[0061] In this example embodiment, the lane detection model in the method further includes:

[0062] The training dataset for lane line detection is based on preset lane line data;

[0063] Establish a lane line detection model that includes a backbone network, a multi-scale fusion module, and a detection module;

[0064] The lane detection model is trained based on the training dataset to complete the construction of the lane detection model.

[0065] In this example embodiment, the method further includes:

[0066] The training dataset for lane line detection is based on preset lane line data, and the data in the training dataset is augmented and the lane line labels are processed into the format of the network output.

[0067] In this example embodiment, the method further includes:

[0068] The backbone network is based on the ResNet-18 model, with an image input size of 3x288x800, and outputs feature maps downsampled by 8x, 16x, and 32x.

[0069] The multi-scale fusion module is based on the multi-scale feature pyramid (FPN) structure, which fuses features downsampled by 8 times, 16 times, and 32 times, and outputs a 256x36x100 feature map.

[0070] The detection module is based on parallel connection of instance detection and structure detection, which performs lane line instance detection and lane line structure detection respectively.

[0071] In this example embodiment, the method further includes:

[0072] Using the training dataset as input, based on the focal-loss loss function and the cross-entropy loss function of the structure detection module, the maximum number of training epochs is set to 100, the optimizer is SGD, the initial learning rate is 0.01, and the batch size is 24. The weight parameter values ​​of each layer are updated through backpropagation and learning rate to complete the model training.

[0073] In step S120, the 2D coordinates of the detected lane line can be converted into 3D coordinates based on the intrinsic and extrinsic information of the vehicle camera.

[0074] In this example embodiment, the method further includes:

[0075] Based on the intrinsic parameter information of the vehicle-mounted camera, the camera coordinate system of the video image is transformed into the vehicle coordinate system;

[0076] Based on the extrinsic information of the vehicle-mounted camera, calculate the true depth value of the acquisition plane of the vehicle-mounted camera;

[0077] Based on the video image and actual depth value of the vehicle coordinate system, the 2D coordinates of the detected lane lines are converted into 3D coordinates.

[0078] In step S130, the approach speed of the vehicle to the lane can be calculated based on the positional changes between the vehicle and the lane between preset frames of the video image, thereby completing the lane departure warning detection.

[0079] In this example embodiment, the method further includes:

[0080] Based on the positional changes between the vehicle and the lane at 5-frame intervals in the video image, the approach speed of the vehicle to the lane is calculated.

[0081] Based on the vehicle's preset three-dimensional coordinate position and vehicle width information, calculate the time the vehicle deviates from the lane;

[0082] The time the vehicle deviates from its lane is compared with a preset threshold. If the time the vehicle deviates from its lane is less than the preset threshold, a vehicle deviation warning is issued.

[0083] Example 2:

[0084] In this example embodiment, reference is made to Figure 2 As shown, the lane departure system of the present invention includes: a lane detection module, a 2D-to-3D conversion module, and a lane departure warning module. The lane detection module uses deep learning algorithms to analyze and process road images acquired by an onboard camera to accurately identify lane lines. The 2D-to-3D conversion module combines the camera's intrinsic and extrinsic parameters to convert the two-dimensional image information output by the lane detection module into a three-dimensional camera coordinate system space. The lane departure warning module comprehensively analyzes the lane line information in three-dimensional space and the vehicle's own motion state to determine lane departure warning. A detailed description of all the functions and principles of this system is as follows:

[0085] In this example embodiment, the lane line detection step includes: referring to Figure 3 As shown, unlike traditional lane line segmentation algorithms, this module instantiates each lane line, defining lane line detection as a position selection classification problem based on the row direction, using a set of points based on row classification to represent the lane line. This method eliminates the need for complex post-processing such as clustering, significantly improving the algorithm's speed. Combining keypoint detection and conditional convolution, this module can efficiently detect multiple lane lines simultaneously. The main detection process of this module is as follows:

[0086] 1. Obtain a large amount of rich lane line data (urban, rural, etc.) as the training dataset for lane line detection;

[0087] 2. Extensive data enhancement processing is performed, including random rotation, random translation, random cropping, and color space transformation;

[0088] 3. Process the lane line labels into the network output format, that is, divide the image into 100 columns for position classification (0-99), and divide it into 36 rows. The format of each lane line is (1x36), and the value of each position is the category of the lane line in that row, where category 0 means that there are no lane lines in that row.

[0089] 4. Constructing the lane line detection model includes:

[0090] Backbone Network: ResNet-18 is used as the backbone network of the model to extract image features. The image input size is 3x288x800, and the output features are divided into three parts: a feature map downsampled by 8 times with a scale of 128x36x100; a feature map downsampled by 16 times with a scale of 256x18x52; and a feature map downsampled by 32 times with a scale of 512x9x25.

[0091] Multi-scale fusion module: This module combines multi-scale feature pyramid (FPN) to extract multi-scale image features, fusing features downsampled by 8x, 16x, and 32x, outputting a 256x36x100 feature map. The fusion module CAM incorporates a channel attention mechanism to improve fusion efficiency. CAM is as follows... Figure 4 As shown, the feature map is first subjected to global average pooling to generate a vector of scale (1×C), where each channel is represented by a numerical value, achieving global low-dimensional embedding of the feature map. Then, a fully connected network transforms this global vector into a weight vector to enhance effective features and weaken ineffective features.

[0092] Detection Module: After extracting rich lane line features, the final detection module of the network includes an instance detection module and a structure detection module, which are connected in parallel to correspond to lane line instance detection and lane line structure detection, respectively. The instance detection module consists of a residual layer and a 1x1 convolutional classification layer, where each residual layer contains two... Figure 5 The residual module shown outputs a 1x36x100 keypoint feature map through its convolutional classification layer. Considering occlusion at near or far distances, this module simultaneously uses the start and end points of the lane lines as keypoints for detection. If the downsampled positions of the start and end points of the lane lines fall on a point in this feature map, that point is responsible for the structure detection of that lane line. In forward mode, a softmax layer is designed for this feature map, and points with values ​​greater than 0.7 are considered valid keypoints. The structure detection module consists of one residual layer and one 1x1 convolutional kernel parameter layer. The structure detection module uses conditional convolution to output a 256x36x100 convolutional parameter feature map. For instance keypoints at position (x, y), 256 elements at the corresponding position are taken as convolutional kernel parameters, resulting in a specific 1x36x100 structure map to represent the lane lines. This method significantly reduces computation and improves model efficiency.

[0093] 5. Model Training:

[0094] During training, the instance detection module uses the focal loss function, and the structure detection module uses the cross-entropy loss function. The maximum number of training epochs is set to 100, the optimizer is SGD, the initial learning rate is 0.01, and the batch size is 24. The model training is completed by updating the weight parameters of each layer through backpropagation and learning rate.

[0095] In this example embodiment, the 2D to 3D conversion step includes:

[0096] Since the camera position is fixed, assuming the ground near the vehicle's driving path is flat, the actual 3D lane line positions on the ground can be calculated based on the pinhole imaging principle. K is the camera's intrinsic parameter. Let R be the extrinsic parameter of the camera relative to the vehicle coordinate system, where R is a 3x3 rotation matrix and T is a 3x1 translation matrix.

[0097] The coordinates of a 3D point in the camera frame are:

[0098]

[0099] The lane line coordinates in the pixel coordinate system are:

[0100]

[0101] The pixel coordinate system converted to the camera coordinate system is as follows:

[0102]

[0103] The camera coordinate system is transformed into the vehicle coordinate system (rotation only):

[0104]

[0105] Since the height h in the camera's extrinsic parameters is fixed, the ratio p of the distance z to h at each point at that height is known. car-rot [1], thus all the true depth values ​​in the plane can be obtained:

[0106]

[0107] By incorporating the translation matrix T, we can obtain the true 3D coordinates of each point on the ground in the vehicle coordinate system.

[0108] p car =p car-rot ×zT.

[0109] In this example embodiment, the deviation warning step includes:

[0110] After obtaining the actual 3D coordinates (x, y, z) of the lane line in the vehicle coordinate system and the vehicle's width W, the vehicle's speed approaching or deviating from the lane line can be directly calculated based on the relative position changes between the vehicle and the lane line. This allows for the calculation of the future time of lane departure and provides early warning. This method does not require any vehicle motion information and is therefore less costly.

[0111] The calculation interval for relative position transformation is set to 5 frames, with a time interval of t. Let the position of the center of the vehicle's front end in the z-th frame be (X... car Y car The coordinates of the point closest to the vehicle are (X). line Y line Let the position of the center of the car's front end in frame z+5 be (X′). car ,Y′ car The coordinates of the point closest to the vehicle are (X′). line ,Y′ line If the vehicle approaches the lane at this point, then the speed at which it does so is:

[0112]

[0113] At this moment, the closest distance between the vehicle and the lane is:

[0114]

[0115] Therefore, the approximate future time of lane departure is:

[0116]

[0117] The threshold for deviation time can be used to determine whether to issue a warning.

[0118] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0119] Furthermore, in this example embodiment, an image-based lane departure warning detection device is also provided. (Refer to...) Figure 6 As shown, the image-based lane departure warning detection device 200 may include: a lane detection module 210, a 2D-to-3D conversion module 220, and a lane departure warning module 230. Wherein:

[0120] The lane detection module 210 is used to acquire video images of a preset angle in front of the vehicle based on an on-board camera placed at a preset position on the vehicle, and to complete lane detection based on a lane detection model and the video images as input.

[0121] The 2D to 3D module 220 is used to convert the 2D coordinates of the detected lane lines into 3D coordinates based on the intrinsic and extrinsic information of the vehicle-mounted camera.

[0122] The lane departure warning module 230 is used to calculate the approach speed of the vehicle to the lane based on the position change between the vehicle and the lane between preset frames of the video image, and then complete the lane departure warning detection.

[0123] The specific details of each of the above-mentioned image-based lane departure warning detection device modules have been described in detail in the corresponding image-based lane departure warning detection method, so they will not be repeated here.

[0124] It should be noted that although several modules or units of the image-based lane departure warning detection device 200 have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0125] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0126] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented as entirely hardware embodiments, entirely software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.”

[0127] The following reference Figure 7 To describe an electronic device 300 according to such an embodiment of the present invention. Figure 7 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0128] like Figure 7As shown, the electronic device 300 is presented in the form of a general-purpose computing device. The components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting different system components (including storage unit 320 and processing unit 310), and a display unit 340.

[0129] The storage unit stores program code that can be executed by the processing unit 310, causing the processing unit 310 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 310 can perform actions such as... Figure 1 Steps S110 to S130 are shown in the diagram.

[0130] Storage unit 320 may include readable media in the form of volatile storage units, such as random access memory (RAM) 3201 and / or cache memory 3202, and may further include read-only memory (ROM) 3203.

[0131] Storage unit 320 may also include a program / utility 3204 having a set (at least one) program module 3205, such program module 3205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0132] Bus 330 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0133] Electronic device 300 can also communicate with one or more external devices 370 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 300, and / or with any device that enables electronic device 300 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 350. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 360. As shown, network adapter 360 communicates with other modules of electronic device 300 via bus 330. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0134] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0135] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section above.

[0136] refer to Figure 8 As shown, a program product 400 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0137] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0138] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0139] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0140] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0141] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0142] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0143] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image-based lane departure warning detection method, characterized in that, The method includes: The vehicle-mounted camera, positioned at a preset location on the vehicle, captures video images from a preset angle in front of the vehicle. Based on a lane detection model, the video images are used as input to complete lane detection. Based on the intrinsic and extrinsic information of the vehicle-mounted camera, the 2D coordinates of the detected lane lines are converted into 3D coordinates. Based on the positional changes between the vehicle and the lane between preset frames of the video image, the approach speed of the vehicle to the lane is calculated, thereby completing the lane departure warning detection.

2. The method as described in claim 1, characterized in that, The lane detection model in the method also includes: The training dataset for lane line detection is based on preset lane line data; Establish a lane line detection model that includes a backbone network, a multi-scale fusion module, and a detection module; The lane detection model is trained based on the training dataset to complete the construction of the lane detection model.

3. The method as described in claim 2, characterized in that, The method further includes: The training dataset for lane line detection is based on preset lane line data, and the data in the training dataset is augmented and the lane line labels are processed into the format of the network output.

4. The method as described in claim 2, characterized in that, The method further includes: The backbone network is based on the ResNet-18 model, with an image input size of 3x288x800, and outputs feature maps downsampled by 8x, 16x, and 32x. The multi-scale fusion module is based on the multi-scale feature pyramid (FPN) structure, which fuses features downsampled by 8 times, 16 times, and 32 times, and outputs a 256x36x100 feature map. The detection module is based on parallel connection of instance detection and structure detection, which performs lane line instance detection and lane line structure detection respectively.

5. The method as described in claim 2, characterized in that, The method further includes: Using the training dataset as input, based on the focal-loss loss function and the cross-entropy loss function of the structure detection module, the maximum number of training epochs is set to 100, the optimizer is SGD, the initial learning rate is 0.01, and the batch size is 24. The weight parameter values ​​of each layer are updated through backpropagation and learning rate to complete the model training.

6. The method as described in claim 1, characterized in that, The method further includes: Based on the intrinsic parameter information of the vehicle-mounted camera, the camera coordinate system of the video image is transformed into the vehicle coordinate system; Based on the extrinsic information of the vehicle-mounted camera, calculate the true depth value of the acquisition plane of the vehicle-mounted camera; Based on the video image and actual depth value of the vehicle coordinate system, the 2D coordinates of the detected lane lines are converted into 3D coordinates.

7. The method as described in claim 1, characterized in that, The method further includes: Based on the positional changes between the vehicle and the lane at 5-frame intervals in the video image, the approach speed of the vehicle to the lane is calculated. Based on the vehicle's preset three-dimensional coordinate position and vehicle width information, calculate the time the vehicle deviates from the lane; The time the vehicle deviates from its lane is compared with a preset threshold. If the time the vehicle deviates from its lane is less than the preset threshold, a vehicle deviation warning is issued.

8. An image-based lane departure warning detection device, characterized in that, The device includes: The lane detection module is used to acquire video images of a preset angle in front of the vehicle based on an onboard camera placed at a preset position on the vehicle, and to complete lane detection based on the lane detection model and the video images as input. The 2D to 3D module is used to convert the 2D coordinates of the detected lane lines into 3D coordinates based on the intrinsic and extrinsic information of the vehicle-mounted camera. The lane departure warning module is used to calculate the approach speed of the vehicle to the lane based on the positional changes between the vehicle and the lane between preset frames of the video image, thereby completing the lane departure warning detection.

9. An electronic device, characterized in that, include Processor; and A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images