Commercial vehicle multi-camera lane line deviation early warning method and system

Abstract

A kind of commercial vehicle multi-camera lane line deviation early warning method and system, the method includes commercial vehicle multi-camera collection front and side rear lane line visual information containing road image;Based on the image collected, the lane line position of front view camera is detected using deep learning, and the lane line position of two side rear view camera is detected using digital image processing;According to lane line position information, calculate lane line deviation distance and deviation speed, finally, the warning level is judged in combination with the calculation results of both, and the corresponding warning information is sent;The system includes information acquisition module, detection module and early warning module;The lane line deviation early warning method proposed in the application calculates the deviation distance and deviation speed of lane line and vehicle simultaneously, and the warning level is divided into multiple levels in combination with both, the deviation condition can be reasonably judged, and the corresponding alarm information is sent according to different conditions, which improves driving comfort while ensuring driving safety.

Description

Technical Field

[0001] The present invention relates to the field of automatic driving assistance for commercial vehicles, and in particular to a multi-camera lane departure warning method and system for commercial vehicles. Background Art

[0002] With the development of intelligent, electronic and high-end commercial vehicles, new demands have been put forward for automatic driving assistance functions in specific driving environments. For example, when commercial vehicles are equipped with lane departure warning functions, during long-term driving on highways, misjudgment caused by the driver's visual fatigue can be detected in time by the visual perception lane departure warning system, and an alarm will be issued to alert the driver, greatly reducing the incidence of traffic accidents.

[0003] Currently, lane departure warning systems are mostly installed on passenger cars, or the methods used on passenger cars are simply transplanted to commercial vehicles, which cannot meet the characteristics of the longer body length of commercial vehicles. The key to the lane departure warning system lies in the accuracy of lane line recognition and the correct judgment of the distance changes between the lane line and the vehicle body. However, the current lane line recognition method is easily affected by uneven lighting, lane line defects or obstruction by the preceding vehicle, resulting in a low lane line recognition rate and unable to meet the requirements of lane departure warning. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-camera lane departure warning method and system for commercial vehicles to solve the problem that the current lane line recognition method is easily affected by uneven lighting, lane line defects or obstruction by the preceding vehicle, resulting in a low lane line recognition rate and failure to meet the requirements of lane line departure warning.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A multi-camera lane departure warning method for commercial vehicles, comprising

[0007] Multiple cameras on commercial vehicles collect road images in front, side, and rear, including lane line visual information;

[0008] Based on the collected images, deep learning is used to detect the lane line position of the front camera, and digital image processing is used to detect the lane line position of the rearview cameras on both sides;

[0009] The lane line deviation distance and deviation speed are calculated based on the lane line position information. Finally, the warning level is determined by combining the two calculation results and the corresponding warning information is sent.

[0010] Furthermore, the forward-looking camera collects images of the front side of the road; the cameras on both sides can simultaneously observe the lane lines and the vehicle body or trailer on one side; the multiple cameras on commercial vehicles are connected to the switch using a network cable, and the switch is connected to the computing and storage media in the vehicle via a network cable.

[0011] Furthermore, deep learning is used to detect the lane line position of the front camera:

[0012] A commercial vehicle collects a certain number of road images containing lane line information and completes the lane line position and category annotation; a deep convolutional neural network model is built, including a feature extraction layer and a classification layer. Using a row selection method, the model outputs the probability of each position in the selected row containing a lane line and the category to which the lane line belongs based on the last feature map; a loss function is designed, and calibration data is used to complete multiple iterative training of the model; the trained model is installed in the commercial vehicle's computing storage medium, and the program is run to perform real-time lane line point detection.

[0013] Furthermore, digital image processing is used to detect the lane line positions in the rearview camera images on both sides: the images are transformed into color space and Gaussian blurred; a bird's-eye view projection transformation is used to convert the pre-processed image into a top-down view; edge detection is used to detect possible lane lines or vehicle body boundaries; lane line extraction and vehicle body area extraction are performed, and the straight line groups after edge extraction are screened according to the lane line geometric characteristics to obtain unique lane line position information.

[0014] Furthermore, the lane deviation distance is calculated:

[0015] The lane line deviation distances of the front camera, left rear view camera, and right rear view camera are calculated respectively. First, a quadratic curve is fitted to the lane line detection points on both sides of the front camera obtained by the deep learning model, and the lane line centerline between the intersection of the two lane line fitting curves is calculated. Finally, the average of the sum of the pixel differences between each row where the lane line centerline is located and the camera centerline is calculated as the lane line deviation distance of the front camera. For the lane line lines in the rear view cameras on both sides, the average of the sum of the pixel differences between the lane line and the vehicle body or trailer boundary line in the intersection row is calculated as the lane line deviation distance of the front camera. Lane line deviation distance on both sides; the lane line deviation distance of the front camera has a direction. When deviating to the left, the deviation distance value is positive, and when deviating to the right, the value is negative, and the greater the deviation, the greater the absolute value; the deviation distance calculated by the left and right rearview cameras has no direction. When deviating to the left, the deviation distance of the left camera gradually decreases. When the value is 0, it indicates that the rear wheel is in a lane-crossing state, and then it will become a negative value. At this time, the deviation distance calculated by the right camera will gradually increase. The degree of deviation is determined by the threshold. The selection of the threshold requires the installation of cameras on both sides and calibration in the middle of the road.

[0016] Furthermore, the lane deviation speed is calculated: the lane deviation distances calculated for each frame by different cameras are stored in a fixed-length queue. When a new lane deviation distance is obtained, it is subtracted from the deviation distance at the previous moment to obtain the current lane deviation speed. The deviation speed at the historical moment is also calculated using this method. After the calculation is complete, the lane deviation distance for this frame is pushed into the queue and the first element in the queue is popped out.

[0017] Furthermore, the calculated lane departure distance and lane departure speed are combined to determine the warning level: the warning levels are divided into four categories: no departure, deviating, rapid departure, and dangerous departure. No warning information is sent for no departure, while for the other three categories, higher-frequency alarm sounds and larger-amplitude steering wheel vibrations are issued as the warning level increases.

[0018] When the lane line deviation distance is greater than the set maximum safe deviation threshold, the warning level is mainly determined based on the lane line deviation distance. At this time, the warning level should reach dangerous deviation; when the lane line deviation distance is less than the safe deviation threshold, based on the lane line deviation speed, when the speed value gradually increases compared with the historical moment, the warning level reaches rapid deviation. When the deviation speed value is small and fluctuates stably within the range, it indicates that deviation is occurring.

[0019] A multi-camera lane departure warning system for commercial vehicles, comprising:

[0020] The information acquisition module is used by multiple cameras on commercial vehicles to collect road images in front, side, and rear that include visual information about lane lines;

[0021] A detection module, which uses deep learning to detect the lane markings of the front camera based on the acquired images, and digital image processing to detect the lane markings of the rearview cameras on both sides;

[0022] The warning module is used to calculate the lane line deviation distance and deviation speed based on the lane line position information, and finally determine the warning level by combining the two calculation results and sending the corresponding warning information.

[0023] Compared with the prior art, the present invention has the following technical effects:

[0024] The present invention simultaneously collects images of the road in front of the vehicle and behind the left and right sides of the vehicle; outputs lane line position information in real time; calculates the deviation distance and deviation speed from the left and right lane lines based on the lane line position information and vehicle body information, selects a threshold based on experience, and determines the warning level; the lane line deviation alarm module sends alarm sounds of different frequencies and steering wheel vibrations of different amplitudes to the cockpit according to the warning level to warn the driver to respond quickly.

[0025] The lane departure warning method proposed in the present invention adopts a method of synchronously observing lane lines with multiple cameras, which is particularly suitable for commercial vehicles with longer lengths. It can calculate in real time whether there is deviation to one side of the lane line during the vehicle's straight driving process. It can also calculate in real time whether the rear body or trailer has crossed the line or is about to cross the line during the vehicle's turning process. It can effectively solve the potential risks of commercial vehicle drivers during fatigue driving or turning.

[0026] The lane departure warning method proposed in the present invention adopts a combination of deep learning model detection and digital image processing detection, which can effectively resolve the contradiction between detection accuracy and detection rate, and can output lane line position information in real time and accurately. When lane lines are detected using different camera positions and different detection methods, when one side of the camera or one method fails temporarily, the other method is used to provide redundancy, which can effectively ensure the stability and continuity of the warning function.

[0027] The lane departure warning method proposed in the present invention simultaneously calculates the deviation distance and deviation speed between the lane line and the vehicle, and combines the two to divide the alarm level into multiple levels. It can reasonably determine the deviation situation and issue corresponding alarm information according to different situations, thereby ensuring driving safety while improving driving comfort. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is an overall flow chart of the multi-camera lane departure warning method for commercial vehicles provided by an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of the installation positions and detection ranges of multiple cameras on a commercial vehicle provided by an embodiment of the present invention;

[0030] Figure 3 A processing flow chart of a deep learning-based lane detection method for a front-view camera provided in an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of lane line detection results from a commercial vehicle's front-view camera according to an embodiment of the present invention;

[0032] Figure 5 A processing flow chart of a lane line detection method based on digital image processing for rearview cameras on both sides provided in an embodiment of the present invention;

[0033] Figure 6 A schematic diagram of lane line detection results from a left rearview camera according to an embodiment of the present invention;

[0034] Figure 7 A flowchart of a method for calculating lane deviation distance and deviation speed of a commercial vehicle provided by an embodiment of the present invention;

[0035] Figure 8A flowchart of a method for determining lane departure warning levels for commercial vehicles provided by an embodiment of the present invention;

[0036] Figure 9 Block diagram of a commercial vehicle lane departure warning system provided by an embodiment of the present invention DETAILED DESCRIPTION

[0037] In order to enable professionals in this technical field to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0038] Figure 1 This is a general processing flow chart of a multi-camera lane departure warning method for commercial vehicles provided by an exemplary embodiment. Figure 1 As shown, it mainly includes the following steps.

[0039] In step S11 , the commercial vehicle collects and transmits multi-camera road images, including road surface and lane line images from the front camera, and vehicle body and lane line images from the left and right rear-view cameras.

[0040] In step S12, the original images collected in the previous step are processed separately to obtain lane line position information. For the front camera image, a deep learning method is used to detect the lane line point sets on the left and right sides of the vehicle in front. For the rear view camera images on both sides, a digital image processing method is used to identify the lane line and the position of the vehicle body on that side.

[0041] In step S13, the lane deviation distance and lane deviation speed of the commercial vehicle are calculated during the driving process.

[0042] In step S14, when the calculated lane deviation distance or deviation speed exceeds a set threshold, a lane deviation alarm is issued to the cab.

[0043] like Figure 2 As shown in the schematic diagram of the installation position and detection range of multiple cameras on a commercial vehicle, the multiple cameras proposed in this method are installed at different positions inside and outside the vehicle to obtain images of the road in front of or behind the vehicle at a certain pitch angle. The front-view camera is installed in the center of the windshield inside the vehicle and is required to be able to observe the lane markings on the left and right sides at the same time; the rear-view cameras on both sides are installed in the rearview mirrors of the vehicle respectively and are required to be able to observe the lane markings on that side and the vehicle body.

[0044] For the front camera image, a deep learning method is used to identify lane lines, such as Figure 3The processing flow chart of the deep learning lane line detection method for the front-view camera is shown, which mainly includes two parts: model training and model prediction.

[0045] Model training involves collecting real-world road data and marking lanes, building a deep neural network model, designing a lane classification loss function, and iteratively training network parameters through backpropagation. Road data containing lane information is collected from a real vehicle and then annotated using a LineStrip multi-point fitting method. Each lane is assigned a corresponding category after it is annotated: the left lane in the current lane is assigned category 1, the right lane is assigned category 2, and the remaining lanes do not need to be annotated.

[0046] The constructed deep neural network includes a feature extraction part and a classification part. The feature extraction part can use the residual network ResNet to extract a feature map containing lane line information. In particular, the row selection method is adopted in the design of the neural network. That is, in each pre-set row h in the extracted feature map, the possible lane line pixels in that row are found, and the row is divided into w grids to obtain the probability of each grid being a lane line pixel. The classification part uses a fully connected layer connection, and the feature map is expanded as the input dimension of the first fully connected layer. The output dimension is reasonably selected and passed through the activation function before being input into the second fully connected layer. The total dimension of the second layer output layer should be C×h×w, where C=2 is the number of lane lines. Finally, the fully connected layer output is dimensionalized to obtain the final model output.

[0047] The design model classification loss function should be consistent with the model output. The present invention can use the cross entropy loss function calculation method. Assuming that the model output probability is P and the true label is T, the loss function is: Loss = L CE (P,T)=-[ylogh θ (x)+(1-y)log(1-h θ (x))], where L CE is the cross entropy loss function, y is the true label, h θ (x) is the output of the model corresponding to the input image.

[0048] During the iterative training process, the training hyperparameters are pre-selected and set, including the learning rate, number of iterations, batch size, etc. Each time, the labeled images of the set batch size are input, the output probability is obtained through the forward propagation of the model, and the back-propagation gradient value is obtained through the loss function calculation. Finally, the weight parameter value of each layer is updated according to the learning rate to complete the model training.

[0049] During the model prediction phase, the trained model must be loaded first, and the road image captured by the forward-looking camera is passed into the model. This allows the lane line detection point in the current frame and the lane line category to which the point belongs to be obtained. The least squares method is used to perform quadratic curve fitting on multiple lane line detection points belonging to the same category to complete the real-time detection task of the forward-looking camera.

[0050] The schematic diagram of the lane line detection results of the front-view camera based on deep learning is as follows Figure 4 As shown, the vertical dotted line in the middle of the image is the camera centerline, the left and right fitting lines are quadratic fitting curves about the lane detection point set, the middle curve is the mean of the left and right lane line curves at the intersection of each row, and the mean of the distance between the camera centerline and the lane centerline is the lane line deviation distance to be calculated. In this figure, the vehicle deviates slightly to the left.

[0051] like Figure 5 The figure shows a process flow chart for lane marking detection based on digital image processing for dual rearview cameras. Step S51 is image preprocessing, including color space transformation and Gaussian blur. The color space transformation converts the original RGB color space image to the HSY color space to make the yellow or white lane markings more distinct.

[0052] S52 is the bird's-eye view conversion, also known as the inverse perspective transformation. After the camera is installed, its installation height h, pitch angle α, and yaw angle β must be measured, and the camera intrinsic parameters must be calibrated. The bird's-eye view conversion matrix is ​​calculated using the following formula:

[0053]

[0054] Where C u 、C v is the coordinate of the principal point of the image, f u 、f v is the focal length of the camera on the x-axis and y-axis. C1, S1, C2, and S2 are calculated based on the pitch angle α and the yaw angle β, as shown in the following formula:

[0055]

[0056]

[0057]

[0058]

[0059] Steps S53 and S54 are line extraction and line screening. In the line extraction step, the bird's-eye view image is first grayscaled. The gradient value of each pixel is then calculated and sorted. Points with larger gradient values ​​are more likely to represent significant boundaries and are therefore selected as seed points. When the gradient value surrounding a seed point exceeds a set threshold, the pixel is included in the region and becomes part of the line. Step S53 extracts multiple line regions. Step S54 screens these multiple line regions based on lane geometry (i.e., whether the line's aspect ratio exceeds a set threshold) and color (i.e., whether the average brightness of the original image corresponding to the line region exceeds a set threshold). The unique line that meets these criteria is retained as the lane detected by the side camera.

[0060] Step S55 requires extracting the commercial vehicle body or trailer boundary in the straight line area for use in the subsequent calculation of the lane line deviation distance. This step can be screened using similar digital image processing methods, or the boundary can be manually selected after the rearview cameras on both sides are installed.

[0061] Figure 6 This is a schematic diagram of the results obtained using the above digital image processing method for the left rearview camera. The original image has been preprocessed and converted from a bird's-eye view to a top view. Canny edge detection and line detection are used in the top view to obtain this result. There are still small straight line segments in the figure, and the lane line is not yet continuous. Therefore, the disconnected straight line segments need to be connected according to the geometric characteristics of the lane line, and all lines need to be screened. The final result should retain the unique lane line, as shown by the gray border in the figure, and the boundary of the commercial vehicle body, as shown by the black line in the figure.

[0062] Figure 7 This is a flow chart of a method for calculating lane departure distance and speed, provided by an exemplary embodiment of the present invention. For the left and right lane line fitting curves calculated by the forward-looking camera using a deep learning method, if the ordinate of the right lane line curve in the pixel coordinate system is also contained within the left lane line, the point is retained; otherwise, the point is discarded. The intersection of the two lane lines is calculated, and the intersection point is the lane centerline. When the camera is mounted in the center of a commercial vehicle, the vehicle centerline serves as the camera centerline. The average pixel distance between each lane centerline and the camera centerline is calculated as the lane departure distance for that frame. This value is then pushed into the lane departure distance storage queue to calculate the lane departure speed.

[0063] Assume that the fixed-length queue is filled with X = [x1, x2, x3, x4, x5, x6, x7, x8, x9, x 10 ], the lane line deviation distance of the current frame is x 11, at the same time, pop the first element x1 from the queue, move the remaining elements forward in sequence, and push this deviation distance into the queue as the 10th element.

[0064] At the same time, based on the lane line deviation distance queue, calculate the lane line deviation speed. Use x2 - x1, x3 - x2, x4 - x3, x5 - x4, x6 - x5, X7 - x6, x8 - x7, x9 - x8, x 10 - x9 as the deviation speed at historical moments, and use x 11 - x 10 as the deviation speed at the current moment. When the lane line deviation speed storage queue is full, pop the first element from the queue, move the remaining elements forward in sequence, and push this deviation speed into the queue as the 10th element.

[0065] This method is equally applicable to the calculation process of the lane line deviation distance of the two side rearview cameras. When calculating the deviation distance, just use the average pixel distance of the intersection of the lane line straight line and the vehicle body boundary straight line.

[0066] Figure 8 This is the flowchart of the method for judging the lane line deviation warning level of commercial vehicles provided by the embodiments of the present invention. The present invention divides the lane line deviation warning level into four levels from safe to urgent according to the lane line deviation distance and lane line deviation speed calculated above, including not deviated, deviating, quickly deviating, and dangerously deviating, and different frequencies of beeping alarms and different amplitudes of steering wheel vibrations should be sent to the cab according to different warning levels.

[0067] Set two thresholds for the lane line deviation distance calculated by the front view camera. A1 is the first threshold, A2 is the second threshold, and A1 < A2. Set two thresholds for the lane line deviation speed calculated. B1 is the first threshold, B2 is the second threshold, and B1 < B2. First, determine the current deviation direction according to the sign of the lane line deviation distance. When the value is positive, it indicates that the vehicle is on the left side of the lane line; otherwise, it deviates to the right. When the lane line deviation distance is not greater than the first threshold A1 or the lane line deviation speed is not greater than the first threshold B1, it indicates that the commercial vehicle is not deviated during driving; when the lane line deviation distance is greater than the first threshold A1 and the lane line deviation speed is greater than the first threshold B1, and does not exceed the second thresholds A2 and B2, the vehicle is in the deviating state; when the lane line deviation speed exceeds the second threshold B2, but the lane line deviation distance does not exceed A2, the vehicle is in the quickly deviating state; when the lane line deviation distance exceeds the second threshold A2, the vehicle is in the dangerously deviating state. When selecting the thresholds, the deviation driving process can be simulated. For example, when the vehicle head presses on the line on one side, record the deviation distance at this time as the second threshold A2.

[0068] The lane line deviation threshold level judgment method calculated by the rearview cameras on both sides is similar to that of the front-view camera. The lane line deviation distance value of the side rearview camera indicates the distance between the lane line and the vehicle body boundary. When this value is less than the threshold, it indicates that the vehicle body or trailer has deviated and is deviating toward the lane line on that side, and a warning information of the corresponding level should be issued.

[0069] When calculating the lane departure warning level, the front wheel angle can be weighted to update the departure speed in real time. The front wheel angle is simply calculated by dividing the steering wheel angle by the steering gear ratio. The steering wheel angle can be obtained from the vehicle's CAN bus or directly from a steering angle sensor. When the departure speed and the front wheel angle are in the same direction, the alarm is triggered early, and the warning level can be increased if the front wheel angle is excessive. When the departure speed and the front wheel angle are opposite, indicating a trend toward corrective deviation, the warning level can be lowered.

[0070] The present invention also provides a system suitable for commercial vehicle lane departure warning method, such as Figure 9 The block diagram of the lane departure warning system for commercial vehicles provided in an embodiment of the present invention mainly includes an information collection module, a detection module and a warning module.

[0071] The information acquisition module includes a front-view camera installed in the center of the commercial vehicle and rear-view cameras installed at the positions of the rearview mirrors on both sides, which simultaneously collect road images containing lane line information; the lane line detection module receives original images from three perspectives, and uses the above-mentioned deep learning method to detect the left and right lane lines for the front-view camera images, and uses the above-mentioned digital image processing method to detect the lane lines on that side and the vehicle body boundary for the rear-view cameras on both sides; the warning module includes a deviation distance and deviation speed calculation module, an alarm level determination module and a lane line departure alarm module; the deviation distance and deviation speed calculation module calculates the lane line position information of each camera separately and performs a stack loop storage operation; the lane line departure warning level determination module judges the vehicle driving status based on the calculated lane line deviation distance and speed according to a pre-set threshold, and divides the warning level into four levels; finally, the lane line departure alarm module executes a buzzer alarm and steering wheel vibration operation according to different levels.

[0072] The technical method and system implementing the method described in the present invention allow vehicles to travel normally in the center of the road. This is particularly true for commercial vehicles with long wheelbases. Warnings can be issued promptly when an unsafe lane deviation occurs, or when the vehicle body exceeds the lane boundary when turning, the driver can be reminded to observe and drive. The present invention combines deep learning with digital image processing, which, compared to traditional methods, improves lane detection accuracy while meeting the real-time requirements of detection. The present invention combines calculation of deviation distance with speed, and categorizes lane deviation warning levels into four levels. This allows for timely transmission of corresponding level warnings to the cab at the onset of a deviation trend, reducing unnecessary alarms and improving driving comfort while ensuring safety.

[0073] The division of system modules in the embodiments of the present invention is illustrative and represents only one logical functional division. Actual implementations may employ different division methods. Furthermore, the functional modules in the various embodiments of the present invention may be integrated into a single processor, exist as separate physical components, or be integrated into a single module. These integrated modules may be implemented in either hardware or software functional modules.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A multi-camera lane departure warning method for commercial vehicles, characterized by: include Multiple cameras on commercial vehicles collect road images in front, side, and rear, including lane line visual information; Based on the collected images, deep learning is used to detect the lane line position of the front camera, and digital image processing is used to detect the lane line position of the rearview cameras on both sides; Calculate the lane deviation distance and speed based on the lane position information, and finally determine the warning level by combining the two calculation results and send the corresponding warning information; Deep learning is used to detect the lane position of the front-view camera, including: A commercial vehicle collects a certain number of road images containing lane information and labels the lane locations and categories. A deep convolutional neural network model is built, including a feature extraction layer and a classification layer. Using row selection, the model outputs the probability of a lane and the lane category at each position in the selected row based on the final feature map. A loss function is designed, and the model is trained multiple times using calibration data. The trained model is then loaded into the commercial vehicle's computing storage medium, and the program is run to detect lane points in real time. Calculate lane deviation distance, including: The lane line deviation distances of the front camera, left rear view camera, and right rear view camera are calculated respectively. First, a quadratic curve is fitted to the lane line detection points on both sides of the front camera obtained by the deep learning model, and the lane line centerline between the intersection of the two lane line fitting curves is calculated. Finally, the average of the sum of the pixel differences between each row where the lane line centerline is located and the camera centerline is calculated as the lane line deviation distance of the front camera. For the lane line lines in the rear view cameras on both sides, the average of the sum of the pixel differences between the lane line and the vehicle body or trailer boundary line in the intersection row is calculated as the lane line deviation distance of the front camera. Lane line deviation distance on both sides; the lane line deviation distance of the front camera has a direction. When deviating to the left, the deviation distance value is positive, and when deviating to the right, the value is negative, and the greater the deviation, the greater the absolute value; the deviation distance calculated by the left and right rearview cameras has no direction. When deviating to the left, the deviation distance of the left camera gradually decreases. When the value is 0, it indicates that the rear wheel is in a lane-crossing state, and then it will become a negative value. At this time, the deviation distance calculated by the right camera will gradually increase. The degree of deviation is determined by the threshold. The selection of the threshold requires the installation of cameras on both sides and calibration in the middle of the road.

2. A multi-camera lane departure warning method for commercial vehicles according to claim 1, characterized in that: The forward-looking camera captures images in front of the commercial vehicle; the cameras on both sides can simultaneously observe the lane lines and the vehicle body or trailer on one side; the multiple cameras on the commercial vehicle are connected to the switch using a network cable, and the switch is connected to the computing and storage media in the vehicle via a network cable.

3. The multi-camera lane departure warning method for commercial vehicles according to claim 1, characterized in that: Digital image processing detects the lane line positions in the rearview camera images on both sides: the images are transformed into color space and Gaussian blurred; a bird's-eye view projection transformation is used to convert the pre-processed image into a top-down view; edge detection is used to detect possible lane lines or vehicle body boundaries; lane line extraction and vehicle body area extraction are performed, and the edge-extracted straight line groups are filtered according to the lane line geometric characteristics to obtain unique lane line position information.

4. The multi-camera lane departure warning method for commercial vehicles according to claim 1, characterized in that: Calculate lane departure speed: The lane departure distances calculated for each frame by different cameras are stored in a fixed-length queue. When a new lane departure distance is obtained, it is subtracted from the previous departure distance to obtain the current lane departure speed. This calculation method is also used to obtain the historical departure speed. After the calculation is complete, the lane departure distance for this frame is pushed into the queue and the first element in the queue is popped out.

5. The multi-camera lane departure warning method for commercial vehicles according to claim 1 is characterized in that: The calculated lane departure distance and lane departure speed are combined to determine the warning level: the warning levels are categorized into four types: no departure, deviating, rapid departure, and dangerous departure. No warning is issued for no departure, while for the other three types, higher-frequency alarm sounds and larger-amplitude steering wheel vibrations are issued as the warning level increases. When the lane line deviation distance is greater than the set maximum safe deviation threshold, the warning level is mainly determined based on the lane line deviation distance. At this time, the warning level should reach dangerous deviation; when the lane line deviation distance is less than the safe deviation threshold, based on the lane line deviation speed, when the speed value gradually increases compared with the historical moment, the warning level reaches rapid deviation. When the deviation speed value is small and fluctuates stably within the range, it indicates that deviation is occurring.

6. A multi-camera lane departure warning system for commercial vehicles, characterized by: include: The information acquisition module is used by multiple cameras on commercial vehicles to collect road images in front, side, and rear that include visual information about lane lines; A detection module, which uses deep learning to detect the lane markings of the front camera based on the acquired images, and digital image processing to detect the lane markings of the rearview cameras on both sides; The warning module is used to calculate the lane deviation distance and speed based on the lane position information, and finally determine the warning level by combining the two calculation results and sending the corresponding warning information; Deep learning is used to detect the lane position of the front-view camera, including: A commercial vehicle collects a certain number of road images containing lane information and labels the lane locations and categories. A deep convolutional neural network model is built, including a feature extraction layer and a classification layer. Using row selection, the model outputs the probability of a lane and the lane category at each position in the selected row based on the final feature map. A loss function is designed, and the model is trained multiple times using calibration data. The trained model is then loaded into the commercial vehicle's computing storage medium, and the program is run to detect lane points in real time. Calculate lane deviation distance, including: The lane line deviation distances of the front camera, left rear view camera, and right rear view camera are calculated respectively. First, a quadratic curve is fitted to the lane line detection points on both sides of the front camera obtained by the deep learning model, and the lane line centerline between the intersection of the two lane line fitting curves is calculated. Finally, the average of the sum of the pixel differences between each row where the lane line centerline is located and the camera centerline is calculated as the lane line deviation distance of the front camera. For the lane line lines in the rear view cameras on both sides, the average of the sum of the pixel differences between the lane line and the vehicle body or trailer boundary line in the intersection row is calculated as the lane line deviation distance of the front camera. Lane line deviation distance on both sides; the lane line deviation distance of the front camera has a direction. When deviating to the left, the deviation distance value is positive, and when deviating to the right, the value is negative, and the greater the deviation, the greater the absolute value; the deviation distance calculated by the left and right rearview cameras has no direction. When deviating to the left, the deviation distance of the left camera gradually decreases. When the value is 0, it indicates that the rear wheel is in a lane-crossing state, and then it will become a negative value. At this time, the deviation distance calculated by the right camera will gradually increase. The degree of deviation is determined by the threshold. The selection of the threshold requires the installation of cameras on both sides and calibration in the middle of the road.

Images