Large transport vehicle violation detection method and system based on improved Mask RCNN model

Abstract

The invention belongs to the technical field of intelligent traffic supervision, and discloses a large transport vehicle violation detection method and system based on an improved Mask RCNN model, and the method comprises the steps: 1, collecting image data of a right lane line and wheels shielded by a vehicle body due to a collection view angle when a large transport vehicle runs in an actual scene; 2, manually marking a left lane line and head and tail wheels of a vehicle in the acquired image, and carrying out classification identification to construct a data set; 3, improving a traditional Mask RCNN model: adding a feature enhancement path from bottom to top and a transverse connection path between different levels in an original FPN network; a bounding box and confidence output unit is removed, and a violation judgment and calculation module is newly added; removing bounding box loss in an original loss function, and increasing edge loss to optimize mask segmentation precision; 4, training a data set by using the improved model and storing a training weight; and 5, detecting images which do not participate in training by using the training weight, and outputting and visualizing a violation detection result.

Description

Technical Field

[0001] This invention belongs to the field of intelligent traffic monitoring technology, specifically relating to a method and system for detecting violations by large transport vehicles based on an improved Mask RCNN model. Background Technology

[0002] With the acceleration of industrial modernization, the role of heavy-duty transportation in infrastructure and other fields is becoming increasingly crucial. Heavy-duty transport vehicles typically have no fewer than 6 axles and no fewer than 1 flatbed, and are characterized by their large weight and size. The compliance of these vehicles during transportation is directly related to road traffic safety.

[0003] Currently, the detection of violations by oversized transport vehicles mainly relies on manual inspections or traditional computer vision methods, but these methods have significant limitations. Manual inspections are inefficient, costly, and difficult to implement 24 / 7 monitoring. Traditional methods perform poorly from certain perspectives. Due to the large size of oversized vehicles, when the image acquisition equipment is positioned slightly above and to the right of the driver's viewpoint, the right lane lines and right wheels are easily obscured, and only the left lane lines and left wheels can be seen. Furthermore, some images exhibit severe perspective distortion, leading to a sharp drop in detection accuracy.

[0004] Existing object detection models such as Mask R-CNN have shortcomings in handling such scenarios: First, the traditional FPN network structure does not sufficiently fuse multi-scale features, resulting in the loss of low-level details such as lane line edges and wheel contours during transmission, making it difficult to effectively detect low-contrast features such as blurred lane lines; second, there is a lack of a dedicated judgment module for violations by large vehicles, making it impossible to effectively combine the relationship between lane lines and wheel positions for violation judgment, leading to a high rate of false positives and false negatives. Therefore, there is an urgent need for a method and system for detecting violations by large transport vehicles that can adapt to special perspectives, improve the detection capability of small objects by improving the feature fusion structure, and combine a dedicated violation judgment module to solve the difficulty of violation detection in occluded scenarios. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method and system for detecting violations in large transport vehicles based on an improved Mask RCNN model. Its core innovations are: (1) To address the problem of insufficient multi-scale feature fusion in traditional FPN networks, this invention proposes to add a bottom-up K2 to K5 feature enhancement path and lateral connections between different levels in the original FPN network, shortening the path for high-level features to acquire low-level features, and strengthening the transmission capability of low-level detail features such as lane line edges and wheel outlines, thereby improving the detection accuracy of small targets and low-contrast features; (2) To address the difficulty in determining violations caused by right-side occlusion under special perspectives, this invention proposes to add a violation determination and calculation module at the end of the model. Through the process of "detection mask - adaptive cropping - parallel line drawing - joint determination of position and distance", this invention solves the problem of determining violations when the right lane line and wheel are not visible. It has been verified that the false judgment rate of this invention is <5%, while the false judgment rate of the traditional method is >20% in occluded scenarios; (3) By improving the loss function, the edge accuracy of the segmentation results is improved, and the influence of perspective transformation is reduced by adaptive cropping of the image, thereby improving the detection accuracy. After verification with 1,000 test images, a detection accuracy of 93.8% was achieved, and the processing time per frame was reduced to 0.15 seconds, meeting the requirements for real-time detection.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A method for detecting violations in heavy transport vehicles based on an improved Mask RCNN model, the method comprising:

[0008] Step 1: Collect image data of the oversized transport vehicle in the actual driving scene, where the right lane line and wheels are obscured by the vehicle body due to the shooting perspective;

[0009] Step 2: Manually label the left lane lines and the front and rear wheels of vehicles in the acquired images, and classify and label them to build a dataset;

[0010] Step 3: Improve the traditional Mask R-CNN model: Add a bottom-up feature enhancement path and lateral connection paths between different layers to the original FPN network; remove the bounding box and confidence output units and add a violation detection and calculation module; remove the bounding box loss in the original loss function and add edge loss to optimize the mask segmentation accuracy;

[0011] Step 4: Apply the improved model to train the dataset and save the training weights;

[0012] Step 5: Use the training weights to detect images that were not included in the training, and output and visualize the violation detection results.

[0013] Preferably, the heavy transport vehicle in step 1 refers to a transport vehicle with no less than 6 axles, no less than 1 flatbed, and characterized by its large weight and large size.

[0014] The acquisition angle refers to the position where the image acquisition device is located 15°-30° above and to the right of the driver's view. Due to the large size of the oversized transport vehicle, the right lane line and the right wheel of the vehicle are obscured by the vehicle body in the image acquired from this angle, and only the left lane line and the left wheel of the vehicle are shown.

[0015] Preferably, in step 3, the functions of the violation determination and calculation module include:

[0016] To reduce computational load and improve processing speed, the original RGB image is converted to grayscale. The training weights are used to detect the left lane line and the vehicle's first and last wheels. The lane line area is filled with a red mask, the first wheel area is filled with a green mask, and the last wheel area is filled with a yellow mask.

[0017] Based on the boundary coordinates of the lane lines and the masked areas of the vehicle's front and rear wheels, an adaptive cropping strategy is used to remove the edge areas with severe perspective transformation around the image, while retaining the core areas of the vehicle body and lane lines, thereby improving the accuracy of subsequent distance calculations.

[0018] Index the coordinates of the lowest point of the wheel mask area, and draw parallel lines at a fixed angle to the bottom edge of the image from each lowest point to the left. Mark the intersection of the lowest point, the parallel line and the edge of the lane mask area as a solid circle.

[0019] A preliminary violation determination is made based on the relative position of the lowest point of the wheel and the lane line mask area. If the lowest point of the wheel is located to the right of the lane line mask area, the actual distance from the lowest point of the wheel to the inside of the lane line is further calculated.

[0020] Based on the preliminary violation assessment results and the actual distance calculation results, a final violation determination is made, and the detection results are displayed.

[0021] Preferably, in step 3, considering that vehicle wheels may intersect with lane lines in real-world scenarios, and that external factors such as lighting, shadows, rain, and snow can cause the wheel area to become blurred, affecting recognition, an edge loss method is introduced in the mask branch to improve the edge accuracy of the wheel segmentation result and the accuracy of the segmentation mask. The improved loss function is as follows:

[0022] L total =αL cls +βL mask +γL edge

[0023] Where α, β, and γ are the respective loss weights; L cls L is the classification loss, used for category prediction; mask For masking loss, the loss is calculated only for the target region, and the background region is ignored; L edge This indicates marginal loss.

[0024] Calculate L edge Edge loss methods include:

[0025] The labeled image is converted into a binary segmentation map of the target mask. Then, the predicted mask and the target mask output from the mask branch are used as input and convolved with a 3×3 Sobel operator. The mean square error of the convolution result is then calculated to obtain the edge loss, which is calculated using the following formula:

[0026]

[0027] Where y represents the labeled target edge; Indicates the predicted edge.

[0028] Preferably, if the lowest point of the wheel is located to the right of the lane line mask area, the calculation of the actual distance from the lowest point of the wheel to the inner side of the lane line includes:

[0029] First, calculate the scale of the first and last wheel areas separately. The calculation formulas for the scale K1 of the first wheel area and the scale K2 of the last wheel area are as follows:

[0030]

[0031] Among them l c S represents the actual width of the lane line. lfp S represents the pixel width of the lane lines in the first-round area. lep The pixel width of the lane line in the rear wheel area; the pixel width of the lane line refers to the total number of pixels along the parallel line segments on the lane line.

[0032] Calculate the actual distance l from the first and last wheels of the vehicle to the lane line respectively. d l e The calculation formula is:

[0033] l d =K1×S wfp

[0034] l e =K2×S wep

[0035] Where S wfp S is the pixel width from the lowest point of the first round of green mask area to the right side of the lane line; wep The pixel width from the lowest point of the yellow mask area of ​​the rear wheel to the right side of the lane line;

[0036] The compliance range is (0, l) a -l b ), where l a With l b These are the actual lane width and axle length, respectively.

[0037] Preferably, the violation determination conditions include:

[0038] If the lowest point of the wheel is located within the lane line mask area, the vehicle is deemed to be driving illegally by crossing the line.

[0039] If the lowest point of the wheel is located to the left of the lane line mask area, it is determined that the driver is illegally crossing the lane to the left.

[0040] If the lowest point of the wheel is located to the right of the lane line mask area, then calculate the actual distance l from the vehicle's first and last wheels to the lane line. d l e If l d l e If all traffic is within the compliant range, then the vehicle is not in violation; otherwise, the vehicle is illegally driving by crossing the right lane line or crossing lane boundaries.

[0041] Preferably, the displayed detection results include:

[0042] The detection mask results for lane lines and the first and last wheels;

[0043] The result of marking parallel lines and solid dots;

[0044] If any violations are found, the specific violation information of the tested vehicle will be displayed;

[0045] If the lowest point of the wheel is to the right of the lane line mask area, the actual distance calculation result from the wheel to the inside of the lane line will be displayed.

[0046] The present invention also provides a violation detection system for oversized transport vehicles based on an improved Mask RCNN model. The system is used to implement the aforementioned method and includes: a data acquisition module, a detection mask module, a violation detection module, and a result display module.

[0047] The data acquisition module is used to acquire RGB images of heavy transport vehicles in actual driving scenarios and convert the original images into grayscale images;

[0048] The detection mask module is used to receive grayscale images, detect the left lane line and the front and rear wheels of the vehicle, and fill the area with different colored masks respectively. At the same time, it adaptively crops the image to reduce the impact of perspective transformation.

[0049] The violation detection module receives the cropped core area image, indexes the lowest point of different wheel mask areas, and draws a parallel line at a fixed angle to the bottom edge of the image, extending to the left from each lowest point. The intersection of the lowest point and the parallel line with the edge of the lane line area is marked with a solid small dot. Based on the positional relationship between the lowest point of the wheel and the lane line mask area, as well as the actual distance of each vehicle from the inside of the lane line, the violation of the detected vehicle is determined.

[0050] The results display module is used to receive the final violation judgment result and the actual distance value. It overlays the detection mask results of lane lines and the first and last wheels on the image, displays the identification results of parallel lines and solid small circles, displays the specific violation information in red text with a black transparent rectangle as the background at the bottom center of the image, and displays the actual distance calculation result of the wheel to the inside of the lane line near the lowest point of the wheel.

[0051] Preferably, the data acquisition module includes: an image acquisition unit and an image conversion unit;

[0052] The image acquisition unit is used to acquire image data of heavy transport vehicles when they are driving in actual scenes, where the right lane line and the right wheel are obscured by the vehicle body due to the acquisition perspective.

[0053] The image conversion unit is used to convert the acquired raw image into a grayscale image.

[0054] Preferably, the detection mask module includes: a detection unit, a mask unit, and a cropping unit;

[0055] The detection unit is used to apply the training weights of the improved Mask RCNN model to detect lane lines and the front and rear wheels of vehicles in the grayscale image;

[0056] The masking unit is used to fill the detected lane lines, first wheel and rear wheel areas with masks of different colors respectively;

[0057] The cropping unit is used to remove redundant areas with severe perspective transformations around the grayscale image with different colored mask regions, retaining only the most critical parts of the tested vehicle and lane lines.

[0058] Preferably, the violation detection module includes: an indexing unit, a drawing and marking unit, a calculation unit, and a judgment unit;

[0059] The detection unit is used to apply the training weights of the improved Mask RCNN model to detect lane lines and the front and rear wheels of vehicles in the grayscale image;

[0060] The masking unit is used to fill the detected lane lines, first wheel and rear wheel areas with masks of different colors respectively;

[0061] The cropping unit is used to remove redundant areas with severe perspective transformations around the grayscale image with different colored mask regions, retaining only the most critical parts of the tested vehicle and lane lines.

[0062] Preferably, the result display module includes: a mask display unit, a parallel line and dot display unit, a violation information display unit, and an actual distance result display unit;

[0063] The mask display unit is used to display the detection status of lane lines and the first and last wheels, as well as the filling status of different colored masks in each area;

[0064] The parallel lines and dots display unit is used to display the parallel lines drawn from the lowest point of each wheel, as well as the markings of the lowest point and the intersection of the parallel lines with the edge lines of the lane area.

[0065] The violation information display unit is used to display the type of vehicle violation, and displays the specific violation information in red text with a black transparent rectangle as the background in the middle of the bottom of the image;

[0066] The actual distance result display unit is used to display the calculated actual distance from the wheel to the inside of the lane line.

[0067] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0068] This invention enhances the multi-scale feature fusion capability by adding a bottom-up feature enhancement path and lateral connections to the original FPN network, thereby reducing the possibility of false detections and improving the detection effect for smaller targets and complex scenes.

[0069] This invention adds a violation detection and calculation module to the end of the Mask RCNN model, which can detect violations in images where lane lines and wheels are obscured due to special acquisition perspectives, with a detection accuracy of up to 93.8%, and has good practical application value.

[0070] This invention improves the edge accuracy of segmentation results by modifying the loss function, reduces the amount of computation by converting grayscale images, and reduces the impact of perspective transformation by adaptively cropping images. This can reduce the possibility of false detection and false negative detection by the model, and at the same time shorten the time for the model to process a single frame image detection task to less than 0.15s, meeting the requirements of real-time detection.

[0071] This invention provides traffic management departments and logistics companies with a practical technical solution for detecting violations by oversized transport vehicles, effectively improving the management efficiency and transportation safety of oversized transport vehicles, and reducing road infrastructure maintenance costs and traffic accident rates. Attached Figure Description

[0072] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a flowchart of a method for detecting violations in heavy transport vehicles based on an improved Mask RCNN model, according to an embodiment of the present invention.

[0074] Figure 2 This is a diagram of the improved FPN network structure according to an embodiment of the present invention;

[0075] Figure 3 This is a diagram of the network structure for detecting violations in large transport vehicles based on the improved Mask RCNN model in an embodiment of the present invention.

[0076] Figure 4 This is a schematic diagram of the actual detection process of a violation detection method for oversized transport vehicles based on an improved Mask RCNN model according to an embodiment of the present invention. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] Example 1

[0080] like Figure 1 As shown, this invention provides a method for detecting violations in large transport vehicles based on an improved Mask RCNN model, comprising the following steps:

[0081] Step 1: Collect image data of the oversized transport vehicle in the actual driving scene, where the right lane line and wheels are obscured by the vehicle body due to the shooting perspective;

[0082] Step 2: Manually label the left lane lines and the front and rear wheels of vehicles in the acquired images, and classify and label them to build a dataset;

[0083] Step 3: Improve the traditional Mask R-CNN model: Add a bottom-up feature enhancement path and lateral connection paths between different layers to the original FPN network; remove the bounding box and confidence output units and add a violation detection and calculation module; remove the bounding box loss in the original loss function and add edge loss to optimize the mask segmentation accuracy;

[0084] Step 4: Apply the improved model to train the dataset and save the training weights;

[0085] Step 5: Use the training weights to detect images that were not included in the training, and output and visualize the violation detection results.

[0086] In this embodiment, the heavy transport vehicle in step 1 refers to a transport vehicle with no less than 6 axles, no less than 1 flatbed, and characterized by its large weight and large size.

[0087] The acquisition angle refers to the position where the image acquisition device is located 15°-30° above and to the right of the driver's view. Due to the large size of the oversized transport vehicle, the right lane line and the right wheel of the vehicle are obscured by the vehicle body in the image acquired from this angle, and only the left lane line and the left wheel of the vehicle are shown.

[0088] The data collected included images of oversized transport vehicles in actual driving scenarios, showing the right lane lines and wheels obscured by the vehicle body due to the viewing angle. This included images of various types of oversized transport vehicles, encompassing a large number of single-bed trailers and a small number of multi-bed trailers, totaling 3600 images. The image acquisition process also included different weather conditions such as sunny days, rain, and snow, as well as different time periods such as morning and afternoon, and images from different angles within the same viewpoint.

[0089] In this embodiment, step 2 includes:

[0090] Step 2.1: Create the closed outline of the polygon marker instance object;

[0091] Step 2.2: Select the category and assign an ID number to the labeled object;

[0092] Step 2.3: Save the annotations as a JSON-formatted tag file to complete the annotation of a single image;

[0093] Step 2.4: Repeat steps 2.1 to 2.3 until all images are labeled;

[0094] Step 2.5: Convert all JSON tag files to XML format;

[0095] Step 2.6: Divide the training set, validation set, and test set according to the preset ratio of 8:1:1, and construct a standard VOC format large vehicle dataset containing XML label files, original image files, dataset partitioning files, and instance segmentation mask files.

[0096] As a preferred technical solution, step 2.2, which involves selecting the category and assigning an ID number to the labeled object, specifically includes the following process:

[0097] If the lane line is not obscured, it is directly classified as a laneline and its ID is set to 1. If the lane line is obscured, causing the original complete lane line to be divided into multiple segments, since it belongs to the same instance being divided into multiple parts, when labeling this type of obscured lane line, each segment of the lane line should be classified as a laneline and the ID of each segment of the lane line should be set to 1.

[0098] When labeling large vehicles on a single-wheel platform, classify the vehicle's first wheel as category fwheel and set its ID to 2; classify the rear wheel as category ewheel and set its ID to 3.

[0099] When labeling large vehicles with multiple pallets, since each pallet has both front and rear wheels, the front wheels of multiple pallets are classified as category fwheel, and the IDs of multiple different front wheels are set as 2, 4, ... respectively; at the same time, the rear wheels of multiple pallets are classified as category ewheel, and the IDs of multiple different rear wheels are set as 3, 5, ... respectively.

[0100] In this embodiment, step 3 includes:

[0101] Step 3.1: Based on the original FPN network structure of the traditional Mask RCNN model, add a bottom-up K2 to K5 feature enhancement path structure and construct a lateral connection path from low level to high level;

[0102] Step 3.2: Remove the bounding boxes and confidence display units at the end of the traditional Mask R-CNN model, and add a violation detection and calculation module;

[0103] Step 3.3: Remove the bounding box loss from the original loss function and add edge loss to optimize the mask segmentation accuracy.

[0104] As a preferred technical solution, step 3.1 adds a bottom-up K2 to K5 feature enhancement path structure to the original FPN network structure of the traditional Mask R-CNN model, and constructs a lateral connection path from low to high level. The specific process is as follows:

[0105] like Figure 2 As shown, a feature enhancement path from bottom to top from K2 to K5 is constructed on the right side of the original FPN structure P5-P2 path, and there are horizontal connections with the P layer such as P2→K2 and P3→K3.

[0106] In the original FPN structure, low-level detailed features are easily lost when passed to higher levels, leading to low accuracy in small object detection. The newly added K-layer, with its bottom-up path, shortens the transmission path of low-level features to the output, allowing for more complete preservation of detailed features such as wheel edges and lane line contours. Therefore, feature enhancement through the K-path improves edge feature extraction and transmission, resulting in clearer segmentation masks for wheels and lane lines and reducing false positives caused by blurred edges. Furthermore, the addition of lateral connections between the P and K layers allows for more flexible interaction and complementarity of features from different levels (C / P / K), breaking the original unidirectional dependency.

[0107] As a preferred technical solution, the violation determination and calculation module in step 3.2 has the following functions:

[0108] To reduce computational load and improve processing speed, the original RGB image is converted to grayscale. The training weights are used to detect the left lane line and the vehicle's first and last wheels. The lane line area is filled with a red mask, the first wheel area is filled with a green mask, and the last wheel area is filled with a yellow mask.

[0109] Based on the boundary coordinates of the lane lines and the masked areas of the vehicle's front and rear wheels, an adaptive cropping strategy is used to remove the edge areas with severe perspective transformation around the image, while retaining the core areas of the vehicle body and lane lines, thereby improving the accuracy of subsequent distance calculations.

[0110] Index the coordinates of the lowest point of the wheel mask area, and draw parallel lines at a fixed angle to the bottom edge of the image from each lowest point to the left. Mark the intersection of the lowest point, the parallel line and the edge of the lane mask area as a solid circle.

[0111] A preliminary violation determination is made based on the relative position of the lowest point of the wheel and the lane line mask area. If the lowest point of the wheel is located to the right of the lane line mask area, the actual distance from the lowest point of the wheel to the inside of the lane line is further calculated.

[0112] Based on the preliminary violation assessment results and the actual distance calculation results, a final violation determination is made, and the detection results are displayed.

[0113] As a preferred technical solution, the principle of the adaptive pruning strategy is as follows:

[0114] First, index the boundary coordinates of the lane lines and the masked areas of the vehicle's first and last wheels. Taking the masked area of ​​the last wheel as an example, extract its minimum boundary coordinates in the x direction, and perform operations to subtract 200 from the x value and 400 from the y value to generate new coordinates. Then, along the horizontal dimension, trim the areas where the y value is less than the new coordinate y value, and along the vertical dimension, trim the areas where the x value is less than the new coordinate x value, thereby completing the accurate removal of redundant areas.

[0115] As a preferred technical solution, the specific process for determining the coordinates of the lowest point of the indexed wheel mask area is as follows:

[0116] Extract the y-values ​​of all pixel coordinates within the green and yellow mask regions respectively; based on the characteristic that "the y-value of the pixel coordinates of the lowest point in the mask region is the largest", select and determine the coordinates of the lowest point in each of the green and yellow mask regions, thereby completing the indexing process of the coordinates of the lowest point in the first and last wheel mask regions.

[0117] As a preferred technical solution, the step of drawing parallel lines extending to the left from each lowest point at a fixed angle to the bottom edge of the image, and marking the intersection of the lowest point, the parallel lines, and the edge of the lane line mask area as solid dots, includes the following process:

[0118] Starting from the coordinates of each lowest point, draw white parallel lines to the left at an angle of 18° to the bottom edge of the image. If the parallel lines do not touch the red mask area of ​​the lane lines during their extension, continue extending to the left to the edge of the image, and draw a small solid white dot at the beginning of each parallel line. If they touch the red mask area of ​​the lane lines, the starting point of the parallel line segment is the coordinates of the lowest point of the green or yellow mask area, and the ending point is the left edge of the red mask area of ​​the lane lines. Simultaneously, draw a small solid white dot at both ends of the parallel line segment. If the parallel lines simultaneously touch the right edge of the red mask area of ​​the lane lines during their extension, draw an additional small solid white dot at the intersection of the parallel line and the right edge. The radius of each small solid white dot is 5.

[0119] As a preferred technical solution, if the lowest point of the wheel is located to the right of the lane line mask area, the actual distance from the lowest point of the wheel to the inner side of the lane line is calculated, including the following process:

[0120] First, calculate the scale of the first and last wheel areas separately. The calculation formulas for the scale K1 of the first wheel area and the scale K2 of the last wheel area are as follows:

[0121]

[0122] Among them l c S represents the actual width of the lane line. lfp S represents the pixel width of the lane lines in the first-round area. lep The pixel width of the lane line in the rear wheel area; the pixel width of the lane line refers to the total number of pixels along the parallel line segments on the lane line.

[0123] Calculate the actual distance l from the first and last wheels of the vehicle to the lane line respectively. d l e The calculation formula is:

[0124] l d =K1×S wfp

[0125] l e =K2×S wep

[0126] Where S wfp S is the pixel width from the lowest point of the first round of green mask area to the right side of the lane line; wep The pixel width from the lowest point of the yellow mask area of ​​the rear wheel to the right side of the lane line;

[0127] The compliance range is (0, l) a -l b ), where l a With l b These are the actual lane width and axle length, respectively.

[0128] As a preferred technical solution, the violation determination conditions include:

[0129] If the lowest point of the wheel is located within the lane line mask area, the vehicle is deemed to be driving illegally by crossing the line.

[0130] If the lowest point of the wheel is located to the left of the lane line mask area, it is determined that the driver is illegally crossing the lane to the left.

[0131] If the lowest point of the wheel is located to the right of the lane line mask area, then calculate the actual distance l from the vehicle's first and last wheels to the lane line. d l e If l d l e If all traffic is within the compliant range, then the vehicle is not in violation; otherwise, the vehicle is illegally driving by crossing the right lane line or crossing lane boundaries.

[0132] As a preferred technical solution, the displayed detection results include:

[0133] The detection mask results for lane lines and the first and last wheels;

[0134] The result of marking parallel lines and solid dots;

[0135] If any violations are found, the specific violation information of the tested vehicle will be displayed;

[0136] If the lowest point of the wheel is to the right of the lane line mask area, the actual distance calculation result from the wheel to the inside of the lane line will be displayed.

[0137] As a preferred technical solution, step 3.3 removes the bounding box loss from the original loss function and adds an edge loss to optimize the mask segmentation accuracy. The improved loss function is as follows:

[0138] L total =αL cls +βL mask +γL edge

[0139] Where α, β, and γ are the respective loss weights; L clsL is the classification loss, used for category prediction; mask For masking loss, the loss is calculated only for the target region, and the background region is ignored; L edge This indicates marginal loss.

[0140] Calculate L edge Edge loss methods include:

[0141] The labeled image is converted into a binary segmentation map of the target mask. Then, the predicted mask and the target mask output from the mask branch are used as input and convolved with a 3×3 Sobel operator. The mean square error of the convolution result is then calculated to obtain the edge loss, which is calculated using the following formula:

[0142]

[0143] Where y represents the labeled target edge; Indicates the predicted edge.

[0144] As a preferred technical solution, the purpose of removing the bounding box loss from the original loss function is that the original bounding box and confidence score can interfere with the index of the lowest point of the wheel mask region, which can easily lead to errors in violation judgment. By removing the bounding box loss, this interference can be avoided, the accuracy of actual distance calculation can be improved, and the accuracy of violation judgment can be guaranteed.

[0145] As a preferred technical solution, the purpose of adding edge loss is to take into account that in real-world scenarios, vehicle wheels may intersect with lane lines, and external factors such as lighting, shadows, rain, and snow can cause the wheel area to become blurred, affecting recognition. Therefore, an edge loss method is introduced into the mask branch to improve the edge accuracy of the wheel segmentation result and the accuracy of the segmentation mask.

[0146] In this embodiment, step 4 includes:

[0147] Step 4.1: Configure the hardware and software environment required for training the improved Mask R-CNN model and set the training parameters;

[0148] Step 4.2: Apply the improved Mask RCNN model to train the large vehicle dataset in standard VOC format and save the training weights.

[0149] As a preferred technical solution, step 4.1, configuring the hardware and software environment required for training the improved Mask R-CNN model and setting the training parameters, specifically includes:

[0150] The hardware environment includes:

[0151] CPU: Intel(R)Core(TM)i7-14700;

[0152] GPU: NVIDIA GeForce RTX 4060Ti (16G);

[0153] Memory: 32GB;

[0154] Hard drive: WD Blue SN580 1TB SSD;

[0155] The software environment includes:

[0156] Operating environment: Ubuntu 20.04;

[0157] Development frameworks: Python 3.9.10, PyTorch 2.3.0, CUDA 12.5, PyTorchvision 0.18.0, PyTorchaudio 2.3.0;

[0158] Editor: PyCharm 2023.2.8;

[0159] Training parameters include:

[0160] Number of trainable layers: 3;

[0161] Number of target categories detected excluding background (num-classes): 3;

[0162] Dataset iterations (epochs): 100;

[0163] Learning rate (lr): 0.01;

[0164] Learning rate decay nodes (lr-steps): epochs=30 and epochs=60;

[0165] Learning rate decay factor (lr-gamma): 0.1;

[0166] Weight decay (wd): 0.0001;

[0167] Batch size for a single training round: 32;

[0168] like Figure 3The diagram shows the network structure for detecting violations in heavy transport vehicles based on the improved Mask RCNN model. This structure mainly consists of ResNet50+Improved FPN, Region Proposal Network (RPN), Region of Interest Alignment (ROIAlign), Fully Convolutional Network (FCN) + Mask branch, Fully connected layers, and a violation determination and calculation module. Starting from the image input, the network goes through feature extraction, region proposal, feature alignment, mask generation, and violation determination, ultimately outputting the violation detection result. Each part works in sequence to complete the task of detecting vehicle violations.

[0169] This invention addresses the problem of difficulty in determining vehicle violations under special acquisition perspectives, where lane lines and wheels on one side of the image are obscured and there is significant perspective distortion. By improving the Mask R-CNN model to detect lane lines and the front and rear wheels of the vehicle on one side of the mask, and combining the positional relationship between the lowest point of the wheel and the lane line mask area with the actual distance from the wheel to the inside of the lane line, this invention can achieve accurate detection of vehicle violations. This enriches and improves the theoretical system of violation detection in the field of intelligent transportation, and provides new ideas and methods for subsequent related research.

[0170] Example 2

[0171] The present invention also provides a violation detection system for oversized transport vehicles based on an improved Mask RCNN model, comprising: a data acquisition module, a detection mask module, a violation detection module, and a result display module;

[0172] The data acquisition module is used to collect images of heavy transport vehicles driving in actual scenarios and convert the original images into grayscale images;

[0173] The detection mask module is used to detect the left lane line and the front and rear wheels of the vehicle in the grayscale image, and fill the area with a mask of different colors respectively. At the same time, it adaptively crops the image to reduce the impact of perspective transformation.

[0174] The violation detection module indexes the lowest point of different masked areas of the wheels. Starting from each lowest point, it draws parallel lines extending to the left at a fixed angle to the bottom edge of the image. The intersection of these parallel lines with the lane line area edge is marked with a solid small dot. Based on the positional relationship between the lowest point of the wheel and the lane line mask area, as well as the actual distance of each vehicle from the inside of the lane line, the module determines the violation status of the detected vehicle.

[0175] The results display module is used to present the detection mask results of lane lines and the first and last wheels in the image, display the identification results of parallel lines and solid small circles, display the specific violation information in red text with a black transparent rectangle as the background in the middle of the bottom of the image, and display the actual distance calculation result of the wheel to the inside of the lane line near the lowest point of the wheel.

[0176] In this embodiment, the data acquisition module includes: an image acquisition unit and an image conversion unit;

[0177] The image acquisition unit is used to acquire image data of heavy transport vehicles when they are driving in actual scenes, where the right lane line and the right wheel are obscured by the vehicle body due to the acquisition perspective.

[0178] The image conversion unit is used to convert the acquired raw image into a grayscale image.

[0179] In this embodiment, the mask detection module includes: a detection unit, a mask unit, and a cropping unit;

[0180] The detection unit is used to apply the training weights of the improved Mask RCNN model to detect lane lines and the front and rear wheels of vehicles in the grayscale image;

[0181] The masking unit is used to fill the detected lane lines, first wheel, and rear wheel areas with masks of different colors respectively;

[0182] The cropping unit is used to remove redundant areas with severe perspective transformations around the grayscale image with different colored mask regions, retaining only the most critical parts of the tested vehicle and lane lines.

[0183] In this embodiment, the violation detection module includes: an indexing unit, a drawing identifier unit, a calculation unit, and a judgment unit;

[0184] The index unit is used to find the coordinates of the lowest point within the different colored mask areas of the vehicle's front and rear wheels;

[0185] The drawing marker unit is used to draw parallel lines extending to the left from each lowest point at a fixed angle to the bottom edge of the image. The intersection of the parallel lines with the lane line mask area and the lowest point are marked as solid small dots.

[0186] The calculation unit is used to calculate the actual distance from the lowest point of the wheel to the inside of the lane line mask area;

[0187] The judgment unit is used to determine the vehicle violation based on the positional relationship between the lowest point of the wheel and the lane line mask area, combined with the actual distance from the wheel to the inside of the lane line.

[0188] In this embodiment, the result display module includes: a mask display unit, a parallel line and dot display unit, a violation information display unit, and an actual distance result display unit;

[0189] The mask display unit is used to display the detection status of lane lines and the first and last wheels, as well as the filling status of different colored masks in each area;

[0190] The parallel lines and dots display unit is used to display the parallel lines drawn starting from the lowest point of each wheel, as well as the positions of the lowest point and the intersection of the parallel lines with the edge lines of the lane line area;

[0191] The violation information display unit is used to display the type of vehicle violation, and displays the specific violation information in red text with a black transparent rectangle as the background in the middle of the bottom of the image;

[0192] The actual distance result display unit is used to display the calculated actual distance from the wheel to the inside of the lane line.

[0193] This invention can provide practical technical solutions for traffic management departments and logistics companies, effectively improve the management efficiency and transportation safety of heavy-duty transport vehicles, reduce the maintenance costs of road infrastructure, promote the sustainable and standardized development of the heavy-duty transport industry, and has important practical significance for promoting the high-quality development of my country's economy and society.

[0194] Example 3

[0195] This embodiment takes the collection of images of oversized transport vehicles traveling on highway bridges for violation detection as an example. Figure 4 This diagram illustrates the actual detection process of a violation detection method for oversized transport vehicles based on an improved Mask RCNN model. It includes a data collector with a data acquisition module, a central server containing a detection mask module and a violation detection module, a display, a cloud server, etc.

[0196] The system includes data acquisition devices deployed on bridge sections that collect real-time images of heavy transport vehicles and simultaneously convert the original images into grayscale images. A central server, based on an improved Mask R-CNN model, uses a mask detection module to identify the left lane line and the vehicle's first and last wheels in the grayscale image, fills in corresponding color masks, and adaptively crops the image. A violation detection module then indexes the lowest point of the wheel mask, draws a line parallel to the bottom edge of the image at an 18° angle, and marks the intersection of this line with the lane line area's edge and the lowest point as a solid dot. The violation is determined by combining the positional relationship between the lowest point of the wheel and the lane line mask area with the actual distance. A display screen shows the violation detection results and the actual distance calculation results. When a vehicle is detected violating traffic rules, the cloud server automatically records the basic information of the violating vehicle and the details of the violation, and simultaneously uploads the collected image data for subsequent management by the monitoring platform.

[0197] This invention can help build an intelligent and automated system for detecting violations in large vehicles, and plays a positive role in promoting the development of smart transportation and the digital transformation of the transportation industry.

[0198] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for detecting violations in heavy transport vehicles based on an improved Mask RCNN model, characterized in that, The method includes: Step 1: Collect image data of the oversized transport vehicle in the actual driving scene, where the right lane line and wheels are obscured by the vehicle body due to the shooting perspective; Step 2: Manually label the left lane lines and the front and rear wheels of vehicles in the acquired images, and classify and label them to build a dataset; Step 3: Improve the traditional Mask R-CNN model: Add a bottom-up feature enhancement path and lateral connection paths between different layers to the original FPN (Feature Pyramid Network); remove the bounding box and confidence output units and add a violation detection and calculation module; remove the bounding box loss in the original loss function and add edge loss to optimize the mask segmentation accuracy; Step 4: Apply the improved model to train the dataset and save the training weights; Step 5: Use the training weights to detect images that were not included in the training, and output and visualize the violation detection results.

2. The method according to claim 1, characterized in that, In step 1: The term "heavy transport vehicle" refers to a transport vehicle with no fewer than 6 axles and no fewer than 1 flatbed, and which is characterized by its large weight and large size. The acquisition angle refers to the position where the image acquisition device is located 15°-30° above and to the right of the driver's view. Due to the large size of the oversized transport vehicle, the right lane line and the right wheel of the vehicle are obscured by the vehicle body in the image acquired from this angle, and only the left lane line and the left wheel of the vehicle are shown.

3. The method according to claim 1, characterized in that, In step 3, the functions of the violation determination and calculation module include: To reduce computational load and improve processing speed, the original RGB image is converted to grayscale. The training weights are used to detect the left lane line and the vehicle's first and last wheels. The lane line area is filled with a red mask, the first wheel area is filled with a green mask, and the last wheel area is filled with a yellow mask. Based on the boundary coordinates of the lane lines and the masked areas of the vehicle's front and rear wheels, an adaptive cropping strategy is used to remove the edge areas with severe perspective transformation around the image, while retaining the core areas of the vehicle body and lane lines, thereby improving the accuracy of subsequent distance calculations. Index the coordinates of the lowest point of the wheel mask area, and draw parallel lines at a fixed angle to the bottom edge of the image from each lowest point to the left. Mark the intersection of the lowest point, the parallel line and the edge of the lane mask area as a solid circle. A preliminary violation determination is made based on the relative position of the lowest point of the wheel and the lane line mask area. If the lowest point of the wheel is located to the right of the lane line mask area, the actual distance from the lowest point of the wheel to the inside of the lane line is further calculated. Based on the preliminary violation assessment results and the actual distance calculation results, a final violation determination is made, and the detection results are displayed.

4. The method according to claim 1, characterized in that, In step 3, considering that vehicle wheels may intersect with lane lines in real-world scenarios, and that external factors such as lighting, shadows, rain, and snow can cause the wheel area to become blurred, affecting recognition, an edge loss method is introduced into the mask branch to improve the edge accuracy of the wheel segmentation result and the accuracy of the segmentation mask. The improved loss function is as follows: L total =αL cls +βL mask +γL edge Where α, β, and γ are the respective loss weights; L cls L is the classification loss, used for category prediction; mask For masking loss, the loss is calculated only for the target region, and the background region is ignored; L edge This indicates marginal loss. Calculate L edge Edge loss methods include: The labeled image is converted into a binary segmentation map of the target mask. Then, the predicted mask and the target mask output from the mask branch are used as input and convolved with a 3×3 Sobel operator. The mean square error of the convolution result is then calculated to obtain the edge loss, which is calculated using the following formula: Where y represents the labeled target edge; Indicates the predicted edge.

5. The method according to claim 3, characterized in that, If the lowest point of the wheel is located to the right of the lane line mask area, then the actual distance from the lowest point of the wheel to the inside of the lane line is calculated, including: First, calculate the scale of the first and last wheel areas separately. The calculation formulas for the scale K1 of the first wheel area and the scale K2 of the last wheel area are as follows: Among them l c S represents the actual width of the lane line. lfp S represents the pixel width of the lane lines in the first-round area. lep The pixel width of the lane line in the rear wheel area; the pixel width of the lane line refers to the total number of pixels along the parallel line segments on the lane line. Calculate the actual distance l from the first and last wheels of the vehicle to the lane line respectively. d l e The calculation formula is: l d =K1×S wfp l e =K2×S wep Where S wfp S is the pixel width from the lowest point of the first round of green mask area to the right side of the lane line; wep The pixel width from the lowest point of the yellow mask area of ​​the rear wheel to the right side of the lane line; The compliance range is (0, l) a -l b ), where l a With l b These are the actual lane width and axle length, respectively.

6. The method according to claim 3, characterized in that, The circumstances under which a violation is determined include: If the lowest point of the wheel is located within the lane line mask area, the vehicle is deemed to be driving illegally by crossing the line. If the lowest point of the wheel is located to the left of the lane line mask area, it is determined that the driver is illegally crossing the lane to the left. If the lowest point of the wheel is located to the right of the lane line mask area, then calculate the actual distance l from the vehicle's first and last wheels to the lane line. d l e If l d l e If all traffic is within the compliant range, then the vehicle is not in violation; otherwise, the vehicle is illegally driving by crossing the right lane line or crossing lane boundaries.

7. The method according to claim 3, characterized in that, The displayed detection results include: The detection mask results for lane lines and the first and last wheels; The result of marking parallel lines and solid dots; If any violations are found, the specific violation information of the tested vehicle will be displayed; If the lowest point of the wheel is to the right of the lane line mask area, the actual distance calculation result from the wheel to the inside of the lane line will be displayed.

8. A violation detection system for oversized transport vehicles based on an improved Mask RCNN model, the system being used to implement the method described in any one of claims 1-7, characterized in that, The system includes: a data acquisition module, a detection mask module, a violation detection module, and a result display module; The data acquisition module is used to acquire RGB images of heavy transport vehicles in actual driving scenarios and convert the original images into grayscale images; The detection mask module is used to receive grayscale images, detect the left lane line and the front and rear wheels of the vehicle, and fill the area with different colored masks respectively. At the same time, it adaptively crops the image to reduce the impact of perspective transformation. The violation detection module receives the cropped core area image, indexes the lowest point of different wheel mask areas, and draws a parallel line at a fixed angle to the bottom edge of the image, extending to the left from each lowest point. The intersection of the lowest point and the parallel line with the edge of the lane line area is marked with a solid small dot. Based on the positional relationship between the lowest point of the wheel and the lane line mask area, as well as the actual distance of each vehicle from the inside of the lane line, the violation of the detected vehicle is determined. The results display module is used to receive the final violation judgment result and the actual distance value. It overlays the detection mask results of lane lines and the first and last wheels on the image, displays the identification results of parallel lines and solid small circles, displays the specific violation information in red text with a black transparent rectangle as the background at the bottom center of the image, and displays the actual distance calculation result of the wheel to the inside of the lane line near the lowest point of the wheel.

9. The system according to claim 8, characterized in that, The data acquisition module includes: an image acquisition unit and an image conversion unit; The image acquisition unit is used to acquire image data of heavy transport vehicles when they are driving in actual scenes, where the right lane line and the right wheel are obscured by the vehicle body due to the acquisition perspective. The image conversion unit is used to convert the acquired raw image into a grayscale image.

10. The system according to claim 8, characterized in that, The detection mask module includes: a detection unit, a mask unit, and a cropping unit; The detection unit is used to apply the training weights of the improved Mask RCNN model to detect lane lines and the front and rear wheels of vehicles in the grayscale image; The masking unit is used to fill the detected lane lines, first wheel and rear wheel areas with masks of different colors respectively; The cropping unit is used to remove redundant areas with severe perspective transformations around the grayscale image with different colored mask regions, retaining only the most critical parts of the tested vehicle and lane lines.

11. The system according to claim 8, characterized in that, The violation detection module includes: an indexing unit, a drawing and marking unit, a calculation unit, and a judgment unit; The index unit is used to find the coordinates of the lowest point within the different colored mask areas of the vehicle's front and rear wheels; The drawing marker unit is used to draw parallel lines extending to the left from each lowest point at a fixed angle to the bottom edge of the image. The intersection of the parallel lines with the lane line mask area and the lowest point are marked as solid small dots. The calculation unit is used to calculate the actual distance from the lowest point of the wheel to the inside of the lane line mask area; The judgment unit is used to determine the vehicle violation based on the positional relationship between the lowest point of the wheel and the lane line mask area, combined with the actual distance from the wheel to the inside of the lane line.

12. The system according to claim 8, characterized in that, The results display module includes: a mask display unit, a parallel line and dot display unit, a violation information display unit, and an actual distance result display unit; The mask display unit is used to display the detection status of lane lines and the first and last wheels, as well as the filling status of different colored masks in each area; The parallel lines and dots display unit is used to display the parallel lines drawn from the lowest point of each wheel, as well as the markings of the lowest point and the intersection of the parallel lines with the edge lines of the lane area. The violation information display unit is used to display the type of vehicle violation, and displays the specific violation information in red text with a black transparent rectangle as the background in the bottom center of the image; The actual distance result display unit is used to display the calculated actual distance from the wheel to the inside of the lane line.

Images