Vehicle violation detection method and detection device, vehicle-mounted electronic equipment

Abstract

The present invention discloses a method and device for detecting vehicle violations, as well as an on-board electronic device. The detection method includes: controlling a camera to capture images of the road in front of the current vehicle based on preset calibration parameters; determining whether other vehicles on the road ahead have crossed the lane during driving based on the relative position corresponding to each road image obtained by analyzing multiple road images; if other vehicles have crossed the lane, extracting the original temporal lane line features of the complete crossing process of the other vehicles; and using a trained behavior judgment model based on the original temporal lane line features to determine whether the crossing of the lane by the other vehicles is an illegal lane change. The present invention solves the technical problem in related technologies that, when performing illegal lane change detection, lane line attributes are not incorporated into the entire temporal process of the vehicle crossing the lane, resulting in a lack of temporal characteristics in the features, and thus lower robustness of the detection results.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis, and in particular to a method and device for detecting vehicle violations, and a vehicle-mounted electronic device. Background Art

[0002] With economic growth and advancements in infrastructure, the number of vehicles on our roads is increasing. However, this heavy traffic also leads to an increasing number of traffic accidents, causing significant harm to individuals, families, and society. Therefore, there is a growing demand to improve travel safety by regulating driver behavior.

[0003] Among the numerous traffic accidents, illegal lane changes on solid lines account for a considerable number. Reducing illegal lane changes on solid lines will effectively reduce the occurrence of traffic accidents. To effectively reduce the number of drivers' illegal lane changes on solid lines, providing a real-time monitoring system for drivers' driving behavior and timely reporting of illegal lane changes has become an effective way to effectively reduce the number of drivers' illegal lane changes on solid lines.

[0004] By installing cameras to capture real-time video of vehicles, the vehicle's trajectory is analyzed and pre-defined rules are used to determine whether the monitored vehicle has violated lane changes. This technology has two core issues: lane line attributes and vehicle trajectory. Based on this, the current detection method is to install high-speed cameras in fixed locations (such as intersections) and calibrate lane line attributes. By analyzing vehicle trajectories and combining them with the established rules for solid lines, it can detect lane changes on solid lines. However, this method requires a lot of manpower for calibration, and once installed, the camera can only monitor a fixed area. Later solutions began to install cameras on vehicles and use statistical features to partially solve the calibration problem. Specifically, after pre-processing the image, the lane line color distribution is calculated, and a specific judgment threshold is set to determine the lane line attributes. The vehicle position is then determined to determine whether the vehicle has crossed the line. These two factors are combined to determine lane changes on solid lines.

[0005] In related technologies, there are two commonly used methods for detecting illegal lane changes: the first is a solid-line lane change detection method based on fixed scene parameter calibration. By calibrating the attributes of the line shape under the camera, combined with the vehicle's running trajectory and the pre-set direction and distance parameters of the illegal lane change, artificial rules are used to determine whether there is any illegal crossing of the line during the entire process; the second is a solid-line lane change detection method based on traditional feature statistical learning. The line shape type is determined by the distribution of black and white dots of the lane line samples after statistical preprocessing, and then the lane position deviation parameters are used, such as crossing a lane, to determine the vehicle crossing the line, and the solid-line lane change is comprehensively determined.

[0006] However, both of the aforementioned illegal lane change detection methods have significant problems. The first method has two drawbacks: First, the method requires extensive calibration, parameter presetting, and manual rule development before operation, which is labor-intensive. Furthermore, after installation, it can only monitor fixed scenes, and recalibration and manual rule design may be required after movement. The second method has the following problems: First, the color distribution-based statistics are affected by environmental factors such as lighting and stains, resulting in low robustness. Second, the use of the vehicle's own deviation parameters can lead to false detections on curves. Finally, this solution does not incorporate lane line attributes into the entire temporal process of the vehicle crossing the lane, resulting in a lack of temporal characteristics and low robustness.

[0007] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0008] Embodiments of the present invention provide a method and device for detecting vehicle violations, as well as an on-board electronic device, to at least solve the technical problem in related technologies that, when performing illegal lane change detection, lane line attributes are not incorporated into the entire temporal process of the vehicle crossing the line, resulting in a lack of temporal characteristics and low robustness of the detection results.

[0009] According to one aspect of an embodiment of the present invention, a method for detecting vehicle violations is provided, comprising: controlling a camera to capture road images in front of a current vehicle based on preset calibration parameters; determining whether other vehicles on the road ahead have crossed the lane during driving based on the relative position corresponding to each road image obtained by analyzing multiple road images; if the other vehicles have crossed the lane, extracting original temporal lane line features during the complete crossing process of the other vehicles; and determining whether the crossing of the lane by the other vehicles is an illegal lane change using a trained behavior determination model based on the original temporal lane line features.

[0010] Optionally, based on the relative position corresponding to each road image obtained by analyzing multiple road images, it is determined whether other vehicles on the road ahead have crossed the line during driving, including: combining the vehicle information and lane line information obtained by analyzing each road image, determining the relative position between the other vehicles and the lane line contained in each road image, wherein the relative position is the ratio of the distance from the vehicle to the lane line to the width of the vehicle; and comprehensively analyzing the relative positions of multiple road images to determine whether other vehicles on the road ahead have crossed the line during driving.

[0011] Optionally, in combination with the vehicle information and lane line information obtained by parsing each of the road images, the relative positions between the other vehicles and the lane lines contained in each of the road images are determined, including: parsing each of the road images to calibrate at least one vehicle detection frame and at least one lane line detection frame, wherein each of the vehicle detection frames corresponds to one other vehicle, and each of the lane line detection frames corresponds to one lane line; and determining the vehicle information and the lane line information of each of the other vehicles through at least one vehicle detection frame and at least one lane line detection frame.

[0012] Optionally, the relative positions of multiple road images are integrated to determine whether other vehicles on the road ahead have crossed the line during driving, including: if the absolute value of the first relative position is less than a first threshold, and there is a positive or negative change between the second relative position and the first relative position, then the other vehicles have crossed the line during driving; otherwise, the other vehicles have not crossed the line during driving.

[0013] Optionally, by parsing the vehicle information and lane line information obtained from each of the road images, the relative positions between the other vehicles contained in each of the road images and the lane lines are determined, including: based on the vehicle information and the lane line information, obtaining the point position of the preset point in the vehicle detection area and the point position of the intersection of the vehicle lane line, wherein the intersection of the vehicle lane line includes the intersection of the horizontal straight line where the preset point in the vehicle detection area is located and the lane line or the lane line extension line, or the vertical intersection of the point position of the preset point in the vehicle detection area and the lane line or the lane line extension line; according to the point position of the preset point in the vehicle detection area, the point position of the intersection of the vehicle lane line and the width value of the vehicle detection frame, the first formula is used to determine the relative position.

[0014] Optionally, the vehicle information and the lane line information of each of the other vehicles are determined through at least one vehicle detection frame and at least one lane line detection frame, including: based on the lane line detection frame, analyzing the lane line direction vector and lane line attribute information to obtain the lane line information; based on the vehicle detection frame, analyzing the vehicle position, vehicle height and vehicle width of each of the other vehicles to obtain the vehicle information.

[0015] Optionally, based on the lane line detection frame, the lane line direction vector and lane line attribute information are analyzed to obtain the lane line information, including: inputting the lane line detection frame into a lane line model, using the lane line model to analyze the attribute feature vector of the lane line to obtain the lane line information, wherein the lane line model is a pre-trained model, and during the training process, a preset classification framework is used to extract a lane line training sample set according to the lane line marking position, and the lane line training sample set is input into a convolutional neural network system to train the detection network to obtain the lane line model, or, the lane line information is obtained by a traditional image processing method, wherein the traditional image processing method is to obtain the lane line marking position after performing image preprocessing on the lane line detection frame, and analyze the lane line direction vector and lane line attribute information based on the lane line marking position to obtain the lane line information, wherein the image preprocessing includes: binarization processing, image denoising, and lane line segmentation.

[0016] Optionally, extracting the original temporal lane line features of the complete crossing process of the other vehicle includes: determining the approach frame and the end frame of the complete crossing process based on the relative positions of the multiple road images, and obtaining a timing diagram of the complete crossing process; extracting the original temporal lane line features from the timing diagram.

[0017] Optionally, determining the approach frame and end frame of the complete crossing process includes: if the absolute value of the relative position is less than a second threshold, determining it as the approach frame of the complete crossing process; if the absolute value of the relative position is greater than a third threshold, and there is a positive or negative change between the relative position and the relative position corresponding to the approach frame, determining it as the end frame of the complete crossing process.

[0018] Optionally, based on the original temporal lane line features, a trained behavior judgment model is used to determine whether the crossing behavior of the other vehicles is an illegal lane change, including: inputting the original temporal lane line features into a solid line confidence network to obtain a temporal solid line confidence; inputting the relative position and the temporal solid line confidence contained in the corresponding frame during the complete crossing process into the behavior judgment model to determine whether the crossing behavior of the other vehicles is an illegal lane change, wherein the behavior judgment model is classified as an integrated behavior judgment model or a temporal behavior judgment model.

[0019] Optionally, when the behavior determination model is an integrated behavior determination model, the detection method includes: combining the relative positions contained in each time series corresponding frame of the complete crossing process, classifying and counting the confidence of the time series solid line, and obtaining a histogram feature; inputting the histogram feature into the integrated behavior determination model to determine whether the crossing behavior of the other vehicle is an illegal lane change.

[0020] Optionally, in combination with the relative positions contained in each time series corresponding frame of the complete line crossing process, the time series solid line confidence is classified and counted to obtain a histogram feature, including: in combination with the relative positions contained in each time series corresponding frame of the complete line crossing process and a preset segmentation threshold, the time series solid line confidence is divided into n confidence sets; for the classification confidence corresponding to each frame of road image in each confidence set, k-dimensional histogram features are extracted to obtain n k-dimensional histogram features; the obtained n k-dimensional histogram features are connected in series according to the time series relationship in the historical process to obtain n*k-dimensional histogram features.

[0021] Optionally, the integrated behavior determination model is a model obtained by performing integrated learning training using information entropy gain.

[0022] Optionally, when the behavior judgment model is a temporal behavior judgment model, the detection method includes: directly inputting the relative position and the temporal solid line confidence contained in the corresponding frame in the complete crossing process into the temporal behavior judgment model in a temporal arrangement to determine whether the crossing behavior of the other vehicle is an illegal lane change.

[0023] Optionally, the temporal behavior determination model is obtained by performing convolution calculation on a basic network and back-propagating training on the calculated cross entropy loss.

[0024] Optionally, the detection method further includes: based on the original time-series lane line features and the vehicle information in each of the road images, using a lane-crossing module to determine whether the lane-crossing behavior of the other vehicle is a lane change that allows crossing the line.

[0025] Optionally, based on the original temporal lane line features and the vehicle information in each of the road images, a cross-line module is used to determine whether the crossing behavior of the other vehicles is a lane change that can be crossed, including: controlling each sub-module of the cross-line module to use a sliding window of a preset temporal length to slide on the temporal lane line features to determine the linear category of the lane line, wherein the cross-line module includes a virtual and real double line sub-module, a bus area sub-module and other cross-line sub-modules; based on the vehicle information and the linear category of the lane line, each sub-module of the cross-line module is used to determine whether the crossing behavior of the other vehicles is a lane change that can be crossed.

[0026] Optionally, if the cross-line module is a virtual-real double-line classification module, the virtual-real double-line classification module is used to determine whether the cross-line behavior of the other vehicle is a lane change that can be crossed, including: using a sliding window with a preset time sequence length to slide on the original time sequence lane line feature to determine multiple confidence levels that the lane line is a virtual-real double line; calculating a first average value of the multiple confidence levels to determine whether the lane line is a virtual-real double line based on the first average value to obtain the linear category of the lane line; based on the vehicle crossing direction and the linear category of the lane line, determine whether the cross-line behavior of the other vehicle is a lane change that can be crossed.

[0027] Optionally, based on the vehicle crossing direction and the linear category of the lane line, determine whether the crossing behavior of the other vehicle is a lane change that can be crossed, including: if the vehicle crossing direction indicates that the crossing behavior of the other vehicle is to cross the line from the right side of the current lane line, and the linear category of the lane line is a left dashed and right solid line, determine that the crossing behavior of the other vehicle is not a lane change that can be crossed; if the vehicle crossing direction indicates that the crossing behavior of the other vehicle is to cross the line from the left side of the current lane line, and the linear category of the lane line is a left solid and right dashed line, determine that the crossing behavior of the other vehicle is not a lane change that can be crossed.

[0028] Optionally, if the cross-line module is a bus area classification module, the bus area classification module is used to determine whether the cross-line behavior of the other vehicles is a lane change that can be crossed, including: using a sliding window with a preset time series length to slide on the original time series lane line feature to determine multiple confidence levels that the lane line is the boundary line of the public vehicle area; calculating a second average value of the multiple confidence levels to determine whether the lane line is the boundary line of the public vehicle area based on the second average value to obtain the linear category of the lane line; based on the linear category of the lane line and the time information in the historical process, determine whether the cross-line behavior of the other vehicles is a lane change that can be crossed.

[0029] Optionally, based on the linear category of the lane line and the time information in the historical process, it is determined whether the crossing-line behavior of the other vehicle is a lane change that allows crossing the line, including: if the linear category of the lane line indicates that the lane line is the boundary line of the public vehicle area, a sliding window with a preset time series length is used to slide on the vehicle attribute characteristics of the other vehicle to determine whether the other vehicle is a public vehicle and obtain a judgment result; if the judgment result indicates that the other vehicle is not a public vehicle, it is determined whether the historical time period in the historical process is a time period for allowing driving; if the historical time period in the historical process is a time period for not allowing driving, it is determined that the crossing-line behavior of the other vehicle is not a lane change that allows crossing the line; if the historical time period in the historical process is a time period for allowing driving, it is determined that the crossing-line behavior of the other vehicle is a lane change that allows crossing the line.

[0030] Optionally, if the line-crossing module is another line-crossing module, whether the line-crossing behavior of the other vehicle is a lane change that can be crossed is determined by the other line-crossing module, including: using a sliding window of a preset time sequence length to slide on the original time sequence lane line feature to determine multiple confidence levels that the lane line is the edge of the line-crossing area; calculating a third average value of the multiple confidence levels; if the third average value of the multiple confidence levels is greater than a preset value, determining that the line-crossing behavior of the other vehicle is a lane change that can be crossed; if the third average value of the multiple confidence levels is less than or equal to a preset value, determining that the line-crossing behavior of the other vehicle is not a lane change that can be crossed.

[0031] Optionally, after using the cross-line module to determine whether the crossing-line behavior of the other vehicle is a lane change that allows crossing the line, the detection method also includes: if it is determined that the crossing-line behavior of the other vehicle is illegal, marking the illegal vehicle; issuing a violation prompt message; and reporting the vehicle violation information and vehicle information to the vehicle management platform.

[0032] Optionally, before controlling the camera to capture images of the road in front of the current vehicle based on preset calibration parameters, the detection method also includes: installing the camera on the front windshield of the current vehicle; calibrating the hood area and the horizon position according to the installation position of the camera; and determining the preset calibration parameters and image detection area based on the calibrated hood area and the horizon position.

[0033] Optionally, based on preset calibration parameters, the step of controlling the camera to capture the road image in front of the current vehicle includes: if the camera type is a telephoto camera and a wide-angle camera, adjusting the focus information when parsing the road image according to the field of view angle information of the camera, and adjusting the calibration information for calibrating the detection frames of other vehicles.

[0034] Optionally, the step of controlling the camera to capture the road image in front of the current vehicle includes: analyzing the ambient light parameters around the current vehicle; if the ambient light parameters are lower than a preset light threshold, using an infrared camera head to capture the road image in front of the current vehicle.

[0035] According to another aspect of an embodiment of the present invention, a device for detecting vehicle violations is also provided, including: a control unit for controlling a camera to capture road images in front of a current vehicle based on preset calibration parameters; a first judgment unit for determining whether other vehicles on the road ahead have crossed the line during driving based on the relative position corresponding to each road image obtained by analyzing multiple road images; an extraction unit for extracting original temporal lane line features of the complete crossing process of the other vehicles when the other vehicles have crossed the line; and a second judgment unit for determining whether the crossing of the line by the other vehicles is an illegal lane change based on the original temporal lane line features and a trained behavior judgment model.

[0036] According to another aspect of an embodiment of the present invention, a road vehicle is also provided, comprising: an on-board camera installed on the windshield in front of the vehicle, for collecting road images of the road ahead; and an on-board control unit connected to the on-board camera, for executing any one of the above-mentioned methods for detecting vehicle violations.

[0037] According to another aspect of an embodiment of the present invention, a vehicle-mounted electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods for detecting vehicle violations by executing the executable instructions.

[0038] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned methods for detecting vehicle violations.

[0039] This application can be applied to the real-time monitoring scenario of illegal lane changing behaviors of vehicles on the road. By monitoring the running trajectory of the vehicle in front, based on the time series state information of the vehicle throughout the entire process, the time series histogram features are extracted, and combined with the behavior judgment model, the solid line lane changing behavior is identified and reported, thereby helping drivers develop good driving habits and reducing the occurrence of traffic accidents caused by similar factors.

[0040] In this application, a camera is controlled to capture images of the road ahead of the current vehicle based on preset calibration parameters. Based on the relative position of each road image obtained by analyzing multiple road images, it is determined whether other vehicles on the road ahead have crossed the lane during their travel. If other vehicles have crossed the lane, the original time-series lane line features of the other vehicles' complete crossing process are extracted. Based on the original time-series lane line features, a trained behavior determination model is used to determine whether the other vehicles' crossing behavior is an illegal lane change. In this embodiment, based on the time-series information of the vehicle's lane change behavior during driving, the vehicle's lane crossing behavior is analyzed to determine whether it is an illegal lane change. The behavior determination model is then used to classify the vehicle's lane crossing behavior, determining whether it is illegal. The analyzed vehicle lane crossing behavior features are time-series, which significantly improves the robustness of the detection results. This solves the technical problem in related technologies where lane line attributes are not incorporated into the entire time-series process of the vehicle crossing, resulting in a lack of time-series features and low robustness of the detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0042] Figure 1 is a flow chart of an optional method for detecting vehicle traffic violations according to an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of an optional method for extracting a vehicle detection frame and a lane line detection frame according to an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of an optional method for calibrating relative positions of other vehicles and lane lines according to an embodiment of the present invention;

[0045] Figure 4 is an optional schematic diagram of time series histogram extraction according to an embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of an optional method of determining vehicle traffic violation behavior by using a line crossing behavior module according to an embodiment of the present invention;

[0047] Figure 6 is a schematic diagram of an optional method of using a wide-angle camera to analyze whether a vehicle has violated traffic regulations according to an embodiment of the present invention;

[0048] Figure 7 is a schematic diagram of an optional method for determining line crossing behavior taking into account different light intensities according to an embodiment of the present invention;

[0049] Figure 8 Schematic diagram of an optional device for detecting vehicle traffic violations according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0051] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0052] This application can be applied to various road monitoring scenarios or vehicle illegal lane change detection scenarios. It uses time-series information from the vehicle's driving process to determine whether the current lane crossing behavior is illegal. Specifically, a deep learning model is used to learn the lane line attributes throughout the entire illegal lane change process. Combined with the lane line attribute characteristics of the entire time-series process, a behavior judgment model is used to classify the final result to determine whether the current lane crossing is illegal, thereby helping drivers regulate their driving behavior. The following describes this application in conjunction with various embodiments.

[0053] Example 1

[0054] According to an embodiment of the present invention, an embodiment of a method for detecting vehicle violations is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0055] Figure 1 FIG. 1 is a flow chart of an optional method for detecting vehicle violations according to an embodiment of the present invention. Figure 1As shown, the method includes the following steps:

[0056] Step S102, based on preset calibration parameters, controlling the camera to capture an image of the road in front of the current vehicle;

[0057] Step S104, based on the relative position corresponding to each road image obtained by analyzing the plurality of road images, determining whether other vehicles on the road ahead have crossed the line during driving;

[0058] Step S106: If other vehicles cross the lane, extract the original time-series lane line features of the complete crossing process of other vehicles;

[0059] Step S108 : Based on the original time-series lane line features, the trained behavior judgment model is used to determine whether the lane-crossing behavior of other vehicles is an illegal lane change.

[0060] Through the above steps, a camera can be controlled to capture road images in front of the current vehicle based on preset calibration parameters. Based on the relative position of each road image obtained by analyzing multiple road images, it is determined whether other vehicles on the road ahead have crossed the lane during their travel. If other vehicles have crossed the lane, the original time-series lane line features of the other vehicles' complete crossing process are extracted. Based on the original time-series lane line features, a trained behavior determination model is used to determine whether the other vehicles' crossing behavior is an illegal lane change. In this embodiment, based on the time-series information of the vehicle's lane change during travel, the vehicle's lane crossing behavior is analyzed to determine whether it is an illegal lane change. The behavior determination model is then used to classify the lane line attributes of the entire time-series process and determine whether the current lane crossing is illegal. The analyzed lane crossing behavior features have time-series characteristics, which can significantly improve the robustness of the detection results. This solves the technical problem in related arts that, when performing illegal lane change detection, lane line attributes are not incorporated into the entire time-series process of the vehicle crossing, resulting in a lack of time-series characteristics and low robustness of the detection results.

[0061] The embodiments of the present invention are described in detail below in conjunction with various implementation steps.

[0062] Optionally, before controlling the camera to capture images of the road in front of the current vehicle based on preset calibration parameters, the detection method also includes: installing the camera on the front windshield of the current vehicle; calibrating the hood area and the horizon position according to the installation position of the camera; and determining the preset calibration parameters and image detection area based on the calibrated hood area and the horizon position.

[0063] In this application, during calibration, the camera can be installed in front of the vehicle (for example, inside the vehicle near the windshield, so as to facilitate the capture of images of the road ahead. By using multiple images or videos captured by the current vehicle, it is possible to analyze whether the lane-changing behavior of the vehicle ahead crosses the line, and whether the crossing-the-line behavior is a traffic violation). Based on the camera installation position, the hood and horizon positions are calibrated (to cope with large field of view FOV). Optionally, in this embodiment, the hood is used as the base, and the height is twice the distance between the hood and the horizon. At the same time, the width is determined according to the aspect ratio of 1.5 to determine the effective detection area.

[0064] In this embodiment, when collecting and annotating materials, a formally implemented camera and corresponding calibration parameters are used to collect image materials on various scene roads (for example, urban and rural roads, elevated roads, highways and other roads of all levels). After obtaining the video, the lane line position and attributes (dashed line or solid line) in the video frame image are annotated, and at the same time, the part of the video with solid line crossing and the part of the video that is not a solid line crossing in the effective detection area are segmented. When collecting materials, different weather conditions such as sunny days, cloudy days, and rainy days and lighting conditions such as strong light and backlight are comprehensively considered. The behavior judgment model is trained by the collected materials. Considering that the behavior of crossing the solid line illegally is a time-series behavior, it is not possible to judge whether a violation has occurred at present by simply using a single state information. It is necessary to combine the attributes of the line shape of the vehicle in the process of crossing the line to comprehensively consider and use the time-series information to judge whether the current crossing behavior is a violation.

[0065] Step S102: Based on preset calibration parameters, the camera is controlled to capture images of the road in front of the current vehicle.

[0066] In this embodiment, the vehicle type of the current vehicle includes but is not limited to: a truck, a sedan, a sports car, a bus, etc.

[0067] The captured road images can be color, grayscale, or binary images. These images are captured by at least one camera positioned near the vehicle's windshield. It should be noted that for the temporal classification network to distinguish between different image types, it must be trained on material of that image type.

[0068] Optionally, in addition to using vehicles traveling on the road to take pictures of the road and vehicles ahead to analyze the vehicle's crossing-line behavior, this embodiment can also automatically monitor the vehicle's illegal lane-changing behavior in fixed scenarios (i.e., through fill lights and cameras on the road side, or road monitoring devices), without the need for additional calibration of lane line attributes and manual formulation of illegal lane-changing rules.

[0069] Step S104 : Based on the relative position corresponding to each road image obtained by analyzing the plurality of road images, it is determined whether other vehicles on the road ahead have crossed the line during driving.

[0070] For the above-mentioned embodiment, based on the relative position corresponding to each road image obtained by analyzing multiple road images, it is determined whether other vehicles on the road ahead have crossed the line during driving, including: combining the vehicle information and lane line information obtained by analyzing each road image, and determining the relative position between the other vehicles and the lane line contained in each road image, wherein the relative position is the ratio of the distance from the vehicle to the lane line to the width of the vehicle; and comprehensively analyzing the relative positions of multiple road images to determine whether other vehicles on the road ahead have crossed the line during driving.

[0071] In this embodiment, vehicle information includes, but is not limited to, vehicle height, width, length, rear end shape, and tire position. Lane marking information includes, but is not limited to, lane marking color, lane marking edge position, lane marking length, and the relative position between the lane marking and the centerline barrier. Vehicle and lane marking information facilitates locating the vehicle's position during driving. Based on vehicle and lane marking information collected at multiple times, the vehicle's trajectory can be further monitored and analyzed.

[0072] An optional implementation of this embodiment, the vehicle information and lane line information obtained by parsing each of the road images, includes: parsing each road image to calibrate at least one vehicle detection frame and at least one lane line detection frame, wherein each vehicle detection frame corresponds to one other vehicle, and each lane line detection frame corresponds to one lane line; determining the vehicle information and lane line information of each other vehicle through at least one vehicle detection frame and at least one lane line detection frame.

[0073] In this embodiment, when parsing a road image, it is possible to distinguish between road pixel regions, lane pixel regions, and vehicle pixel regions in the image. Based on this, a vehicle detection frame and a lane detection frame are calibrated. These two detection frames can be understood as regions of interest (ROIs) obtained through image recognition. Detection frame types include, but are not limited to, rectangles, squares, and circles. The vehicle detection frame can encompass all or part of each vehicle. For example, the rectangle can be used to distinguish the rear end of a vehicle ahead or the rear end of a vehicle in an oblique direction. (Since vehicles often follow a road in a straight line, and there are often multiple roads on the road where vehicles can travel in parallel, the vehicle ahead can observe not only the rear end of the vehicle ahead but also vehicles in other lanes in an oblique direction.) The lane detection frame needs to encompass all or part of the lane lines. Since vehicles may obstruct portions of the lane lines, lane lines that appear in the lane detection frame may be exposed outside the vehicle. Based on the position of the lane lines, the extension direction and length of the lane lines can be inferred to determine a lane detection frame that encompasses the entire lane line.

[0074] Figure 2 is a schematic diagram of an optional extraction of a vehicle detection frame and a lane line detection frame according to an embodiment of the present invention, such as Figure 2 As shown, after analyzing the image in front of the vehicle, multiple vehicle detection frames are divided and the lane line detection frame is determined (in Figure 2 The information contained in the vehicle and lane detection frames (indicated by a line frame in the image) allows the relative position of the vehicle and lane lines to be clearly defined over time. In this embodiment, the detection frames also display the lane line number, lane line image confidence, and relative position information in real time. The lane line image confidence can also be displayed over time.

[0075] This embodiment comprehensively considers the attributes of the lane the target vehicle is in throughout the entire process of crossing the lane, and uses time series information to determine whether the current crossing behavior is a traffic violation. First, it is determined whether the target vehicle has crossed the lane. Specifically, when the target vehicle completes the three states of approaching, crossing, and leaving the same lane according to the time sequence, the target vehicle is considered to have crossed the lane. At this time, all information about the target vehicle's entire crossing process is recorded.

[0076] As an optional implementation of this embodiment, the relative position between the other vehicles and lane lines contained in each road image is determined in combination with the vehicle information and lane line information obtained by parsing each road image, including: obtaining the point position of the preset point of the vehicle detection area and the point position of the intersection of the vehicle lane line based on the vehicle information and lane line information, wherein the intersection of the vehicle lane line includes the intersection of the preset point of the vehicle detection area and the lane line or the lane line extension line, or the vertical intersection of the preset point of the vehicle detection area and the lane line or the lane line extension line; according to the point position of the preset point of the vehicle detection area, the point position of the intersection of the vehicle lane line and the width value of the vehicle detection frame, the relative position is determined using the first formula. Specifically, the preset point of the vehicle detection area is any point in the vehicle detection frame area, for example, the center of the bottom edge of the vehicle detection frame, the center of the vehicle detection frame, etc.

[0077] Figure 3 is a schematic diagram of an optional method for calibrating the relative positions of other vehicles and lane lines according to an embodiment of the present invention, such as Figure 3 As shown, taking the intersection of the vehicle lane line as the intersection of the preset point in the vehicle detection area and the lane line extension line as an example, p refers to the midpoint of the bottom edge of the vehicle detection frame, and q refers to the vehicle lane line intersection point (indicating the intersection of the midpoint of the bottom edge of the vehicle detection frame and the lane line extension line). The following first formula is used to determine the relative position between other vehicles in the detection frame and the lane line. Then, based on the three states of approaching, crossing, and leaving the same lane line in a time sequence, the vehicle's crossing behavior is analyzed. The first formula is: Among them, x p is the point position of the bottom edge of the vehicle detection frame, x q The width is the intersection of the horizontal line where the center of the vehicle bottom is located and the extension line of the lane line. vehicle is the width value of the vehicle detection frame (i.e. the detection frame width), and ratio is the ratio of the distance from the center of the bottom edge of the image vehicle to the lane line to the vehicle detection frame, which can be used to characterize and determine the relative position between other vehicles in the detection frame and the lane line. From a perspective perspective, the farther the other detected vehicles are from the acquisition position, the smaller the vehicle width and the width of the lane line. If the distance from the center of the bottom edge of the image vehicle detection frame to the lane line is simply used to characterize the relative position, even if the distance between the two vehicles and the lane line is the same, the relative position calculated for the vehicle farther from the acquisition position will be larger than the relative position calculated for the vehicle closer to the acquisition position, which is inconsistent with the actual situation. The present application combines the detection frame width to calculate the ratio to characterize the relative position, ensuring that as long as the distance between the vehicle and the lane line is the same, no matter how far the vehicle is from the acquisition position, the ratio calculated by the first formula will not change, and the relative positions obtained are all under the same evaluation standard.

[0078] Optionally, the vehicle information and lane line information of each other vehicle are determined through at least one vehicle detection frame and at least one lane line detection frame, including: based on the lane line detection frame, analyzing the lane line direction vector and lane line attribute information to obtain the lane line information; based on the vehicle detection frame, analyzing the vehicle position, vehicle height and vehicle width of each other vehicle to obtain the vehicle information.

[0079] Another optional method is to analyze the lane line direction vector and lane line attribute information based on the lane line detection frame to obtain the lane line information, including: inputting the lane line detection frame into the lane line model, using the lane line model to analyze the attribute feature vector of the lane line to obtain the lane line information, wherein the lane line model is a pre-trained model, and during the training process, a preset classification framework is used to extract the lane line training sample set according to the lane line marking position, and the lane line training sample set is input into the convolutional neural network system to train the detection network to obtain the lane line model, or, the lane line information is obtained by using a traditional image processing method, wherein the traditional image processing method is to obtain the lane line marking position after performing image preprocessing on the lane line detection frame, and analyze the lane line direction vector and lane line attribute information based on the lane line marking position to obtain the lane line information, wherein the image preprocessing includes: binarization processing, image denoising, and lane line segmentation.

[0080] In this embodiment, the relative positions of multiple road images are integrated to determine whether other vehicles on the road ahead have crossed the line during driving, including: if the absolute value of the first relative position is less than a first threshold, and there is a positive or negative change between the second relative position and the first relative position, then the other vehicle has crossed the line during driving; otherwise, the other vehicle has not crossed the line during driving.

[0081] Specifically, the crossing-line behavior of a vehicle during driving is not a behavior at a certain moment, but rather the observation of the vehicle by completing a complete movement from left to right, or from right to left relative to the same lane line, to determine whether the vehicle in front has crossed the line. Combined with the above definition of relative position, it can be seen that when the vehicle is on both sides of the lane line, the corresponding relative position is different in positive and negative. In the complete process of the vehicle crossing the line, multiple road images contain corresponding time-series relative positions. There must be a situation where the absolute value of the relative position is less than the preset first threshold, and there are positive and negative changes in the relative position. Based on the relative positions of multiple road images, the following first judgment condition is used to determine whether the vehicle has crossed the line during driving:

[0082]

[0083] Where ratio1 is the relative position of the first road image, and ratio2 is the relative position of the second road image. The second road image is temporally arranged after the first road image. Lane change indicates a lane crossing has occurred, and not indicates no lane crossing has occurred.

[0084] In this embodiment, the time-series relative positions of multiple road images are integrated to determine whether the vehicle ahead has crossed the lane by observing the vehicle's complete crossing process from left to right, or from right to left relative to the same lane line, rather than relying solely on position information at a single moment. This avoids the situation where a temporary crossing of the line is mistakenly judged as a crossing, and improves the accuracy and robustness of judging crossing behavior.

[0085] Step S106: If other vehicles cross the lane, extract the original time sequence lane line features of the complete crossing process of other vehicles.

[0086] An optional implementation method in this embodiment is to extract the original temporal lane line features of the complete crossing process of other vehicles, including: determining the approach frame and end frame of the complete crossing process based on the relative positions of multiple road images, and obtaining a timing diagram of the complete crossing process; extracting the original temporal lane line features from the timing diagram.

[0087] The timing diagram is the corresponding frames of the vehicle's complete crossing process, which can include: time point, relative position, and lane line attribute information; using the lane line as the timing baseline, it records the vehicle's driving position and crossing behavior as time progresses during the driving process.

[0088] In this embodiment, since lane lines are conventionally displayed on the road as long lines located at the end of the road or at the intersection of two roads, a solid or dotted line perpendicular to the direction of the road extension will appear in front of the road, and is usually marked in white on the road. Therefore, when analyzing the attribute feature vectors of lane lines through the lane line model, the main focus is on detecting the position of the vehicle approaching (or having crossed or left the lane line) the intersection or the end of the road, and analyzing the attribute information of the vehicle's current line.

[0089] When traffic is heavy, lane line information contained in a single frame image is incomplete. For example, a dashed line may be misidentified as a solid line due to obstruction by a vehicle. To avoid this, when other vehicles cross the same lane line, it is necessary to further determine the lane line attribute information during the complete crossing process. The complete crossing process includes three states: approaching, crossing, and leaving. The type and state of the sequential lane line do not change during the complete crossing process. Therefore, extracting and recording the lane line attribute information of the corresponding frame and performing detection and discrimination based on the sequential lane line characteristics is beneficial to the stability and accuracy of the detection results.

[0090] Optionally, determining the approach frame and end frame of the complete crossing process includes: if the absolute value of the relative position is less than a second threshold, determining it as the approach frame of the complete crossing process; if the absolute value of the relative position is greater than a third threshold, and there is a positive or negative change between the relative position and the relative position corresponding to the approach frame, determining it as the end frame of the complete crossing process.

[0091] If the vehicle is already close to the lane line, first record the attribute information of the current line shape. Some (but not limited to) specific implementation methods are as follows:

[0092] For lane line attribute information, this embodiment can use the traditional machine learning algorithm xgboost, or the basic network structure of deep learning networks such as Mobilenet, Resnet, VGG, and Imagenet as the classification framework. After adaptive adjustment based on the lane line annotation position, the lane line training sample set is extracted and input into the convolutional neural network system for detection network training. After the convolution calculation of the basic network, the cross entropy loss is calculated for back propagation. The target lane line pixel area is then input into the trained lane line model to obtain the attribute feature vector f i In this embodiment, the attribute information extracted at the same time can be the features of a certain layer or the final classification score. In addition, for lane line attributes, traditional description information such as HOG, Haar, LBP, etc. can also be extracted.

[0093] After determining that the vehicle is approaching / close to the lane line, the vehicle may cross the lane line or leave the lane line over time. First, the following second judgment condition is used to determine whether the vehicle is close to the lane line, that is, whether it is an approach frame:

[0094] Among them, close means that the car has crossed the lane line, while not means that the car has not yet approached the lane line.

[0095] If the vehicle has crossed the lane line, first record the attribute information of the current lane line and determine whether the vehicle has completed the complete crossing behavior. Then use the following third judgment condition to determine whether the vehicle has left the lane line, that is, whether it is a leaving frame:

[0096] Among them, leave means that the vehicle crossing the lane has left the lane line, and not means that the vehicle crossing the lane has not left the lane line.

[0097] For the behavior that has been determined as lane changing, after determining the time sequence diagram of the complete crossing process, the time sequence information of the entire crossing process is analyzed from the time sequence diagram to obtain a series of original time sequence lane features (f1, f2, f3, ... f n-1 ,f n), and then input these original time-series lane line features into the behavior judgment model, and determine whether the crossing behavior is illegal based on the time-series information of the entire process.

[0098] In this embodiment, the situation in which the vehicle crosses the line during driving can be analyzed to determine whether the vehicle violates the traffic rules in the crossing-the-line behavior.

[0099] Step S108 : Based on the original time-series lane line features, the trained behavior judgment model is used to determine whether the lane-crossing behavior of other vehicles is an illegal lane change.

[0100] As an optional implementation of this embodiment, based on the original temporal lane line features, a trained behavior judgment model is used to determine whether the crossing behavior of other vehicles is an illegal lane change, including: inputting the original temporal lane line features into the solid line confidence network to obtain the temporal solid line confidence; inputting the relative position and temporal solid line confidence contained in the corresponding frame in the complete crossing process into the behavior judgment model to determine whether the crossing behavior of other vehicles is an illegal lane change, wherein the behavior judgment model is classified as an integrated behavior judgment model or a temporal behavior judgment model.

[0101] In this embodiment, the behavior determination model needs to combine time series information to analyze whether the vehicle's crossing behavior is a traffic violation during the entire crossing process. The behavior determination model in this embodiment can include two scenarios: an integrated behavior determination model and a time series behavior determination model. Optionally, in this embodiment, multiple implementations can be used to determine vehicle violations.

[0102] In a first implementation, an integrated behavior determination model is used to determine vehicle violations.

[0103] As an optional implementation of this embodiment, when the behavior determination model is an integrated behavior determination model, the method includes: combining the relative positions contained in the corresponding frames of each time series of the complete crossing process, classifying and counting the confidence of the time series solid line, and obtaining histogram features; inputting the histogram features into the integrated behavior determination model to determine whether the crossing behavior of other vehicles is an illegal lane change.

[0104] Alternatively, a histogram feature is obtained by classifying and counting the time-series solid line confidences based on the relative positions of the corresponding frames in each time series of the complete crossing process. This includes: dividing the time-series solid line confidences into n confidence sets based on the relative positions of the corresponding frames in each time series of the complete crossing process and a preset segmentation threshold; extracting k-dimensional histogram features from the classification confidences corresponding to each frame of the road image in each confidence set to obtain n k-dimensional histogram features; and concatenating the obtained n k-dimensional histogram features according to the temporal relationship in the historical process to obtain an n*k-dimensional histogram feature. Specifically, the preset segmentation threshold can be adaptively adjusted based on the recognition accuracy.

[0105] Figure 4 is an optional schematic diagram of time series histogram extraction according to an embodiment of the present invention, such as Figure 4 As shown in , by analyzing the confidence of the time series lane line image and the corresponding time series vehicle lane line relative position, the time series histogram features are obtained. Figure 4 In this example, video stream refers to the video stream captured by the vehicle. The confidence level of the lane line images indicated by each video frame in the video stream at different times and the corresponding relative position of the vehicle lane lines are analyzed.

[0106] First, the feature f obtained in each frame during the lane change process i The corresponding sequential solid line confidence is obtained through the solid line confidence network. Based on the ratio of the distance from the vehicle to the lane line to the vehicle width in each frame image during the lane crossing process (referred to as p in this application), the sequential solid line confidence is divided into n sets using the following second formula, where the second formula is:

[0107]

[0108] Among them, θ is the pre-set segmentation threshold, for each set m i Confidence of intra-frame images Extract statistical k-dimensional histogram features w(w1, w2, w3…w n ).

[0109] In this embodiment, when extracting the histogram features, the following third formula is used for extraction, where the third formula is:

[0110]

[0111] Among them, w i is the ratio of the number of frames within the confidence range of the ith solid line to the total number of frames. Finally, in order to utilize the temporal information of the illegal crossing process, the n k-dimensional features are concatenated in a temporal manner to obtain the n*k-dimensional temporal histogram feature.

[0112] Optionally, the integrated behavior determination model is a model obtained by performing integrated learning training using information entropy gain.

[0113] In this embodiment, after extracting fixed-length features from lane-crossing violations of varying time series lengths, a unified-dimensional time series histogram feature can be obtained through classification and statistics. This unified-dimensional time series histogram feature is then input into an integrated behavior determination model trained through ensemble learning for classification, thereby improving the generalization of the model. For example, a random forest is used to perform ensemble learning training using information entropy gain to ultimately obtain an integrated behavior determination model. This application uses the integrated behavior determination model to analyze whether other vehicles have violated lane-changing regulations during the crossing process based on the original time series lane line features.

[0114] The second implementation method uses a temporal behavior determination model to determine the violation behavior.

[0115] As one implementation of this embodiment, when the behavior determination model is a sequential behavior determination model, the detection method includes directly inputting the relative positions and sequential solid line confidence scores contained in corresponding frames during the complete lane crossing process into the sequential behavior determination model in a sequential order to determine whether the other vehicle's lane crossing behavior constitutes an illegal lane change. Optionally, the sequential behavior determination model is obtained through convolutional computation of a basic network and backpropagation training of the calculated cross-entropy loss.

[0116] Specifically, for the behavior that has been determined to be a lane change, a series of original time-series lane features of indefinite length (f1, f2, f3, ... f n-1 ,f n ), input these models into the temporal behavior judgment model, arrange the features in time sequence, input them into the transform-XL model, train the detection network, and after the convolution calculation of the basic network, calculate the cross entropy loss for back propagation to obtain the trained behavior recognition model.

[0117] Optionally, for the network type used in the temporal behavior judgment model, traditional machine learning algorithms such as SVM can be used, or temporal behavior discrimination networks such as RNN, LSTM, transfrom, ViT, etc. can be used.

[0118] In this embodiment, the method of using the time series behavior judgment model to judge violations is a data-driven method with strong data fitting capabilities. On the one hand, under the premise of sufficient platform computing power, the more sufficient the training samples are, the better the judgment effect of the time series behavior judgment model obtained through training. On the other hand, the richer the input time series information, the lower the computational efficiency of the time series behavior judgment model and the more computing resources it occupies.

[0119] For lane changes that have been identified on a solid line, further confirmation is needed to determine whether the lane change is illegal. In practice, not all lane changes on a solid line are illegal. This application uses the "crossing lane" module to filter and eliminate vehicles that cross the lane, further improving the accuracy of illegal behavior identification.

[0120] Optionally, the detection method further includes: using a lane-crossing module to determine whether the lane-crossing behavior of other vehicles is a lane-changing behavior that allows crossing the lane based on the original time-series lane line features and vehicle information in each road image.

[0121] Figure 5 FIG. 1 is a schematic diagram of an optional method for determining vehicle violation behavior by using a line crossing behavior module according to an embodiment of the present invention. Figure 5 As shown, by taking pictures (with Figure 5 After analyzing the relative positions of the vehicle and lane lines in the time series information (each video frame in the video stream in the video stream), the time series lane line map of the crossing behavior is analyzed to obtain the lane line attribute network. Combined with the time series lane line attribute characteristics, this is input into the behavior judgment model. The three cross-line modules (virtual and real double-line module, bus area module, and other cross-line modules) are used to obtain the judgment results. The judgment results indicate whether the vehicle crossing the line is illegal.

[0122] Figure 5 In the process, the relative position of the vehicle and lane markings is first used to identify lane crossing samples. Lane marking attributes are recorded for each frame. Similar to the training method, these lane marking attributes are then fed into the trained behavior determination model to determine whether a lane crossing event occurred. Due to the computing power limitations of onboard hardware and the relatively high computational complexity of deep learning, the forward computation of the entire network requires hardware-specific optimization before it can be ported to the corresponding device for real-time detection of lane changes by the vehicle ahead.

[0123] As an optional implementation of this embodiment, based on the original temporal lane line features and the vehicle information in each road image, a cross-line module is used to determine whether the crossing behavior of other vehicles is a lane change that can be crossed, including: controlling each sub-module of the cross-line module to slide on the temporal lane line features using a sliding window of a preset temporal length to determine the linear category of the lane line, wherein the cross-line module includes a virtual and real double-line sub-module, a bus area sub-module and other cross-line sub-modules; based on the vehicle information and the linear category of the lane line, each sub-module of the cross-line module is used to determine whether the crossing behavior of other vehicles is a lane change that can be crossed.

[0124] For the solid line lane change behavior that has been identified, it is necessary to further confirm whether the solid line lane change is an illegal solid line lane change. Among them, the cross-line module includes: virtual and real double line module, bus area module, and other cross-line modules.

[0125] (1) For virtual and real dual-line modules.

[0126] In this embodiment, if the lane-crossing module is a virtual-real double-line classification module, the virtual-real double-line classification module is used to determine whether the lane-crossing behavior of other vehicles is a lane change that allows crossing the line, including: using a sliding window of a preset time series length to slide on the original time series lane line features to determine multiple confidence levels that the lane line is a virtual-real double line; calculating a first average of the multiple confidence levels to determine whether the lane line is a virtual-real double line based on the first average value to obtain the linear category of the lane line; based on the vehicle crossing direction and the lane line linear category, determining whether the lane-crossing behavior of other vehicles is a lane change that allows crossing the line.

[0127] Optionally, based on the vehicle crossing direction and the linear category of the lane line, determine whether the crossing behavior of other vehicles is a lane change that can be crossed, including: if the vehicle crossing direction indicates that the crossing behavior of other vehicles is to cross the line from the right side of the current lane line, and the linear category of the lane line is left dashed and right solid line, determine that the crossing behavior of other vehicles is not a lane change that can be crossed; if the vehicle crossing direction indicates that the crossing behavior of other vehicles is to cross the line from the left side of the current lane line, and the linear category of the lane line is left solid and right dashed line, determine that the crossing behavior of other vehicles is not a lane change that can be crossed.

[0128] In the virtual and real double lane module, for lane changes on a real line, we determine whether it is a virtual and real double lane. Then, based on the direction of the vehicle crossing the lane, we determine whether it is a traffic violation. One (but not limited to) implementation solution is as follows:

[0129] 1) Slide a window with a time series length of k on the time series lane line features obtained at the time of the violation to obtain the average confidence level of its virtual and real double line related attributes, and determine whether it is a virtual and real double line. The judgment rules are as follows:

[0130]

[0131] Among them, ldrs represents a left dashed line and a right solid line, and lsrd represents a left solid line and a right dashed line. After determining the specific type of the dashed and real double line, proceed to step 2);

[0132] 2) Determine the direction in which the vehicle crossed the line and the obtained linear attributes, and comprehensively determine whether it violated the traffic rules. The specific determination is as follows:

[0133]

[0134] Among them, abnormal indicates a violation, normal indicates no violation, right indicates that the vehicle crosses the lane starting from the right side of the current lane line, and left indicates that the current vehicle crosses the lane starting from the left side of the lane line.

[0135] (2) The bus area is divided into modules.

[0136] Another optional method is that if the cross-line module is a bus area classification module, the bus area classification module is used to determine whether the cross-line behavior of other vehicles is a lane change that can be crossed, including: using a sliding window of a preset time series length to slide on the original time series lane line feature to determine multiple confidence levels that the lane line is the boundary line of the public vehicle area; calculating a second average value of the multiple confidence levels to determine whether the lane line is the boundary line of the public vehicle area based on the second average value to obtain the linear category of the lane line; based on the linear category of the lane line and the time information in the historical process, determine whether the cross-line behavior of other vehicles is a lane change that can be crossed.

[0137] In this embodiment, based on the linear category of the lane line and the time information in the historical process, it is determined whether the crossing-line behavior of other vehicles is a lane change that is allowed to cross the line, including: if the linear category of the lane line indicates that the lane line is the boundary line of the public vehicle area, a sliding window with a preset time series length is used to slide on the vehicle attribute characteristics of the other vehicles to determine whether the other vehicles are public vehicles and obtain a judgment result; if the judgment result indicates that the other vehicles are not public vehicles, it is determined whether the historical time period in the historical process is a time period in which driving is allowed; if the historical time period in the historical process is a time period in which driving is not allowed, it is determined that the crossing-line behavior of the other vehicles is not a lane change that is allowed to cross the line; if the historical time period in the historical process is a time period in which driving is not allowed, it is determined that the crossing-line behavior of the other vehicles is a lane change that is allowed to cross the line.

[0138] In the bus zone module, for lane changes on solid lines, we determine whether they are in a bus zone. We then combine the vehicle attributes (whether it is a bus) and the time it crossed the line to determine whether it is a traffic violation. One (but not limited to) implementation solution is as follows:

[0139] 1) Slide a window with a time series length of k on the time series lane line features obtained at the violation time to obtain the average confidence that it is the bus area boundary line. Determine whether it is the bus area boundary line. If so, proceed to step 2);

[0140] 2) If the lane line that crossed the line is confirmed to be the bus area boundary, a window with a time series length of k is slid on the time series vehicle attribute features obtained at the time of the violation to determine whether it is a bus. If not, proceed to 3)

[0141] 3) After confirming that the vehicle crossing the line is not a bus, check whether the time is available for travel. If not, an alarm will be issued.

[0142] (3) For other modules that can cross the line.

[0143] Other lane-crossing modules in this embodiment may include, but are not limited to: a vehicle continuous dashed line lane-changing module and a lane-riding driving behavior.

[0144] Optionally, if the line-crossing module is another line-crossing module, whether the line-crossing behavior of other vehicles is a lane change that can be crossed is determined by using the other line-crossing module, including: using a sliding window of a preset time series length to slide on the original time series lane line feature to determine multiple confidence levels that the lane line is the edge of the line-crossing area; calculating the third average value of the multiple confidence levels; if the third average value of the multiple confidence levels is greater than the preset value, determining that the line-crossing behavior of other vehicles is a lane change that can be crossed; if the third average value of the multiple confidence levels is less than or equal to the preset value, determining that the line-crossing behavior of other vehicles is not a lane change that can be crossed.

[0145] Other lane-crossing modules determine whether a lane-changing behavior on a solid line is within a cross-line area and then whether it is a traffic violation. One (but not limited to) implementation scheme is as follows: a window with a time series length of k is slid over the time series lane line attribute features obtained at the time of the violation to obtain the average confidence that it is the edge of the cross-line area (such as the stop line). If not, an alarm is issued.

[0146] Another optional method is that after using the cross-line module to determine whether the crossing-line behavior of other vehicles is a lane change that allows crossing the line, the detection method also includes: if it is determined that the crossing-line behavior of other vehicles is illegal, marking the illegal vehicle; issuing a violation prompt message; and reporting the vehicle violation information and vehicle information to the vehicle management platform.

[0147] The video stream is collected by the camera and the image can be output in real time. When a vehicle changes lanes illegally from a solid line, the illegal vehicle will be marked, an illegal alarm message will be issued, and the information will be reported to the corresponding platform.

[0148] Since many violations occur during the peak congestion period after get off work, many vehicles violate traffic rules at close range. At this time, a wide-angle camera with a larger FOV (field of view) is needed to capture the violations. At this time, interference information such as the hood will inevitably be introduced. At this time, after the camera is fixed, the user side can manually mark the position of the engine in the image to complete the hood position calibration, and pass it to the algorithm side to automatically remove the area. The material collection, training, detection and output result scheme are the same as the above implementation method.

[0149] Figure 6 FIG. 1 is a schematic diagram of an optional method of using a wide-angle camera to analyze whether a vehicle has violated traffic regulations according to an embodiment of the present invention. Figure 6As shown in the figure, after analyzing the relative position of the time-series vehicle and the lane line through the captured image, the time-series lane line map of the crossing behavior is used, combined with different field of view angles, to determine whether it is necessary to use the hood calibration in front of the vehicle. If the field of view is small, that is, a conventional near-field camera is used to capture the image, the time-series lane line attribute features can be directly input into the behavior judgment model to obtain the judgment result. If the field of view is large, that is, a wide-angle camera or a telephoto camera is used to capture the image, the hood can be calibrated, and then the time-series lane line attribute features can be input into the behavior judgment model to obtain the judgment result.

[0150] In practical applications, the camera's field of view can be used to determine whether it is a wide-angle camera, and the corresponding model can be used to detect illegal crossing events.

[0151] In this embodiment, based on preset calibration parameters, the step of controlling the camera to capture the road image in front of the current vehicle includes: if the camera type is a telephoto camera and a wide-angle camera, adjusting the focus information when parsing the road image according to the field of view angle information of the camera, and adjusting the calibration information for calibrating the detection frames of other vehicles.

[0152] Given the fields of view (FOVs) of the telephoto and wide-angle cameras, ε1 and ε2, respectively, the closest camera can be determined based on the known FOV of the current camera, automatically determining whether calibration is required. Finally, combining the information collected by both cameras allows for better detection of both near and far crossing violations.

[0153] Considering that in real life, the ambient light is relatively dim due to factors such as night, the image quality of ordinary RGB cameras is poor in relatively dim scenes. At this time, infrared cameras can be used to capture violations, and then the material collection, training, detection and output result schemes are the same as the above implementation method.

[0154] In practical applications, a relatively stable image type switching can be automatically achieved based on the fourth judgment condition according to the actual light intensity β outside the vehicle and the light intensity thresholds β1 and β2 (β1 < β2), and the corresponding model can be used for illegal crossing detection. The fourth judgment condition is as follows:

[0155]

[0156] An optional step of controlling a camera to capture images of the road in front of a current vehicle includes: analyzing ambient light parameters around the current vehicle; if the ambient light parameters are lower than a preset light threshold, using an infrared camera head to capture images of the road in front of the current vehicle.

[0157] Figure 7is a schematic diagram of an optional method for determining line crossing behavior taking into account different light intensities according to an embodiment of the present invention. Figure 7 As shown in the figure, after obtaining the relative positions of each vehicle and lane line in the time series information through analysis of the captured road images, the time series lane line map of the crossing behavior can be analyzed to obtain the lane line attribute characteristics. Then, according to different light intensities, different time series behavior judgment models are used to analyze the crossing behavior. When the light intensity is weak, for example, at night or in thick fog, the infrared model can be used to analyze whether the vehicle's crossing behavior is illegal. If the light intensity is strong, the conventional RGB model can be used to analyze whether the vehicle's crossing behavior is illegal and obtain the judgment result.

[0158] In this embodiment, the current camera type can be determined based on the scene illumination intensity information, and the corresponding model can be automatically selected, so that better detection of violations in dim scenes can be performed.

[0159] Through the above embodiment, the timing information of the entire lane changing process is used to confirm whether the current line crossing behavior is illegal. Specifically, a deep learning model is used to learn the lane line attributes of the entire illegal lane changing process. Combined with the lane line attribute characteristics of the entire timing process, the behavior judgment model is used to classify and obtain the final result to determine whether the current line crossing is illegal, thereby helping the driver to standardize driving behavior.

[0160] Example 2

[0161] This embodiment provides a device for detecting vehicle traffic violations. The various implementation units included in the detection device correspond to the various implementation steps in the above-mentioned embodiment 1.

[0162] Figure 8 is a schematic diagram of an optional vehicle violation detection device according to an embodiment of the present invention, such as Figure 8 As shown, the detection device may include: a control unit 81, a first judgment unit 83, an extraction unit 85, and a second judgment unit 87, wherein:

[0163] A control unit 81 is configured to control the camera to capture an image of the road in front of the current vehicle based on preset calibration parameters;

[0164] A first judging unit 83 is configured to judge whether other vehicles on the road ahead have crossed the line during driving based on the relative position corresponding to each road image obtained by analyzing the plurality of road images;

[0165] An extraction unit 85 is used to extract the original time sequence lane line features of the complete lane crossing process of other vehicles when other vehicles have crossed the lane;

[0166] The second judgment unit 87 is used to judge whether the lane-crossing behavior of other vehicles is an illegal lane change based on the original time-series lane line features and using the trained behavior judgment model.

[0167] The above-mentioned device for detecting vehicle violations can control the camera to collect road images in front of the current vehicle based on preset calibration parameters through the control unit 81, and judge whether other vehicles on the road ahead have crossed the line during driving based on the relative position corresponding to each road image obtained by analyzing multiple road images through the first judgment unit 83. When other vehicles have crossed the line, the original time-series lane line features of the complete crossing process of other vehicles are extracted through the extraction unit 85. The second judgment unit 87 uses the original time-series lane line features and the trained behavior judgment model to judge whether the crossing of other vehicles is an illegal lane change based on the original time-series lane line features. In this embodiment, based on the temporal information of the vehicle's lane changing behavior during driving, it is analyzed whether the vehicle's lane crossing behavior is an illegal lane change. Combined with the lane line attribute characteristics in the entire temporal process, the behavior judgment model classification is used to obtain the final result to determine whether the current lane crossing is illegal. The analyzed vehicle lane crossing behavior characteristics have temporal characteristics, which can significantly improve the robustness of the detection results, thereby solving the technical problem in related technologies that when performing illegal lane change detection, the lane line attributes are not combined with the entire temporal process of the vehicle crossing the line, the characteristics lack temporal characteristics, and the detection results have low robustness.

[0168] Optionally, the first judgment unit includes: a first determination module, used to combine the vehicle information and lane line information obtained by analyzing each road image to determine the relative position between the other vehicles contained in each road image and the lane line, wherein the relative position is the ratio of the distance from the vehicle to the lane line to the width of the vehicle; a first discrimination module, used to integrate the relative positions of multiple road images to determine whether other vehicles on the road ahead have crossed the line during driving.

[0169] Optionally, the first determination module includes: a first parsing submodule, used to parse each road image to calibrate at least one vehicle detection frame and at least one lane line detection frame, wherein each vehicle detection frame corresponds to one other vehicle, and each lane line detection frame corresponds to one lane line; a first determination submodule, used to determine the vehicle information and lane line information of each other vehicle through at least one vehicle detection frame and at least one lane line detection frame.

[0170] Optionally, the first judgment module includes: a second determination submodule, which is used to determine that if the absolute value of the first relative position is less than the first threshold, and there is a positive or negative change between the second relative position and the first relative position, then other vehicles have crossed the line during driving; otherwise, other vehicles have not crossed the line during driving.

[0171] Optionally, the first determination module includes: a first acquisition submodule, used to obtain the point position of the preset point in the vehicle detection area and the point position of the intersection of the vehicle lane line based on vehicle information and lane line information, wherein the intersection of the vehicle lane line includes the intersection of the horizontal straight line where the preset point in the vehicle detection area is located and the lane line or the lane line extension line, or the vertical intersection of the point position of the preset point in the vehicle detection area and the lane line or the lane line extension line; a third determination submodule, used to determine the relative position using the first formula based on the point position of the preset point in the vehicle detection area, the point position of the intersection of the vehicle lane lines and the width value of the vehicle detection frame.

[0172] Optionally, the first determination submodule includes: a first analysis submodule, used to analyze the lane line direction vector and lane line attribute information based on the lane line detection frame to obtain lane line information; a second analysis submodule, used to analyze the vehicle position, vehicle height and vehicle width of each other vehicle based on the vehicle detection frame to obtain vehicle information.

[0173] Optionally, the first analysis submodule includes: a first input submodule, used to input the lane line detection frame into the lane line model, and use the lane line model to analyze the attribute feature vector of the lane line to obtain lane line information, wherein the lane line model is a pre-trained model. During the training process, a preset classification framework is used to extract the lane line training sample set according to the lane line marking position, and the lane line training sample set is input into the convolutional neural network system to train the detection network to obtain the lane line model, or, a second acquisition submodule, used to obtain the lane line information using a traditional image processing method, wherein the traditional image processing method is to obtain the lane line marking position after performing image preprocessing on the lane line detection frame, and analyze the lane line direction vector and lane line attribute information based on the lane line marking position to obtain the lane line information, wherein image preprocessing includes: binarization processing, image denoising, and lane line segmentation.

[0174] Optionally, the extraction unit includes: a second determination module, used to determine the approach frame and end frame of the complete crossing process based on the relative positions of multiple road images, and obtain a timing diagram of the complete crossing process; a first extraction module, used to extract the original timing lane line features from the timing diagram.

[0175] Optionally, the second determination module includes: a fourth determination submodule, used to determine it as a close frame of the complete crossing process when the absolute value of the relative position is less than the second threshold; a fifth determination submodule, used to determine it as the end frame of the complete crossing process when the absolute value of the relative position is greater than the third threshold and there is a positive or negative change between the relative position and the relative position corresponding to the close frame.

[0176] Optionally, the second judgment unit is used to: input the original temporal lane line features into the solid line confidence network to obtain the temporal solid line confidence; input the relative position and the temporal solid line confidence contained in the corresponding frame in the complete crossing process into the behavior judgment model to determine whether the crossing behavior of the other vehicle is an illegal lane change, wherein the behavior judgment model is classified as an integrated behavior judgment model or a temporal behavior judgment model.

[0177] Optionally, when the behavior determination model is an integrated behavior determination model, the detection device is also used to: combine the relative positions contained in each time series corresponding frame of the complete crossing process, classify and count the time series solid line confidences, and obtain histogram features; input the histogram features into the integrated behavior determination model to determine whether the crossing behavior of the other vehicles is an illegal lane change.

[0178] Optionally, the detection device is also used to: divide the time series solid line confidence into n confidence sets based on the relative positions contained in each time series corresponding frame of the complete crossing process and a preset segmentation threshold; extract k-dimensional histogram features for the classification confidence corresponding to each frame of road image in each confidence set to obtain n k-dimensional histogram features; and connect the obtained n k-dimensional histogram features in series according to the time series relationship in the historical process to obtain n*k-dimensional histogram features.

[0179] Optionally, the integrated behavior determination model is a model obtained by performing integrated learning training using information entropy gain.

[0180] Optionally, when the behavior judgment model is a temporal behavior judgment model, the detection device is also used to: directly input the relative position and the temporal solid line confidence contained in the corresponding frame in the complete crossing process into the temporal behavior judgment model in a temporal arrangement to determine whether the crossing behavior of the other vehicle is an illegal lane change.

[0181] Optionally, the temporal behavior determination model is obtained by performing convolution calculation on a basic network and back-propagating training on the calculated cross entropy loss.

[0182] Optionally, the detection device further includes: a determination unit for determining whether the lane-crossing behavior of other vehicles is a lane-changing behavior that allows crossing the line based on the original temporal lane line features and vehicle information in each road image using a lane-crossing module.

[0183] Optionally, the discrimination unit includes: a first control module, used to control each submodule of the cross-line module to slide on the temporal lane line feature using a sliding window of a preset time sequence length to determine the linear category of the lane line, wherein the cross-line module includes a virtual and real double line submodule, a bus area submodule and other cross-line submodules; a first discrimination module, used to use each submodule of the cross-line module to determine whether the crossing behavior of other vehicles is a cross-line lane change based on vehicle information and the linear category of the lane line.

[0184] Optionally, if the cross-line module is a virtual or real double line module, the discrimination unit includes: a first sliding module, used to slide a sliding window of a preset time sequence length on the original time sequence lane line feature to determine multiple confidence levels that the lane line is a virtual or real double line; a first calculation module, used to calculate a first average value of the multiple confidence levels to determine whether the lane line is a virtual or real double line based on the first average value to obtain the linear category of the lane line; a second discrimination module, used to determine whether the crossing behavior of other vehicles is a lane change that can be crossed based on the vehicle's crossing direction and the linear category of the lane line.

[0185] Optionally, the second judgment module includes: a sixth determination submodule, used to indicate in the direction of the vehicle crossing the line that the crossing behavior of other vehicles is starting from the right side of the current lane line, and the linear category of the lane line is a left dashed line and a right solid line, and determine that the crossing behavior of other vehicles is not a lane change that can cross the line; a seventh determination submodule, used to indicate in the direction of the vehicle crossing the line that the crossing behavior of other vehicles is starting from the left side of the current lane line, and the linear category of the lane line is a left solid line and a right dashed line, and determine that the crossing behavior of other vehicles is not a lane change that can cross the line.

[0186] Optionally, if the cross-line module is a bus area classification module, the judgment unit includes: a second sliding module, which is used to slide a sliding window with a preset time series length on the original time series lane line feature to determine multiple confidence levels that the lane line is the boundary line of the public vehicle area; a second calculation module, which is used to calculate a second average value of the multiple confidence levels to determine whether the lane line is the boundary line of the public vehicle area based on the second average value to obtain the linear category of the lane line; a third judgment module, which is used to judge whether the crossing behavior of other vehicles is a lane change that can be crossed based on the linear category of the lane line and the time information in the historical process.

[0187] Optionally, the third discrimination module includes: a first sliding submodule, for sliding a sliding window of a preset time series length on the vehicle attribute characteristics of other vehicles when the linear category of the lane line indicates that the lane line is the boundary line of the public vehicle area, to determine whether the other vehicles are public vehicles and obtain a judgment result; a fourth discrimination module, for judging whether the historical time period in the historical process is a time period in which driving is allowed when the judgment result indicates that the other vehicles are not public vehicles; an eighth determination submodule, for determining that the crossing-line behavior of other vehicles is not a lane change that allows crossing the line when the historical time period in the historical process is a time period in which driving is not allowed; and a ninth determination submodule, for determining that the crossing-line behavior of other vehicles is a lane change that allows crossing the line when the historical time period in the historical process is a time period in which driving is not allowed.

[0188] Optionally, if the cross-line module is other cross-line sub-modules, the judgment unit includes: a third sliding module, which is used to slide a sliding window of a preset time sequence length on the original time sequence lane line feature to determine multiple confidence levels that the lane line is the edge of the cross-line area; a third calculation module, which is used to calculate the third average value of the multiple confidence levels; a tenth determination sub-module, which is used to determine that the crossing behavior of other vehicles is a lane change that can be crossed when the third average value of the multiple confidence levels is greater than a preset value; and an eleventh determination sub-module, which is used to determine that the crossing behavior of other vehicles is not a lane change that can be crossed when the third average value of the multiple confidence levels is less than or equal to a preset value.

[0189] Optionally, the detection device also includes: a marking unit, which is used to mark the illegal vehicle if it is determined that the crossing of the line by other vehicles is illegal after using the crossing-line module to determine whether the crossing of the line by other vehicles is a lane change that can be crossed; a first issuing unit, which is used to issue a violation prompt message; and a first reporting unit, which is used to report the vehicle violation information and vehicle information to the vehicle management platform.

[0190] Optionally, the detection device also includes: an installation unit, used to install the camera on the front windshield of the current vehicle before controlling the camera to capture the road image in front of the current vehicle based on preset calibration parameters; a calibration unit, used to calibrate the hood area and horizon position according to the installation position of the camera; and a determination unit, used to determine the preset calibration parameters and image detection area based on the calibrated hood area and horizon position.

[0191] Optionally, the control unit includes: an adjustment module for adjusting the focus information when parsing the road image according to the field of view angle information of the camera when the camera type is a telephoto camera and a wide-angle camera, and adjusting the calibration information for calibrating the detection frame of other vehicles.

[0192] Optionally, the control unit is further used to analyze the ambient light parameters around the current vehicle; if the ambient light parameters are lower than a preset light threshold, an infrared camera head is used to capture the road image in front of the current vehicle.

[0193] The above-mentioned vehicle violation detection device can also include a processor and a memory. The above-mentioned control unit 81, first judgment unit 83, extraction unit 85, second judgment unit 87, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0194] The processor includes a core that retrieves the corresponding program unit from memory. One or more cores can be configured. By adjusting kernel parameters, the processor uses a trained behavior determination model to determine whether a vehicle's lane-crossing behavior constitutes an illegal lane change based on raw time-series lane line features.

[0195] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0196] According to another aspect of an embodiment of the present invention, a road vehicle is also provided, comprising: an on-board camera installed on the windshield in front of the vehicle, for collecting road images of the road ahead; and an on-board control unit connected to the on-board camera, for executing any one of the above-mentioned methods for detecting vehicle violations.

[0197] According to another aspect of an embodiment of the present invention, a vehicle-mounted electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods for detecting vehicle violations by executing the executable instructions.

[0198] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned methods for detecting vehicle violations.

[0199] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialized program having the following method steps: based on preset calibration parameters, controlling a camera to capture road images in front of a current vehicle; based on the relative position corresponding to each road image obtained by analyzing multiple road images, determining whether other vehicles on the road ahead have crossed the lane during driving; if other vehicles have crossed the lane, extracting the original time-series lane line features of the complete crossing process of the other vehicles; based on the original time-series lane line features, using a trained behavior judgment model to determine whether the crossing of the lane by other vehicles is an illegal lane change.

[0200] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0201] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0203] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0204] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0206] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for detecting vehicle violations, characterized in that: include: Based on preset calibration parameters, the camera is controlled to collect images of the road in front of the current vehicle; Based on the relative position corresponding to each road image obtained by analyzing the plurality of road images, determining whether other vehicles on the road ahead have crossed the line during driving; If the other vehicle crosses the lane, extract a time sequence diagram of the complete lane crossing process of the other vehicle, and obtain original time sequence lane line features based on the time sequence diagram, wherein the time sequence diagram uses the lane line as the time sequence baseline and records the driving position and lane crossing behavior of the vehicle as time progresses during the driving process; Based on the original temporal lane line features, a trained behavior determination model is used to determine whether the lane crossing behavior of the other vehicle is an illegal lane change, including: inputting the original temporal lane line features into a solid line confidence network to obtain a temporal solid line confidence; inputting the relative position and the temporal solid line confidence contained in the corresponding frame during the complete lane crossing process into the behavior determination model to determine whether the lane crossing behavior of the other vehicle is an illegal lane change, wherein the behavior determination model is classified as an integrated behavior determination model or a temporal behavior determination model; When the behavior determination model is an integrated behavior determination model, the detection method includes: combining the relative positions contained in each time series corresponding frame of the complete lane crossing process, classifying and counting the confidence of the solid line of the time series to obtain a histogram feature; inputting the histogram feature into the integrated behavior determination model to determine whether the lane crossing behavior of the other vehicle is an illegal lane change; In combination with the relative positions contained in the corresponding frames of each time series of the complete crossing process, the histogram features are obtained by classifying and counting the confidence of the time series solid line, including: combining the relative positions contained in the corresponding frames of each time series of the complete crossing process and a preset segmentation threshold, dividing the confidence of the time series solid line into n confidence sets; for each confidence set, dividing the confidence of the time series solid line in the confidence set into k confidence ranges based on the numerical value, and calculating the proportion of the frame pictures corresponding to the confidence of the time series solid line in each confidence range to the total number of frames in the confidence set, taking the proportion of the k confidence ranges as the k-dimensional histogram features, calculating the k-dimensional histogram features of all confidence sets, and obtaining n k-dimensional histogram features; connecting the obtained n k-dimensional histogram features in series according to the temporal relationship in the historical process to obtain n*k-dimensional histogram features; When the behavior judgment model is a temporal behavior judgment model, the detection method includes: directly inputting the relative position and the temporal solid line confidence contained in the corresponding frame during the complete crossing process into the temporal behavior judgment model in a temporal arrangement to determine whether the crossing behavior of the other vehicle is an illegal lane change, wherein the temporal behavior judgment model is obtained through basic network convolution calculation.

2. The detection method according to claim 1, wherein Based on the relative position corresponding to each road image obtained by analyzing the plurality of road images, determining whether other vehicles on the road ahead have crossed the line during driving, including: Determining the relative position between the other vehicles and the lane line in each of the road images by combining the vehicle information and lane line information obtained by analyzing each of the road images, wherein the relative position is a ratio of the distance from the vehicle to the lane line to the width of the vehicle; The relative positions of the plurality of road images are integrated to determine whether other vehicles on the road ahead have crossed the line during driving.

3. The detection method according to claim 2, characterized in that The vehicle information and lane line information obtained by analyzing each of the road images include: Parsing each of the road images to calibrate at least one vehicle detection frame and at least one lane line detection frame, wherein each of the vehicle detection frames corresponds to another vehicle, and each of the lane line detection frames corresponds to a lane line; The vehicle information and the lane line information of each of the other vehicles are determined through at least one vehicle detection frame and at least one lane line detection frame.

4. The detection method according to claim 2, characterized in that Combining the relative positions of the plurality of road images to determine whether other vehicles on the road ahead have crossed the line during driving, includes: If the absolute value of the first relative position is less than the first threshold, and there is a positive or negative change between the second relative position and the first relative position, then the other vehicle has crossed the line during driving; Otherwise, the other vehicles do not cross the line during driving.

5. The detection method according to claim 2, characterized in that Determining the relative position between the other vehicles and the lane lines in each of the road images by combining the vehicle information and lane line information obtained by analyzing each of the road images includes: Based on the vehicle information and the lane line information, obtaining the point position of a preset point in the vehicle detection area and the point position of a vehicle lane line intersection point, wherein the vehicle lane line intersection point includes the intersection point of a horizontal straight line where the preset point in the vehicle detection area is located and the lane line or the lane line extension line, or the vertical intersection point of the point position of the preset point in the vehicle detection area and the lane line or the lane line extension line; The relative position is determined based on the point position of the preset point in the vehicle detection area, the point position of the intersection of the vehicle lane lines, and the width value of the vehicle detection frame.

6. The detection method according to claim 3, characterized in that Determining the vehicle information and the lane line information of each of the other vehicles using at least one vehicle detection frame and at least one lane line detection frame includes: Analyzing the lane line direction vector and lane line attribute information based on the lane line detection frame to obtain the lane line information; Based on the vehicle detection frame, the vehicle position, vehicle height and vehicle width of each of the other vehicles are analyzed to obtain the vehicle information.

7. The detection method according to claim 6, characterized in that Based on the lane line detection frame, the lane line direction vector and lane line attribute information are analyzed to obtain the lane line information, including: The lane line detection frame is input into a lane line model, and the lane line attribute feature vector is analyzed using the lane line model to obtain the lane line information, wherein the lane line model is a pre-trained model. During the training process, a preset classification framework is used to extract a lane line training sample set based on the lane line annotation position, and the lane line training sample set is input into a convolutional neural network system to train the detection network to obtain the lane line model, or, The lane line information is obtained by using a traditional image processing method, wherein the traditional image processing method is to obtain the lane line marking position after performing image preprocessing on the lane line detection frame, and analyze the lane line direction vector and lane line attribute information based on the lane line marking position to obtain the lane line information, wherein the image preprocessing includes: binarization processing, image denoising, and lane line segmentation.

8. The detection method according to claim 1, wherein Extracting the original time-series lane line features during the complete lane crossing process of the other vehicle, including: Determining an approach frame and an end frame of a complete line crossing process according to the relative positions of the plurality of road images, and obtaining a timing diagram of the complete line crossing process; The original time-series lane line features are extracted from the time-series graph.

9. The detection method according to claim 8, characterized in that Determine the approach frame and end frame of the complete crossing process, including: If the absolute value of the relative position is less than a second threshold, it is determined to be a close frame of the complete line crossing process; If the absolute value of the relative position is greater than a third threshold, and there is a positive or negative change between the relative position and the relative position corresponding to the approaching frame, it is determined to be the end frame of the complete line crossing process.

10. The detection method according to claim 1, characterized in that The integrated behavior determination model is a model obtained by performing integrated learning training using information entropy gain.

11. The detection method according to claim 1, characterized in that The temporal behavior determination model is obtained by back-propagation training of the cross entropy loss obtained by convolution calculation of the basic network.

12. The detection method according to claim 1, characterized in that The detection method further comprises: Based on the original time-series lane line features and the vehicle information in each of the road images, a lane-crossing module is used to determine whether the lane-crossing behavior of the other vehicle is a lane change that allows crossing the lane.

13. The detection method according to claim 12, characterized in that: Based on the original time-series lane line features and the vehicle information in each of the road images, a lane-crossing module is used to determine whether the lane-crossing behavior of the other vehicle is a lane-changing behavior that allows lane crossing, including: Controlling each submodule of the cross-line module to slide a sliding window of a preset time sequence length on the time sequence lane line feature to determine the linear category of the lane line, wherein the cross-line module includes a virtual and real double line submodule, a bus area submodule, and other cross-line submodules; Based on the vehicle information and the linear category of the lane line, the submodules of the lane-crossing module are used to determine whether the lane-crossing behavior of the other vehicle is a lane change that allows crossing the line.

14. The detection method according to claim 13, characterized in that If the lane-crossing module is a virtual-real double-lane classification module, determining whether the lane-crossing behavior of the other vehicle is a lane-changing that allows crossing the lane by using the virtual-real double-lane classification module includes: Sliding a sliding window of a preset time series length on the original time series lane line features to determine multiple confidence levels of whether the lane line is a virtual or real double line; Calculating a first average of the plurality of confidence levels to determine whether the lane line is a double line based on the first average, thereby obtaining a linear category of the lane line; Based on the lane crossing direction of the vehicle and the linear category of the lane line, it is determined whether the lane crossing behavior of the other vehicle is a lane change that allows crossing the lane.

15. The detection method according to claim 14, characterized in that: Determining whether the lane-crossing behavior of the other vehicle is a lane change that allows crossing the lane based on the vehicle's crossing direction and the linearity category of the lane line includes: If the vehicle crossing direction indicates that the other vehicle's crossing behavior is to cross the lane from the right side of the current lane line, and the linear type of the lane line is a left dashed line and a right solid line, determining that the other vehicle's crossing behavior is not a lane change that allows crossing the lane line; If the vehicle crossing direction indicates that the other vehicle's crossing behavior starts from the left side of the current lane line, and the linear category of the lane line is a left solid line and a right dashed line, it is determined that the other vehicle's crossing behavior is not a lane change that allows crossing the line.

16. The detection method according to claim 13, characterized in that: If the lane-crossing module is a bus area module, determining whether the lane-crossing behavior of the other vehicle is a lane-changing that allows crossing the lane by using the bus area module includes: Sliding a sliding window of a preset time series length on the original time series lane line features to determine multiple confidence levels that the lane line is a boundary line of a public vehicle area; Calculating a second average of the plurality of confidence levels to determine whether the lane line is a boundary line of a public vehicle area based on the second average, thereby obtaining a linear category of the lane line; Based on the linearity of the lane line and the time information in the historical process, it is determined whether the lane-crossing behavior of the other vehicle is a lane change that allows crossing the lane line.

17. The detection method according to claim 16, characterized in that: Determining whether the lane-crossing behavior of the other vehicle is a lane-changing behavior that allows lane-crossing based on the linearity category of the lane line and the time information of the historical process includes: If the linear category of the lane line indicates that the lane line is a boundary line of a public vehicle area, sliding a sliding window of a preset time sequence length on the vehicle attribute features of other vehicles to determine whether the other vehicles are public vehicles, and obtaining a determination result; If the determination result indicates that the other vehicle is not a public vehicle, determining whether the historical time period in the historical process is a permitted travel time period; If the historical time period in the historical process is a non-permitted driving time period, determining that the lane-crossing behavior of the other vehicle is not a lane-changing behavior that allows lane-crossing; If the historical time period in the historical process is a permitted driving time period, it is determined that the lane-crossing behavior of the other vehicle is a lane-changing behavior in which the lane is allowed to cross the line.

18. The detection method according to claim 13, characterized in that: If the lane-crossing module is another lane-crossing module, determining whether the lane-crossing behavior of the other vehicle is a lane-changing behavior that allows crossing the lane by using the other lane-crossing module includes: Sliding a sliding window of a preset time series length on the original time series lane line features to determine multiple confidence levels that the lane line is a boundary of a crossable area; calculating a third average of the plurality of confidence levels; If a third average value of the plurality of confidence levels is greater than a preset value, determining that the lane-crossing behavior of the other vehicle is a lane-changing behavior that allows crossing the lane; If the third average value of the plurality of confidence levels is less than or equal to a preset value, it is determined that the lane-crossing behavior of the other vehicle is not a lane-changing behavior that allows crossing the lane.

19. The detection method according to claim 13, characterized in that: After determining whether the lane-crossing behavior of the other vehicle is a lane-changing behavior that allows the vehicle to cross the lane using the lane-crossing module, the detection method further includes: If it is determined that the crossing of the line by the other vehicle is illegal, marking the illegal vehicle; Issue violation reminder information; Report vehicle violation information and vehicle information to the vehicle management platform.

20. The detection method according to claim 1, characterized in that Before controlling the camera to capture an image of the road in front of the current vehicle based on the preset calibration parameters, the detection method further includes: Install the camera on the front windshield of the current vehicle; According to the camera installation position, calibrate the hood area and horizon position; Based on the calibrated hood area and horizon position, the preset calibration parameters and image detection area are determined.

21. The detection method according to claim 1, characterized in that The steps of controlling the camera to capture an image of the road in front of the current vehicle based on preset calibration parameters include: If the camera is a telephoto camera or a wide-angle camera, the focus information when parsing the road image is adjusted according to the field of view information of the camera, and the calibration information for calibrating the detection frames of other vehicles is adjusted.

22. The detection method according to claim 1, characterized in that The steps of controlling the camera to collect the road image in front of the current vehicle include: Analyzing ambient light parameters around the current vehicle; If the ambient light parameter is lower than a preset light threshold, an infrared camera is used to capture an image of the road in front of the current vehicle.

23. A device for detecting vehicle violations, characterized in that: include: A control unit, configured to control the camera to capture an image of the road in front of the current vehicle based on preset calibration parameters; a first judging unit configured to judge whether other vehicles on the road ahead have crossed the line during driving based on the relative position corresponding to each road image obtained by analyzing the plurality of road images; an extraction unit, configured to extract a time sequence diagram of the complete lane crossing process of the other vehicle when the other vehicle crosses the lane, and obtain original time sequence lane line features based on the time sequence diagram, wherein the time sequence diagram uses the lane line as a time sequence reference line and records the driving position and lane crossing behavior of the vehicle as time progresses during the driving process; A second judgment unit is configured to determine whether the lane-crossing behavior of the other vehicle is an illegal lane change based on the original time-series lane line features and using a trained behavior judgment model; The second judgment unit includes: a first acquisition module, configured to input the original time-series lane line features into a solid line confidence network to obtain a time-series solid line confidence; a discrimination module, configured to input the relative position and the time-series solid line confidence contained in the corresponding frame during the complete lane crossing process into the behavior judgment model to judge whether the lane crossing behavior of the other vehicle is an illegal lane change, wherein the behavior judgment model is an integrated behavior judgment model or a time-series behavior judgment model; When the behavior determination model is an integrated behavior determination model, the detection device is further configured to: combine the relative positions contained in the corresponding frames of each time series of the complete lane crossing process, classify and count the confidence levels of the solid lines in the time series to obtain a histogram feature; input the histogram feature into the integrated behavior determination model to determine whether the lane crossing behavior of the other vehicle is an illegal lane change; The detection device is further used to: combine the relative positions contained in the corresponding frames of each time series of the complete crossing process and the preset segmentation threshold to divide the time series solid line confidence into n confidence sets; for each confidence set, divide the time series solid line confidence in the confidence set into k confidence ranges based on the numerical value, and calculate the proportion of the frame pictures corresponding to the time series solid line confidence in each of the confidence ranges to the total number of frames in the confidence set, use the proportion of the k confidence ranges as the k-dimensional histogram feature, calculate the k-dimensional histogram features of all confidence sets, and obtain n k-dimensional histogram features; connect the obtained n k-dimensional histogram features in series according to the temporal relationship in the historical process to obtain n*k-dimensional histogram features; When the behavior judgment model is a temporal behavior judgment model, the detection device is also used to: directly input the relative position and the temporal solid line confidence contained in the corresponding frame during the complete crossing process into the temporal behavior judgment model in a temporal arrangement to determine whether the crossing behavior of the other vehicle is an illegal lane change, wherein the temporal behavior judgment model is obtained through basic network convolution calculation.

24. The detection device according to claim 23, characterized in that The first judgment unit includes: a first determining module, configured to determine a relative position between the other vehicles and the lane line contained in each of the road images, based on the vehicle information and lane line information obtained by analyzing each of the road images, wherein the relative position is a ratio of the distance from the vehicle to the lane line to the width of the vehicle; The first determination module is configured to integrate the relative positions of the plurality of road images to determine whether other vehicles on the road ahead have crossed the line during driving.

25. The detection device according to claim 24, characterized in that The first determining module includes: a first parsing submodule, configured to parse each of the road images to calibrate at least one vehicle detection frame and at least one lane line detection frame, wherein each vehicle detection frame corresponds to another vehicle, and each lane line detection frame corresponds to a lane line; The first determining submodule is configured to determine the vehicle information and the lane line information of each of the other vehicles through at least one vehicle detection frame and at least one lane line detection frame.

26. The detection device according to claim 24, characterized in that The first discrimination module includes: a second determining submodule, configured to determine, if the absolute value of the first relative position is less than a first threshold value and there is a positive or negative change between the second relative position and the first relative position, that the other vehicle has crossed the line during driving; Otherwise, the other vehicles do not cross the line during driving.

27. The detection device according to claim 24, characterized in that The first determining module includes: A first obtaining submodule is configured to obtain, based on the vehicle information and the lane line information, a point position of a preset point in a vehicle detection area and a point position of a vehicle lane line intersection, wherein the vehicle lane line intersection includes an intersection of a horizontal straight line where the preset point in the vehicle detection area is located and the lane line or an extension of the lane line, or a vertical intersection of the point position of the preset point in the vehicle detection area and the lane line or an extension of the lane line; The third determination submodule is used to determine the relative position based on the point position of the preset point in the vehicle detection area, the point position of the intersection of the vehicle lane lines, and the width value of the vehicle detection frame.

28. The detection device according to claim 25, characterized in that The first determination submodule includes: A first analysis submodule is configured to analyze the lane line direction vector and lane line attribute information based on the lane line detection frame to obtain the lane line information; The second analysis submodule is configured to analyze the vehicle position, vehicle height, and vehicle width of each of the other vehicles based on the vehicle detection frame to obtain the vehicle information.

29. The detection device according to claim 28, characterized in that The first analysis submodule includes: The first input submodule is configured to input the lane line detection frame into a lane line model, and use the lane line model to analyze the attribute feature vector of the lane line to obtain the lane line information, wherein the lane line model is a pre-trained model. During the training process, a preset classification framework is used to extract a lane line training sample set based on the lane line annotation position, and the lane line training sample set is input into a convolutional neural network system to train the detection network to obtain the lane line model, or The second acquisition submodule is used to obtain the lane line information using a traditional image processing method, wherein the traditional image processing method is to obtain the lane line marking position after performing image preprocessing on the lane line detection frame, and analyze the lane line direction vector and lane line attribute information based on the lane line marking position to obtain the lane line information, wherein the image preprocessing includes: binarization processing, image denoising, and lane line segmentation.

30. The detection device according to claim 24, characterized in that The extraction unit comprises: a second determining module, configured to determine an approach frame and an end frame of a complete line crossing process according to the relative positions of the plurality of road images, and obtain a timing diagram of the complete line crossing process; The first extraction module is used to extract the original time sequence lane line features from the time sequence graph.

31. The detection device according to claim 30, characterized in that The second determining module includes: a fourth determining submodule, configured to determine, when the absolute value of the relative position is less than a second threshold, that the frame is a close frame of the complete line crossing process; The fifth determining submodule is configured to determine that the frame is the end frame of the complete line crossing process if the absolute value of the relative position is greater than a third threshold and there is a positive or negative change between the relative position and the relative position corresponding to the approaching frame.

32. A road vehicle, characterized in that include: The vehicle-mounted camera is installed on the windshield in front of the vehicle and is used to collect road images of the road ahead; An on-board control unit is connected to the on-board camera and executes the method for detecting vehicle violations as described in any one of claims 1 to 22.

33. An in-vehicle electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the vehicle violation detection method described in any one of claims 1 to 22 by executing the executable instructions.

34. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the vehicle violation detection method according to any one of claims 1 to 22.

Images