Method, device and equipment for predicting violation behavior

Abstract

The application provides a method, device and equipment for predicting illegal behavior, comprising: acquiring a road image of a road on which a target vehicle travels and vehicle travel data; inferring the road image and the vehicle travel data through a preset behavior prediction model to obtain a target probability that the target vehicle triggers a target illegal behavior; and predicting whether the target vehicle triggers the target illegal behavior according to the target probability. Thus, real-time, fast and high-accuracy illegal behavior prediction is achieved.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a method, apparatus and device for predicting traffic violations. Background Technology

[0002] Traffic violations can lead to points deductions and fines for vehicle owners, and even cause traffic accidents that threaten lives. Current violation identification methods are typically reactive, unable to identify and alert drivers in real time, thus failing to minimize losses and prevent accidents. Therefore, predicting traffic violations during driving is the technical problem this application aims to solve. Summary of the Invention

[0003] This application provides a method, apparatus, and device for predicting traffic violations, which can achieve real-time, fast, and highly accurate prediction of traffic violations.

[0004] In a first aspect, this application provides a method for predicting traffic violations, comprising: acquiring road images of the road on which the target vehicle is traveling and vehicle driving data; reasoning about the road images and vehicle driving data using a preset behavior prediction model to obtain a target probability that the target vehicle will trigger a target traffic violation; and predicting whether the target vehicle will trigger a target traffic violation based on the target probability.

[0005] Secondly, this application provides a traffic violation prediction device, comprising: an acquisition module for acquiring road images of the road on which the target vehicle travels and vehicle driving data; an inference module for inferring from the road images and vehicle driving data using a preset behavior prediction model to obtain a target probability that the target vehicle will trigger a target traffic violation; and a prediction module for predicting whether the target vehicle will trigger a target traffic violation based on the target probability.

[0006] Thirdly, this application provides an electronic device, including: a processor and a memory, the memory for storing a computer program, and the processor for calling and running the computer program stored in the memory to perform the methods as described in the first aspect or its various implementations.

[0007] Fourthly, this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform the methods described in the first aspect or its various implementations.

[0008] Fifthly, this application provides a computer program product including computer program instructions that cause a computer to perform the methods as described in the first aspect or its various implementations.

[0009] Sixthly, this application provides a computer program that causes a computer to perform the methods described in the first aspect or its various implementations.

[0010] The technical solution of this application can acquire road images and vehicle driving data of the road where the target vehicle is traveling; it can then use a preset behavior prediction model to infer the target probability of the target vehicle triggering a target violation based on the road images and vehicle driving data; and finally, based on the target probability, it can predict whether the target vehicle will trigger the target violation. In this process, because information from both road images and vehicle driving data is introduced simultaneously, the behavior prediction model can more comprehensively infer and reconstruct the vehicle's actual driving situation, avoiding misjudgments that may result from relying on only a single piece of information. Furthermore, through the quantitative output of the target probability, it can quickly and accurately predict the triggering of the target violation even before the violation has fully occurred but when the risk is high. Therefore, it can achieve real-time, fast, and highly accurate violation prediction.

[0011] Other technical features and effects involved in the technical solution of this application will be described in subsequent embodiments, and will not be repeated here to avoid repetition. Attached Figure Description

[0012] The accompanying drawings used in the following description of the embodiments are introduced.

[0013] Figure 1 A flowchart illustrating a method for predicting traffic violations provided in this application embodiment; Figure 2 A schematic diagram of the violation prediction device provided in the embodiments of this application; Figure 3 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0016] In one embodiment, the technical solution of this application can be used in scenarios of predicting and alerting vehicle violations, for example, it can be applied to real-time violation warning scenarios during the driver's driving process.

[0017] In one embodiment, the solution provided in this application can be executed by any electronic device with data processing capabilities. For example, the electronic device can be a server, specifically a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Alternatively, the electronic device can be a terminal device, specifically a tablet computer, laptop computer, or desktop computer. Furthermore, the electronic device can be a combination of a server and a terminal device, wherein the server and terminal device in the combination can communicate wirelessly or via wired means. This application does not impose specific limitations on the electronic device.

[0018] Furthermore, the electronic device can also be the target vehicle (e.g., a vehicle with an in-vehicle intelligent system, a connected car equipped with an in-vehicle computing unit, or a vehicle capable of supporting local deployment and operation of behavior prediction models, etc.) or installed in the target vehicle (e.g., installed in the in-vehicle computing unit of the target vehicle). In this deployment method, the technical solution of this application can be completed locally in the vehicle, thus eliminating the need for additional network transmission and ensuring stable operation in weak network environments such as tunnels and mountainous areas. Moreover, compared to methods that require cloud processing (e.g., data needs to be transmitted to the cloud for processing), it can achieve low latency, high real-time performance, and data privacy protection.

[0019] It should be noted that all technical solutions in this application can be combined in any way to form optional embodiments of this application. To avoid repetition, these will not be elaborated upon.

[0020] Furthermore, the specific embodiments of this application involve data, information, or instructions such as road images and vehicle driving data. When the embodiments of this application are applied to specific products or technologies, user permission, consent, or authorization is required, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards.

[0021] The technical solution of this application will be described in detail below: In one embodiment, Figure 1 This is a flowchart illustrating a method for predicting traffic violations provided in an embodiment of this application. This method can be executed by the aforementioned electronic device, but is not limited thereto. Figure 1 As shown, the method may include S110-S130.

[0022] S110: Acquire road images and vehicle driving data of the road on which the target vehicle is traveling.

[0023] For example, the target vehicle can be an unmanned vehicle or a manually driven vehicle, and this application does not limit it.

[0024] For example, electronic devices can acquire road images through a camera. For instance, a camera can be installed in front of a target vehicle, continuously and in real-time capturing images of the road conditions ahead. The road image may include at least one of the following: lane lines, traffic lights (or traffic lights), traffic signs (such as one-way signs, lane arrows, stop lines, etc.), vehicles ahead or to the side, pedestrians, obstacles, etc.

[0025] Alternatively, one can acquire road video of the route being traveled, and then extract the road image by extracting frames from the road video.

[0026] For example, vehicle driving data can be obtained by reading the Controller Area Network (CAN) bus of the target vehicle. The CAN bus is the internal communication network of the target vehicle, where status data from various components (e.g., engine, steering wheel, lights, etc.) can be aggregated and shared. Electronic devices can obtain real-time vehicle driving data by connecting to and reading the CAN bus. For example, vehicle driving data may include at least one of the following: vehicle speed (i.e., the current speed of the target vehicle), steering angle (e.g., the rotation angle of the steering wheel or the steering angle of the wheels), turn signal status (e.g., whether the left / right turn signal is currently illuminated or off, and the duration of illumination), brake signal, throttle position, etc.

[0027] In addition to acquiring vehicle driving data via the CAN bus, when the target vehicle lacks CAN bus read permissions, CAN bus data is missing, or the data is used for redundancy verification, the target vehicle driving data can also be acquired in the following ways, but not limited to these: An Inertial Measurement Unit (IMU) can be integrated into or connected to the target vehicle. The IMU contains an accelerometer and a gyroscope. By integrating the three-axis acceleration data collected by the accelerometer over time, the driving speed of the target vehicle can be calculated. By integrating the angular velocity data collected by the gyroscope over time, the heading angle change of the target vehicle can be calculated, thereby obtaining the vehicle's steering trend and steering angle.

[0028] For example, while acquiring road images and vehicle driving data, the timestamps of the road images and vehicle driving data can be acquired to obtain a first timestamp and a second timestamp. Then, the road images and vehicle driving data can be synchronized according to the first timestamp and the second timestamp to facilitate the correspondence of data during subsequent inference.

[0029] In the above-mentioned embodiment of S110, by acquiring multi-source data such as road images and vehicle driving data, which are related to the external traffic environment and the vehicle's own dynamics, a multi-dimensional data foundation is provided for the accurate prediction of subsequent violations. Compared with data collection from a single source, this multi-dimensional data acquisition can effectively reduce the possibility of misjudgment caused by reliance on a single visual solution such as complex lighting or extreme weather. It also provides data support for subsequent cross-validation and ensures the realization of high real-time and high reliability vehicle-side violation warnings.

[0030] S120: By using a preset behavior prediction model to infer from road images and vehicle driving data, the target probability of the target vehicle triggering the target violation behavior is obtained.

[0031] For example, the target traffic violation may include at least one of the following, but is not limited to: crossing the line, driving against traffic, running a red light, and changing lanes without signaling. A traffic violation refers to an act that violates road traffic safety regulations, which typically results in fines, demerit points on the driver's license, or other administrative penalties.

[0032] For example, road images and vehicle driving data can be synchronously input into a pre-trained behavior prediction model. This behavior prediction model can be a large-scale artificial intelligence model deployed on the target vehicle's computing unit, i.e., the vehicle-side computing unit (i.e., the behavior prediction model can be an edge-side large-scale model). It can extract visual features from the input road images frame by frame or in partial frames (e.g., keyframes), identifying lane lines, traffic lights, traffic signs, and the relative positional relationships between the target vehicle and these elements in the road environment. Simultaneously, it can also synchronously parse the input vehicle driving data, i.e., the current vehicle speed, steering wheel angle, and the on / off state of the turn signals. Through neural network operations within the behavior prediction model, the visual information (i.e., the identified lane lines, traffic lights, traffic signs, etc., and the relative positional relationships between the target vehicle and these elements) and the vehicle dynamic information (i.e., the parsed current vehicle speed, steering wheel angle, and the on / off state of the turn signals) are deeply fused and correlated. For each preset target violation behavior, the behavior prediction model can calculate a probability value for triggering the target violation behavior, i.e., the target probability. For example, when the behavior prediction model detects that the front wheels of a target vehicle are approaching the lane line and the steering angle indicates that the target vehicle is moving laterally, it calculates the probability of crossing the lane line. When it detects that the red light is on and the target vehicle is crossing the stop line, it calculates the probability of running a red light. Ultimately, the behavior prediction model can output a result set containing at least one traffic violation and its corresponding probability.

[0033] For example, the aforementioned reasoning based on road images and vehicle driving data could refer to the fusion and correlation analysis of road images and vehicle driving data.

[0034] For example, the behavior prediction model can be deployed in the target vehicle, such as in the vehicle's on-board computing unit.

[0035] For example, a behavior prediction model can be obtained by lightweighting and compressing a pre-defined large model. Specifically, the large model can be weighted and its precision quantized.

[0036] Regarding weight compression, the original large model has a large number of parameters, and its weight file usually occupies a large amount of storage space, making it difficult to deploy directly on resource-constrained in-vehicle computing units. Therefore, through techniques such as model pruning and knowledge distillation, the weight file of the original large model can be compressed to a size of no more than 4GB. This allows the compressed large model to be fully loaded and run in the storage and memory space of the in-vehicle computing unit, providing the basic conditions for the smooth operation of the model on the vehicle.

[0037] For precision quantization, INT8 quantization technology can be used. INT8 quantization refers to converting the weight parameters and activation values ​​in the model from 32-bit floating-point numbers to 8-bit integers for storage and calculation. Since large original models typically use high-precision floating-point numbers for parameter calculation during training, this high-precision calculation, while accurate, is computationally intensive and slow. Therefore, INT8 quantization can reduce the computational resource requirements of large models, decrease memory usage and computation time during runtime, and simultaneously maintain the recognition accuracy of large models, enabling them to run efficiently on in-vehicle chips.

[0038] After the aforementioned lightweight compression, the resulting behavior prediction model can be fully run on the in-vehicle computing unit with high and comprehensive inference speed. Experimental data shows that the behavior prediction model can complete at least two inference operations per second. In other words, the behavior prediction model can predict traffic violations every 0.5 seconds based on the current road environment and vehicle driving data, ensuring rapid response to changes in vehicle driving status and providing ample time for subsequent real-time alerts, thus meeting the stringent real-time requirements of the in-vehicle environment.

[0039] For example, a behavior prediction model may include the following modules: a feature extraction module, a feature fusion module, and a probability output module. The feature extraction module performs deep convolution operations on the input road image, extracting visual features layer by layer, including lane line positions, traffic light status, traffic sign content, and the relative distance between vehicles and lane lines. The feature fusion module aligns and fuses the visual features with the input vehicle driving data (vehicle speed, steering angle, turn signal status) to obtain comprehensive features, establishing a correlation between the external environment and the vehicle's own state. The probability output module can calculate the trigger probability (target probability) of each target violation behavior based on the fused comprehensive features, using fully connected layers and a normalized exponential function.

[0040] The feature extraction module can be a visual encoder built on a deep convolutional neural network to extract multi-scale visual features from road images. Specifically, when a road image frame is input to the feature extraction module, it passes through multiple cascaded convolutional layers, pooling layers, and activation function layers. In the shallow network of the feature extraction module (i.e., the first few convolutional, pooling, and activation function layers), low-level features such as edges, lines, corners, and color gradients can be extracted. These low-level features are used to identify the direction of lane lines, the position of traffic lights, and vehicle outlines. In the deep network of the feature extraction module (i.e., the later convolutional, pooling, and activation function layers), higher-level semantic features can be extracted layer by layer, such as the complete shape of lane lines, the on / off state of red or green lights, the specific meaning of one-way signs or directional arrows, and the relative distance between vehicles and stop lines. After processing by the feature extraction module, the road image is transformed into a set of high-dimensional feature maps, i.e., visual features. These feature maps contain all the key information related to violation judgment in the road image, providing rich input data for subsequent feature fusion and probability output.

[0041] The feature fusion module can be a feature fusion network built on a multi-head attention mechanism, used to deeply correlate and align the visual features output by the feature extraction module with vehicle driving data (vehicle speed, steering angle, and turn signal status). Specifically, when the visual features output by the feature extraction module and the vehicle driving data read from the CAN bus are simultaneously input into the feature fusion module, the module can encode the vehicle driving data. Since vehicle speed, steering angle, and turn signal status are different types of numerical data, the module can normalize and embed them separately: vehicle speed, as a continuous numerical value, can be mapped to a continuous vector space; steering angle, as a continuous numerical value, can be encoded as an angle feature vector; and turn signal status, as a discrete Boolean value (on or off), can be mapped to two independent category embedding vectors. After encoding, the embedding representations corresponding to the above three types of driving data can be concatenated into a multi-dimensional vehicle state feature vector. Then, the multi-head attention mechanism in the feature fusion module can use the visual features as a query and the vehicle state feature vector as a key and value for attention calculation. Through attention calculations, the feature fusion module can learn complex relationships such as "whether the current vehicle speed and steering angle support the judgment when a tendency to cross the line appears in a road image" or "whether the vehicle's actual motion state is crossing the line when the red light is on." Because the multi-head attention mechanism allows the model to learn multiple interaction patterns between visual features and vehicle driving data in parallel from different representation subspaces, it more comprehensively captures the intrinsic connections between the two. Furthermore, the feature fusion module can also include a gating fusion unit to dynamically adjust the weights of visual features and vehicle state feature vectors in the final fusion result. In some scenarios, such as nighttime or rain / fog weather causing blurred visual features, the dependency weight on the vehicle state feature vector can be automatically increased; while under good visual conditions, more emphasis is placed on the analysis of visual features. This dynamic weight adjustment mechanism ensures the robustness and reliability of the fused features under different environments.

[0042] The probability output module can be a parallel classifier based on fully connected layers and a normalized exponential function. It maps the comprehensive features output by the feature fusion module to the probability of the target violation, i.e., the target probability. Specifically, dimensionality reduction and feature abstraction can be performed using a multi-layer fully connected network to obtain intermediate features. The fully connected network can contain multiple hidden layers, each followed by a non-linear activation function to enhance the model's expressive power. The output of the fully connected network can then be input into four parallel, independent output branches. Each branch corresponds to a specific target violation: crossing the line, driving against traffic, running a red light, or changing lanes without signaling. Each branch consists of an independent fully connected layer that converts the corresponding intermediate feature into a numerical value. This numerical value is then input into the normalized exponential function for parallel processing, converting the four values ​​into four probability values, i.e., the target probabilities. These probability values ​​are between 0 and 1, and the sum of the four probability values ​​is 1. Taking the calculation of running a red light as an example, when the comprehensive features output by the feature fusion module contain strong related information such as "red light is on", "vehicle has crossed the stop line" and "vehicle speed is not zero", the fully connected layer of the running red light branch will produce a higher value. After calculation by the normalized exponential function, the final output probability value of running a red light will be significantly higher than the probability values ​​of the other three types of violations.

[0043] For example, the training process of a behavior prediction model includes the following steps: First, a training dataset is constructed containing 1.3 million frames of video images of the road ahead of a training vehicle (similar to a target vehicle). At least 300,000 frames each represent violations such as crossing lane lines, driving against traffic, running red lights, and changing lanes without signaling. Each frame is manually labeled, including whether a violation exists and the specific type of violation. Second, the labeled training dataset is input into the behavior prediction model in batches. The model calculates the predicted probability of a violation for each frame during forward propagation. Then, this predicted value is compared with the actual labeled value of the image, and the prediction error is calculated using a loss function. Finally, the error gradient is propagated back layer by layer using the backpropagation algorithm, and the weight parameters within the behavior prediction model are continuously adjusted until the recognition accuracy of the behavior prediction model on the validation set (similar to the training dataset above) reaches a preset threshold and the loss function converges, thus obtaining the behavior prediction model.

[0044] For example, the reasoning process of a behavior prediction model includes the following steps: First, a real-time road image captured by a camera in front of the target vehicle and vehicle driving data (vehicle speed, steering angle, turn signal status) synchronously read from the CAN bus are used as inputs. These inputs have been trained and deployed in the behavior prediction model within the onboard computing unit. After receiving the input, the model performs calculations sequentially through its internal modules. In the feature extraction stage, key visual elements in the road image, such as lane line positions, whether red lights are on, and the presence of one-way signs, are identified to obtain visual features. In the feature fusion stage, the visual features are jointly analyzed with the current vehicle speed, steering wheel angle, and turn signal status to determine whether the target vehicle's dynamics match the visual environment, resulting in comprehensive features. Finally, four probability values ​​can be calculated based on the comprehensive features, representing the likelihood of triggering four traffic violations in the current scene: crossing lane lines, driving against traffic, running a red light, and changing lanes without signaling.

[0045] For example, S120 may include at least one of the following, but is not limited to: The probability of triggering lane-crossing behavior is obtained by comparing a first distance between the target vehicle's wheels and the lane line with a first distance threshold, and the duration for which the first distance is less than the first distance threshold. (For example, the probability of triggering lane-crossing behavior can be obtained by determining the degree to which the first distance is less than the first distance threshold (or the difference between the first distance threshold and the first distance). The larger the degree or difference, the closer the wheel is to or past the lane line, and the higher the probability of triggering lane-crossing behavior. Similarly, the longer the duration, the higher the probability of triggering lane-crossing behavior. For example, the lateral distance between the wheel and the lane line can be calculated in real time. When the lateral distance is less than 5 centimeters, a timer can be started. If this lateral distance of less than 5 centimeters lasts for more than 0.5 seconds, a higher target probability can be determined, for example, greater than 0.8. If it is only a momentary crossing of the lane line (e.g., 0.2 seconds) or a close proximity to the lane line, it can be determined as unstable driving or vibration, and a lower target probability can be determined, for example, less than 0.5). By comparing the first angle between the target vehicle's direction of travel and the lane arrow with a first angle threshold, and / or by comparing the direction of travel with a one-way street sign, the probability of triggering a wrong-way driving violation can be obtained (for example, the probability of triggering a wrong-way driving violation can be obtained by determining the degree to which the first angle is greater than the first angle threshold; the greater the degree to which the first angle is greater than the first angle threshold, the greater the deviation of the direction of travel from the prescribed direction, and the greater the probability of triggering a wrong-way driving violation. For example, ground arrows can be identified, and the angle between the vehicle's front direction and the arrow can be calculated. If it is greater than 150°, it indicates wrong-way driving; and / or, one-way street or no-entry signs can be identified on the roadside. If they are identified and the vehicle's front is facing the sign, it is determined to be wrong-way driving, and a higher target probability can be determined). The probability of triggering a red-light violation is determined by whether the target vehicle has crossed the stop line and the traffic light status (for example, the greater the distance the target vehicle has crossed the stop line and the longer the traffic light remains red, the higher the probability of triggering a red-light violation. For example, the position of the stop line and the color of the traffic light can be determined first; if a red light is detected and a vehicle is detected crossing the stop line, it is determined as a red-light violation, thus establishing a higher target probability. In addition, if it is determined that the vehicle was already close to the stop line when the yellow light was on but had not stopped, the speed and distance can be used to determine whether it was "running the yellow light" or "unable to stop," and the two can correspond to different target probabilities). By comparing the lateral displacement of the target vehicle with a second distance threshold and the turn signal illumination duration of the target vehicle with a first duration threshold, the probability of triggering the lane change without signaling behavior is obtained. (For example, the probability of triggering the lane change without signaling behavior can be obtained by determining the degree to which the lateral displacement exceeds the second distance threshold. The greater the degree to which the lateral displacement exceeds the second distance threshold and the greater the degree to which the turn signal illumination duration is less than the first duration threshold, the greater the probability of triggering the lane change without signaling behavior. For example, the lateral displacement of the vehicle can be monitored in real time. If the lateral displacement exceeds 0.5 meters, it indicates that the target vehicle is changing lanes. At the same time, the turn signal status on the CAN bus is read: if the turn signal is not lit, it is directly determined as a violation, and a high target probability can be determined. If the turn signal is lit, but the illumination time is less than 3 seconds, it can be determined as "no signal" or "signaling too late", which can also determine a high target probability.)

[0046] For example, to determine the probability of triggering a lane-crossing behavior, a first rule for determining the probability of triggering the lane-crossing behavior can be determined based on the type of lane markings, and the probability of triggering the lane-crossing behavior can be obtained based on the first rule. For example, determining the first rule includes at least one of the following: If the lane markings are solid lines or double yellow lines, then the first rule indicates that: a first distance threshold is greater than or equal to a first value and / or the judgment threshold corresponding to the duration is less than or equal to a second value; wherein, the judgment threshold corresponding to the duration can refer to a value used to judge the magnitude of the duration. Accordingly, the probability of triggering the lane crossing behavior is obtained according to the first rule, including: in response to the first distance being less than the first distance threshold (or the degree to which the first distance is less than the first distance threshold being greater than a degree threshold) and / or in response to the duration being greater than the judgment threshold corresponding to the duration, outputting the probability of triggering the lane crossing behavior that is higher than a preset threshold.

[0047] If the lane markings are dashed, a first rule is established: the probability of triggering a lane-crossing behavior is determined based on the target vehicle's driving intention, a first distance threshold, and a determination threshold corresponding to the duration, wherein the first distance threshold is less than a first value and / or the duration is greater than a second value; wherein, at least one of the target vehicle's turn signal status, steering wheel angle, or lateral speed can be obtained as the driving intention. Accordingly, the probability of triggering a lane-crossing behavior is obtained according to the first rule, including: if the driving intention indicates that the target vehicle is actively changing lanes and the lane-changing direction is consistent with the lane-crossing direction, then the probability of triggering a lane-crossing behavior is reduced; if the driving intention indicates that the target vehicle has no active lane-changing intention, then the probability of triggering a lane-crossing behavior is calculated based on the first distance and duration.

[0048] If the lane line type is a specified type, then the first rule instruction is determined: the probability of triggering the lane crossing behavior is determined based on the historical probability of the corresponding lane crossing behavior (e.g., the probability determined based on multiple consecutive frames of road images acquired before executing S110), a first distance threshold, and the duration. The specified type includes any of the following: lane line types with a blur level greater than a blur level threshold, lane line types with a wear level greater than a wear level threshold, and lane line types with an occlusion level greater than an occlusion level threshold. Accordingly, the probability of triggering the lane crossing behavior is obtained according to the first rule, including: calculating the percentage of frames in the multiple frames of road images that are determined to trigger the lane crossing behavior; if the percentage of frames is greater than or equal to a preset percentage threshold, then the historical probability is set to a first historical probability value; if the percentage of frames is less than the preset percentage threshold, then the historical probability is set to a second historical probability value, where the first historical probability value is greater than the second historical probability value; the historical probability is used as the probability of triggering the lane crossing behavior, or the historical probability is fused with the probability calculated based on the first distance and duration to obtain the probability of triggering the lane crossing behavior.

[0049] For example, different thresholds can be set for different types of lane markings (solid lines, dashed lines, double yellow lines). Specifically, when a target vehicle is detected crossing a solid line or double yellow line, since these two types of lane markings usually indicate a strict prohibition on crossing, a higher violation detection sensitivity can be assigned (e.g., setting a larger first distance threshold and a smaller duration threshold). Even if the lateral distance is slightly larger or the duration is slightly shorter, a higher target probability can still be determined. When a target vehicle is detected crossing a dashed line, a comprehensive judgment can be made based on the target vehicle's driving intention. For example, if the target vehicle activates its turn signal while crossing the dashed line, the target probability can be reduced; conversely, if the target vehicle crosses the dashed line but does not activate its turn signal, the corresponding target probability can be calculated based on the lateral distance and duration using a lower detection sensitivity. In addition, for lane markings in complex scenarios (such as blurred, worn, or snow-covered lane markings), the detection results of historical frames can be combined for smoothing to avoid false alarms caused by single-frame false detections.

[0050] For example, to determine the probability of triggering a wrong-way driving violation, the probability can be obtained based on the distance and duration of the wrong-way driving. Specifically, after identifying a target vehicle that may enter a wrong-way driving state, the vehicle's trajectory can be continuously tracked, and the following can be calculated in real time: the distance the target vehicle travels in the wrong-way driving area and the duration of the wrong-way driving state. If the vehicle has just crossed the boundary of the wrong-way driving area, i.e., the wrong-way driving distance is short (e.g., less than 1 meter) and the wrong-way driving duration is very short (e.g., less than 0.5 seconds), and the driver is simultaneously detected actively correcting the direction (e.g., increasing the steering wheel angle in the opposite direction), it can be judged as "accidentally entering the edge of wrong-way driving" or "slightly crossing the line and driving in the wrong direction," and a lower target probability is determined. If the wrong-way driving distance continues to increase (e.g., more than 5 meters), or the wrong-way driving duration is long (e.g., more than 2 seconds), and the vehicle's heading always maintains a large angle with the correct direction, it is judged as "substantial wrong-way driving," and a higher target probability is determined.

[0051] In the above-mentioned embodiment of S120, the probability of triggering a traffic violation can be predicted by deploying a lightweight large model, namely a behavior prediction model, on the vehicle side. This can effectively overcome the recognition bottleneck of traditional small models in complex scenarios. Since the behavior prediction model has incorporated a large amount of driving data in adverse environments such as nighttime, backlight, rain, and fog during the training phase, it can still maintain a high accuracy in identifying traffic violations in the above scenarios. The detection rate can be stabilized at over 90%, while the false alarm rate is controlled within 1%, avoiding the interference to the driver caused by frequent false alerts due to environmental changes.

[0052] Furthermore, since the inference process is completed locally on the in-vehicle computing unit, there is no need to upload data to the cloud. This eliminates the impact of network latency and communication blind spots (such as tunnels and underground parking garages) on real-time performance. Even in areas where 4G / 5G signals are unavailable, it can still operate stably and complete traffic violation prediction and alerts within 0.5 seconds. At the same time, because road images and driving data are always stored on the vehicle, user privacy and driving safety are protected, avoiding the data leakage risks that may arise during cloud transmission.

[0053] S130: Based on the target probability, predict whether the target vehicle will trigger the target violation.

[0054] Specifically, cross-validation can be performed on target probability and vehicle driving data to predict whether the target vehicle will trigger the target violation.

[0055] For example, it can be predicted that a target vehicle will trigger a target violation when the target probability is greater than a probability threshold (e.g., 0.8) and cross-validation of the vehicle driving data passes. If the target probability is greater than the probability threshold but cross-validation of the vehicle driving data fails, it can be predicted that the target vehicle will not trigger the target violation.

[0056] For example, the above cross-validation of the target probability and vehicle driving data includes: obtaining vehicle state parameters related to the target violation in the vehicle driving data; and performing cross-validation based on the vehicle state parameters and the behavior determination conditions corresponding to the target violation (such as the behavior determination conditions corresponding to the target probability).

[0057] For example, for the act of crossing the line, the relevant vehicle status parameters could be vehicle speed and steering wheel angle. For the act of running a red light, the relevant vehicle status parameters could be vehicle speed. For the act of changing lanes without using turn signals, the relevant vehicle status parameters could be turn signal status and steering wheel angle.

[0058] For example, regarding the behavior of crossing the line, assuming the target probability output by the behavior prediction model is 0.9 and the vehicle speed is read as 30 km / h, and the "line crossing judgment condition" set for the behavior is: target probability greater than 0.8 and vehicle speed greater than 5 km / h, then the current data (0.9>0.8, 30>5) meets the conditions, so it can be determined that the cross-validation passed and the prediction will trigger the line crossing behavior. Assuming the target probability output by the behavior prediction model is 0.85 (possibly because the car stopped on the shaded line), and the vehicle speed is read as 0 km / h, then the current data does not meet the condition: vehicle speed greater than 5 km / h, so it can be determined that the cross-validation failed and the prediction will not trigger the line crossing behavior.

[0059] For example, regarding the behavior of changing lanes without signaling, assuming the target probability output by the behavior prediction model is 0.88, and it reads "steering wheel angle change greater than 10°" and "left turn signal status is off"; the behavior judgment conditions are set as follows: target probability greater than 0.8, obvious steering wheel turning action, and the corresponding turn signal not being turned on. Then, it can be determined that the cross-validation passed, and the prediction will trigger the behavior of changing lanes without signaling.

[0060] In the embodiment of S130 described above, by cross-validating the target probability output by the model with the vehicle's own driving data, it is possible to accurately distinguish between real traffic violations and visual interference in highly dynamic real-world driving environments with drastic changes in lighting. For example, when a vehicle is stationary waiting at a red light, even if the wheels happen to be on the lane line, it will not be incorrectly predicted as triggering a lane-crossing behavior. Therefore, through the above cross-validation mechanism, the technical problem of false alarms or missed alarms easily generated by single visual recognition in complex scenarios can be solved, controlling the false alarm rate to below 1%, and improving robustness and user acceptance in real-world driving environments.

[0061] In some embodiments, in response to a prediction that the target vehicle has triggered a target violation, the system may also prompt the target vehicle to trigger a target violation via voice and / or display screen.

[0062] For example, different warning messages can be broadcast through the vehicle's speakers for different traffic violations, such as "Please note, you are about to cross the line" or "Red light, please stop." A warning icon can also pop up on the target vehicle's central control screen or dashboard, accompanied by a flashing red frame.

[0063] In the above embodiments, the entire process from predicting the occurrence of a violation to issuing a reminder can be controlled within 0.5 seconds, thereby effectively avoiding the occurrence of violations.

[0064] Figure 2 This is a schematic diagram of a violation prediction device provided in an embodiment of this application. Figure 2 As shown, the violation prediction device 200 includes: The acquisition module 210 is used to acquire road images of the road where the target vehicle is traveling and vehicle driving data; The reasoning module 220 is used to reason about road images and vehicle driving data through a preset behavior prediction model to obtain the target probability of the target vehicle triggering the target violation behavior. The prediction module 230 is used to predict whether the target vehicle will trigger the target violation based on the target probability.

[0065] In some embodiments, the reasoning module 220 is specifically used for at least one of the following: obtaining the probability of triggering a lane crossing behavior by comparing a first distance between the target vehicle's wheels and the lane line with a first distance threshold and the duration during which the first distance is less than the first distance threshold; obtaining the probability of triggering a wrong-way driving violation by comparing a first angle between the target vehicle's driving direction and the lane arrow with a first angle threshold, and / or comparing the driving direction with a one-way street sign; obtaining the probability of triggering a red light running behavior by whether the target vehicle crosses the stop line and the traffic light status; and obtaining the probability of triggering a lane change without signaling behavior by comparing the target vehicle's lateral displacement with a second distance threshold and the target vehicle's turn signal illumination duration with a first duration threshold.

[0066] In some embodiments, the reasoning module 220 is specifically used for at least one of the following: determining a first rule for judging the probability of triggering a lane crossing behavior based on the type of lane line, obtaining the probability of triggering a lane crossing behavior based on the first rule; and obtaining the probability of triggering a wrong-way driving violation based on the wrong-way driving distance and the duration of the wrong-way driving behavior.

[0067] In some embodiments, the inference module 220 is specifically used for at least one of the following: if the lane line type is a solid line or a double yellow line, then determine a first rule indication: a first distance threshold is greater than or equal to a first value and / or a determination threshold corresponding to the duration is less than or equal to a second value; if the lane line type is a dashed line, then determine a first rule indication: determine the probability of triggering a lane crossing behavior based on the target vehicle's driving intention, the first distance threshold, and the duration, wherein the first distance threshold is less than the first value and / or the duration is greater than the second value; if the lane line type is a specified type, then determine a first rule indication: determine the probability of triggering a lane crossing behavior based on the historical probability of triggering a lane crossing behavior, the first distance threshold, and the duration, wherein the specified type includes any of the following: lane line types with a lane line blur degree greater than a blur degree threshold, lane line types with a lane line wear degree greater than a wear degree threshold, and lane line types with a lane line occlusion degree greater than an occlusion degree threshold.

[0068] In some embodiments, the prediction module 230 is specifically used to: perform cross-validation on the target probability and vehicle driving data to predict whether the target vehicle will trigger the target violation.

[0069] In some embodiments, the prediction module 230 is specifically used to: obtain vehicle state parameters related to the target violation in vehicle driving data; and perform cross-validation based on the vehicle state parameters and the behavior judgment conditions corresponding to the target violation.

[0070] In some embodiments, the behavior prediction model is deployed in the target vehicle; the behavior prediction model is obtained by lightweight compression of a pre-defined large model.

[0071] In some embodiments, the violation prediction device 200 further includes a prompting module for prompting the target vehicle to trigger the target violation via voice and / or display screen in response to the prediction that the target vehicle has triggered the target violation.

[0072] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 2 The apparatus 200 shown can execute the above-described method embodiments, and the aforementioned and other operations and / or functions of each module in the apparatus 200 are respectively for implementing the corresponding processes in the above-described methods. For the sake of brevity, they will not be described in detail here.

[0073] The apparatus 200 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0074] Figure 3 A schematic diagram of an electronic device provided in an embodiment of this application.

[0075] like Figure 3 As shown, the electronic device 300 may include: The device includes a memory 310 and a processor 320. The memory 310 stores computer programs and transfers the program code to the processor 320. In other words, the processor 320 can retrieve and run the computer program from the memory 310 to implement the methods described in this application embodiment. The electronic device can be the target vehicle described above, or it can be installed in the target vehicle, or it can communicate with the target vehicle.

[0076] For example, the processor 320 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0077] In some embodiments of this application, the processor 320 may include, but is not limited to: General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0078] In some embodiments of this application, the memory 310 includes, but is not limited to: Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0079] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 310 and executed by the processor 320 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0080] like Figure 3 As shown, the electronic device may further include: Transceiver 330, which can be connected to processor 320 or memory 310.

[0081] The processor 320 can control the transceiver 330 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 330 may include a transmitter and a receiver. The transceiver 330 may further include antennas, and the number of antennas may be one or more.

[0082] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0083] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0084] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, the computer can perform all or part of the corresponding processes in the methods of the embodiments of this application, producing the functions achievable by the methods of the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid state disks (SSDs)).

[0085] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0087] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0088] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of predicting a violation, characterized by, include: Acquire road images and vehicle driving data of the route traveled by the target vehicle; By reasoning about the road images and vehicle driving data using a preset behavior prediction model, the target probability of the target vehicle triggering the target violation behavior is obtained. Based on the target probability, predict whether the target vehicle will trigger the target violation.

2. The method of claim 1, wherein, The step of reasoning from the road image and the vehicle driving data using a preset behavior prediction model to obtain the target probability of the target vehicle triggering the target violation includes at least one of the following: The probability of triggering lane crossing behavior is obtained by comparing a first distance between the target vehicle's wheels and the lane line with a first distance threshold and the duration during which the first distance is less than the first distance threshold. By comparing the first angle between the target vehicle's driving direction and the lane arrow with a first angle threshold, and / or by comparing the driving direction with a one-way street sign, the probability of triggering a wrong-way driving violation is obtained. The probability of triggering a red light violation is obtained by checking whether the target vehicle has crossed the stop line and the status of the traffic light. The probability of triggering lane change without signaling is obtained by comparing the lateral displacement of the target vehicle with a second distance threshold and the turn signal illumination duration of the target vehicle with a first duration threshold.

3. The method according to claim 2, characterized in that, It also includes at least one of the following: A first rule is determined based on the type of lane markings to determine the probability of triggering the lane crossing behavior, and the probability of triggering the lane crossing behavior is obtained based on the first rule. The probability of triggering the wrong-way driving violation is obtained based on the wrong-way driving distance and the duration of the wrong-way driving behavior.

4. The method according to claim 3, characterized in that, The first rule for determining the probability of triggering the lane crossing behavior based on the type of lane markings includes at least one of the following: If the lane line type is a solid line or a double yellow line, then the first rule indicates that the first distance threshold is greater than or equal to a first value and / or the determination threshold corresponding to the duration is less than or equal to a second value. If the lane line type is a dashed line, then the first rule instruction is determined: the probability of triggering the lane crossing behavior is determined based on the driving intention of the target vehicle, the first distance threshold, and the duration, wherein the first distance threshold is less than a first value and / or the determination threshold corresponding to the duration is greater than a second value; If the lane line type is a specified type, then the first rule instruction is determined: the probability of triggering the lane line behavior is determined based on the historical probability of triggering the lane line behavior, the first distance threshold, and the duration. The specified type includes any of the following: lane line types with a lane line blur degree greater than a blur degree threshold, lane line types with a lane line wear degree greater than a wear degree threshold, and lane line types with a lane line occlusion degree greater than an occlusion degree threshold.

5. The method according to claim 1, characterized in that, The step of predicting whether the target vehicle will trigger the target violation based on the target probability includes: Cross-validation is performed on the target probability and the vehicle driving data to predict whether the target vehicle will trigger the target violation.

6. The method according to claim 5, characterized in that, The cross-validation of the target probability and the vehicle driving data includes: Obtain vehicle status parameters related to the target violation from the vehicle driving data; The cross-validation is performed based on the vehicle status parameters and the behavior determination conditions corresponding to the target violation.

7. The method according to any one of claims 1-6, characterized in that, The behavior prediction model is deployed in the target vehicle; The behavior prediction model is obtained by lightweighting and compressing a pre-set large model.

8. The method according to any one of claims 1-6, characterized in that, Also includes: In response to the prediction that the target vehicle has triggered the target violation, the system prompts the target vehicle to trigger the target violation via voice and / or display screen.

9. A device for predicting traffic violations, characterized in that, include: The acquisition module is used to acquire road images of the roads traveled by the target vehicle and vehicle driving data. The reasoning module is used to reason about the road image and the vehicle driving data through a preset behavior prediction model to obtain the target probability that the target vehicle will trigger the target violation behavior. The prediction module is used to predict whether the target vehicle will trigger the target violation based on the target probability.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-8.

Images