Driving behavior analysis and reasoning method for vehicle recorders based on visual large model

Abstract

The application relates to the technical field of video analysis, and particularly discloses a driving record video driving behavior analysis and reasoning method based on a visual large model, wherein a driving record video frame sequence and a timestamp are collected; a visual large model is used to extract a high-dimensional visual feature sequence, and a steering light brightness pulse, a brake light state and a lateral deviation speed are extracted in parallel as atomic event numerical trajectories; a behavior-scene joint coding is generated; a differentiable time sequence logic state machine is maintained according to the atomic event numerical trajectories, a logic conflict in a reasoning text token generation process is detected, and a token probability distribution is adjusted through a negative gradient to suppress an output that violates a time sequence physical law; and the adjusted token sequence is reorganized into a structured analysis result containing a behavior classification, a risk level, a causal explanation and a confidence degree; and the application uses low-layer physical signals to constrain a reasoning process of the visual large model, thereby improving the reliability of driving behavior analysis under a poor environment.

Description

Technical Field

[0001] This invention relates to the field of video analytics, specifically to a method for analyzing and reasoning driving behavior from dashcam videos based on a large visual model. Background Technology

[0002] Currently, dashcams are widely used in various vehicles, and the recorded video data is frequently used in scenarios such as determining liability in traffic accidents, assessing driving behavior, and handling insurance claims. Traditional analysis methods mainly rely on manual playback of video segments for judgment, which is inefficient and highly subjective. In recent years, with the development of deep learning technology, behavior recognition methods based on convolutional neural networks have emerged, which can automatically detect abnormal events such as sudden braking and sharp turns from videos.

[0003] The existing technology has the following shortcomings:

[0004] When dashcam video suffers from visual feature degradation due to nighttime, rain, or lens damage, how can we ensure that the inference output of the visual model strictly follows the underlying physical laws such as the timing of turn signal flashing, the duration of brake light status, and the lateral deviation trajectory of the vehicle? This would prevent the visual model from generating driving behavior assertions that contradict reality due to visual illusions, and achieve credible inference from "visual impressions" to "physical fact constraints." Summary of the Invention

[0005] The purpose of this invention is to provide a method for analyzing and reasoning driving behavior from dashcam videos based on a large visual model, in order to solve the problems mentioned above.

[0006] The objective of this invention can be achieved through the following technical solutions: A method for analyzing and inferring driving behavior from dashcam videos based on a large visual model includes the following steps: S1: Collect the continuous video frame sequence output by the dashcam and the timestamp corresponding to each frame as the raw input data; S2: Use a large visual model to perform spatiotemporal feature decomposition on the original input data to generate a high-dimensional visual feature sequence. Simultaneously extract the turn signal brightness pulse timing curve, brake light state duration sequence, and vehicle lateral offset speed trajectory as atomic event numerical trajectories. S3: Map the high-dimensional visual feature sequence to preliminary driving behavior labels and road scene context embeddings to form behavior-scene joint encoding; S4: Maintain a differentiable temporal logic state machine based on the numerical trajectory of atomic events, and in the process of converting behavior-scene joint encoding into a sequence of inference text tokens, perform logical conflict detection between each token to be generated and the event facts in the state machine, and adjust the probability distribution of the tokens in reverse through gating gradients to suppress token outputs that violate the temporal physical laws. S5: Reassemble the adjusted token sequence into the final inference text, and output a structured analysis result that includes driving behavior classification, risk level, causal explanation and logical consistency confidence.

[0007] As a further aspect of the present invention: Spatiotemporal feature decomposition of the original input data is performed using a large visual model, specifically including: A continuous sequence of video frames is input into a large visual model consisting of multiple stacked spatiotemporal decomposition blocks in chronological order. Each spatiotemporal decomposition block first separates the low-frequency components in the temporal domain through a local sliding window attention layer, and then extracts the high-frequency changes in the temporal domain through an adaptive gated recurrent unit. The low-frequency components output by each spatiotemporal decomposition block are accumulated frame by frame to generate a high-dimensional visual feature sequence. At the same time, the high-frequency variation features are fed into three parallel numerical projection heads. The first projection head outputs the brightness pulse intensity value of the region of interest of the turn signal, the second projection head outputs the cumulative value of the binary state duration of the brake light region, and the third projection head outputs the lateral displacement velocity of the center point of the vehicle detection box. The values ​​output from the three projectors are arranged in chronological order to obtain the turn signal brightness pulse timing curve, the brake light state duration sequence, and the vehicle lateral offset velocity trajectory, which are used as atomic event numerical trajectories.

[0008] As a further aspect of the present invention: the separation of time-domain low-frequency components through a local sliding window attention layer specifically includes: The features of consecutive video frames are divided into multiple overlapping windows along the time sequence. Each window contains a fixed number of frames, and adjacent windows share half of the frames. Within each window, an attention weight that decays exponentially with time offset is calculated for each frame, and the features of all frames within the window are weighted and summed to obtain the time-domain low-frequency compressed vector of the window. The temporal low-frequency compression vectors of adjacent windows are linearly interpolated and fused according to the number of overlapping frames to generate a temporal low-frequency component sequence that maintains the same frame rate as the original.

[0009] As a further aspect of the present invention: S3 specifically includes: The high-dimensional visual feature sequence is divided into multiple non-overlapping segments along the time dimension. The temporal maximum entropy pooling value is calculated in each segment. The key frame features with the most information in the non-overlapping segments are extracted. The key frame features are then matched with a predefined behavioral primitive feature library using cosine similarity to output preliminary driving behavior labels. Differential attention is performed on the high-dimensional visual feature sequence and the scene context embedding output from the previous time step to obtain the temporal change saliency map of each pixel position. Then, the spatial relative positional relationship and topological connection order between road participants are extracted from the saliency map to output the road scene context embedding. The feature vectors corresponding to the initial driving behavior labels and the feature vectors embedded in the road scene context are input into a trainable dual-gated fusion unit. The two are then summed element-wise using adaptive weights to generate a behavior-scene joint encoding.

[0010] As a further aspect of the present invention: obtaining the temporal variation saliency map of each pixel position specifically includes: The original difference feature map is obtained by performing a pixel-by-pixel subtraction operation between the feature map of the current frame in the high-dimensional visual feature sequence and the corresponding spatial location feature map embedded in the scene context of the previous time step. For each pixel position in the original difference feature map, take a local neighborhood window centered on itself, calculate the standard deviation of the difference values ​​of all pixels within the window, and then multiply the standard deviation by the difference value of the center pixel to obtain the difference value for local contrast enhancement. The local contrast enhancement difference value at each pixel location is sequentially passed through a three-segment nonlinear mapping. The first segment compresses negative values ​​to zero, the second segment linearly amplifies small values ​​near zero, and the third segment saturates large values ​​to a fixed upper limit, outputting a saliency map of the temporal changes at each pixel location.

[0011] As a further aspect of the present invention: S4 specifically includes: The turn signal brightness pulse timing curve in the atomic event numerical trajectory is divided into rising edge sets according to time windows. At the same time, the brake light state duration sequence is converted into Boolean state flags, and the lateral displacement cumulative amount is obtained by integrating the vehicle lateral offset velocity trajectory. The flash counter, brake timer, and offset start flag stored inside the state machine are updated based on the rising edge set, Boolean state flags, and lateral displacement accumulation to obtain the physical fact state vector at the current moment. When generating each reasoning text token, the predicate embedding corresponding to the reasoning text token is used to calculate the dot product similarity with the physical fact state vector. If the similarity is lower than the threshold, a negative gradient is generated. The negative gradient is backpropagated to the probability distribution of the token generation layer to reduce the probability of selecting tokens that violate the temporal physical laws.

[0012] As a further aspect of the present invention: the process of obtaining the physical fact state vector specifically includes: The time interval between two adjacent rising edges in the rising edge set is compared with a preset flashing period threshold. If the interval is less than the threshold, the flashing counter is incremented; otherwise, the flashing counter is reset to zero. The braking timer is updated based on the duration of the Boolean state flag being true, and the peak value of the timer is retained as the duration of a braking event when the state flag changes from true to false. The lateral offset multiple is obtained by dividing the cumulative lateral displacement by the nominal vehicle width. When the lateral offset multiple changes from less than 1 to greater than 1, the current moment is recorded as the offset start mark, and the offset start mark is continuously output until the multiple falls back to less than 1. The updated blink counter, the current value of the brake timer, and the presence or absence of the offset start marker are combined into a fixed-length numerical vector, which serves as the physical fact state vector at the current moment.

[0013] As a further aspect of the present invention: S5 specifically includes: The adjusted token sequence is aligned with the timestamp of the original video frame according to the generation time, and then concatenated into a natural language phrase sequence, where each phrase contains the subject of the behavior, the action predicate, and the time interval. The severity coefficients of behavioral predicates are extracted from natural language phrase sequences, multiplied by the weights of the corresponding behaviors in a preset risk matrix, and summed to obtain the risk level of driving behavior classification. The absolute values ​​of the negative gradients generated by each logical conflict detection are accumulated and then divided by the total length of the token sequence to obtain the logical consistency confidence level between 0 and 1. The risk level and logical consistency confidence level are appended to the end of the natural language phrase sequence, and causal explanatory phrases are inserted based on the conflict detection records to output structured analysis results.

[0014] As a further aspect of the present invention: the calculation process of the severity coefficient is as follows: Each behavioral predicate in the natural language phrase sequence is mapped to the semantic embedding space. The semantic distance between the corresponding embedding and each anchor vector in the predefined danger level anchoring vector library is calculated. The danger level score corresponding to the minimum distance is selected as the baseline severity of the predicate. Obtain the time interval length corresponding to the behavior predicate, compare the time interval length with the short-term danger threshold, and calculate the time compression factor when the length is less than the threshold. The time compression factor is equal to the square of the threshold divided by the length. Multiplying the baseline severity by the time compression factor yields the severity coefficient of the behavioral predicate, where a shorter time interval results in a larger severity coefficient.

[0015] The beneficial effects of this invention are:

[0016] (1) By introducing a differentiable temporal logic state machine to perform real-time conflict detection and gradient adjustment on the tokens generated by the visual large model, the model can actively suppress the output of erroneous tokens that violate the temporal rules based on the physical facts in the numerical trajectory of atomic events under the condition of visual feature degradation, thereby reducing false inference conclusions caused by visual illusions and improving the output reliability of driving behavior analysis in harsh environments.

[0017] (2) By extracting the turn signal brightness pulse timing curve, brake light state duration sequence and vehicle lateral offset speed trajectory in parallel as independent atomic event numerical trajectories, and combining them with the time interval length of the behavioral predicate to calculate the severity coefficient, the risk level assessment integrates the timing characteristics of low-level physical signals, and realizes a more granular quantitative assessment of driving behavior risk. Attached Figure Description

[0018] The invention will now be further described with reference to the accompanying drawings.

[0019] Figure 1 This is a flowchart of the method of the present invention;

[0020] Figure 2 This is a flowchart of the process of obtaining the physical fact state vector in this invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please see Figure 1 As shown, this invention is a method for analyzing and reasoning driving behavior from dashcam videos based on a large visual model, comprising the following steps: S1: Collect the continuous video frame sequence output by the dashcam and the timestamp corresponding to each frame as the raw input data; S2: Use a large visual model to perform spatiotemporal feature decomposition on the original input data to generate a high-dimensional visual feature sequence. Simultaneously extract the turn signal brightness pulse timing curve, brake light state duration sequence, and vehicle lateral offset speed trajectory as atomic event numerical trajectories. S3: Map the high-dimensional visual feature sequence to preliminary driving behavior labels and road scene context embeddings to form behavior-scene joint encoding; S4: Maintain a differentiable temporal logic state machine based on the numerical trajectory of atomic events, and in the process of converting behavior-scene joint encoding into a sequence of inference text tokens, perform logical conflict detection between each token to be generated and the event facts in the state machine, and adjust the probability distribution of the tokens in reverse through gating gradients to suppress token outputs that violate the temporal physical laws. S5: Reassemble the adjusted token sequence into the final inference text, and output a structured analysis result that includes driving behavior classification, risk level, causal explanation and logical consistency confidence.

[0023] In S1, the continuous video frame sequence output by the dashcam and the timestamp corresponding to each frame are collected as raw input data, specifically including: The raw video stream is obtained by connecting the dashcam's data output port to the computing unit's data input port via a data cable, or by directly reading video files from the dashcam's built-in memory card. This raw video stream uses a fixed frame rate encoding of 25 or 30 frames per second, with each frame containing the road ahead view and a portion of the vehicle's dashboard at the corresponding moment. Timestamps are extracted frame-by-frame from the video stream's encapsulation information; these timestamps record the offset of each frame relative to the start time of the video, measured in milliseconds. Consecutive video frames are arranged in ascending chronological order to form a continuous video frame sequence, and each frame is stored in pairs with its corresponding timestamp, constituting the raw input data. If the dashcam supports GPS time synchronization, the GPS-provided time reference is used as the absolute time reference for the timestamps; otherwise, the time reference generated by the dashcam's internal crystal oscillator is used. During acquisition, the original resolution and color space of the video are preserved without compression or format conversion to ensure the integrity and losslessness of the input data for subsequent processing.

[0024] In S2, a large visual model is used to perform spatiotemporal feature decomposition on the original input data to generate a high-dimensional visual feature sequence. Simultaneously, the turn signal brightness pulse timing curve, brake light state duration sequence, and vehicle lateral offset velocity trajectory are extracted in parallel as atomic event numerical trajectories, specifically including: First, a large visual model is constructed, consisting of five stacked spatiotemporal decomposition blocks. The obtained continuous video frame sequence is input into the first spatiotemporal decomposition block in chronological order. Within each spatiotemporal decomposition block, the following operations are performed: first, the temporal low-frequency components are separated using a local sliding window attention layer; then, high-frequency temporal changes are extracted using an adaptive gated recurrent unit. Specifically, the local sliding window attention layer is implemented as follows: the input continuous video frame features are divided into multiple overlapping windows in chronological order, each window containing 16 frames, with adjacent windows sharing 8 frames; within each window, an attention weight that decays exponentially with the time offset is calculated for each frame, with a larger offset resulting in a smaller weight, and a decay coefficient of 0.8; then, the features of all frames within the window are weighted and summed to obtain the temporal low-frequency compression vector for that window; the temporal low-frequency compression vectors of adjacent windows are linearly interpolated and fused according to the number of overlapping frames, meaning the final value of each overlapping frame is equal to the weighted average of the low-frequency compression values ​​of the two windows at that frame position, with the weights being the distances to the center of the preceding and following windows, respectively, thus generating a temporal low-frequency component sequence consistent with the original frame rate. The adaptive gated loop unit receives the above-mentioned time-domain low-frequency component sequence and extracts the time-domain high-frequency variation features through the control of the internal update gate and reset gate, wherein the update gate coefficient is 0.7 and the reset gate coefficient is 0.3.

[0025] Secondly, the low-frequency temporal components output by each spatiotemporal decomposition block are accumulated frame by frame. That is, for each time position, the low-frequency feature values ​​output by the five spatiotemporal decomposition blocks are summed to obtain a high-dimensional visual feature sequence. Simultaneously, the high-frequency temporal variation features output by each spatiotemporal decomposition block are fed into three parallel numerical projection heads. The first projection head consists of two fully connected layers. Its input is the portion of the high-frequency variation features corresponding to the region of interest for the turn signal, and its output is an integer between 0 and 255, representing the brightness pulse intensity value of that region in the current frame. The second projection head consists of a single binary classification layer. Its input is the portion of the high-frequency variation features corresponding to the brake light region, and its output is 0 or 1, representing the brake light on / off state. The number of consecutive frames with an input of 1 is accumulated as the duration of that state. The third projection head consists of a single linear regression layer. Its input is the portion of the high-frequency variation features corresponding to the center point of the vehicle detection box, and its output is a floating-point number representing the change in lateral displacement between adjacent frames. This floating-point number is then divided by the time interval between adjacent frames to obtain the lateral displacement velocity.

[0026] Finally, the brightness pulse intensity values ​​output by the first projection head are arranged in chronological order to obtain the turn signal brightness pulse timing curve; the cumulative values ​​of the binary state durations output by the second projection head are arranged in chronological order to obtain the brake light state duration sequence; and the lateral displacement velocities output by the third projection head are arranged in chronological order to obtain the vehicle's lateral offset velocity trajectory. These three curves, sequences, and trajectories are collectively output as atomic event numerical trajectories, used for logical constraints in subsequent steps.

[0027] In S3, high-dimensional visual feature sequences are mapped to initial driving behavior labels and road scene context embeddings, forming a behavior-scene joint encoding, specifically including: First, the high-dimensional visual feature sequence is divided into multiple non-overlapping segments along the time dimension, with each segment containing 8 frames. Within each segment, the temporal maximum entropy pooling value is calculated: for the feature vector of each frame in the segment, its information entropy is calculated by multiplying the negative probability by the sum of the log probabilities, where the probability is the normalized distribution of the frame's feature value in the feature space. The frame with the highest entropy value is selected as the keyframe with the most information in that segment, and its feature vector is extracted. A pre-built feature library of behavioral primitives is established, containing feature vectors of 20 common driving behavior primitives, such as "left turn," "right turn," "emergency braking," and "normal deceleration." The feature vectors of each primitive are obtained by averaging a large number of labeled samples. The extracted keyframe feature vectors are matched with the feature vectors of each primitive in the library using cosine similarity. The cosine similarity is equal to the dot product of the two vectors divided by the product of their magnitudes. The primitive with the highest similarity and a similarity greater than the threshold of 0.6 is selected as the initial driving behavior label for that segment.

[0028] Next, a pixel-by-pixel subtraction operation is performed between the feature map of the current frame in the high-dimensional visual feature sequence and the feature map of the corresponding spatial location in the scene context embedding output at the previous time step to obtain the original difference feature map. The initial value of the scene context embedding at the previous time step is set to zero. Then, for each pixel position in the original difference feature map, a local neighborhood window of 3 pixels by 3 pixels is taken with itself as the center. The standard deviation of all pixel difference values ​​within the window is calculated. The standard deviation is calculated as follows: first, the arithmetic mean of the 9 difference values ​​within the window is calculated; then, the square of the difference between each difference value and the mean is calculated; the average of these squares is calculated; and finally, the square root is taken. The obtained standard deviation is multiplied by the difference value of the center pixel to obtain the difference value for local contrast enhancement. Next, the local contrast enhancement difference value at each pixel location is sequentially passed through a three-segment non-linear mapping: In the first segment, if the value is less than 0, it is compressed to 0; in the second segment, if the value is between 0 and 0.2, it is linearly amplified by a factor of 2, meaning the output equals the input multiplied by 2; in the third segment, if the value is greater than 0.2, it is saturated to a fixed upper limit of 1.0. After this mapping, a temporal saliency map of each pixel location is output.

[0029] Then, the spatial relative positions and topological connection order among road participants are extracted from the aforementioned temporal change saliency map. Specifically, the saliency map is binarized, and pixels greater than 0.5 are marked as foreground, resulting in multiple connected regions, each corresponding to one road participant. The center point coordinates of each connected region are calculated and sorted in ascending order according to the center point's ordinate (i.e., the image row coordinates) to obtain the distance order of road participants in the image. Simultaneously, the Euclidean distance between the center points of every two connected regions is calculated; those with a distance less than 50 pixels are considered adjacent, and the adjacency relationship is recorded. This yields the spatial relative positions and topological connection order among road participants, and the output is a road scene context embedding, which is a vector containing the position, distance, and order information of all participants.

[0030] Finally, the feature vectors corresponding to the initial driving behavior labels and the feature vectors embedded in the road scene context are input into a trainable dual-gated fusion unit. This fusion unit contains two parallel gating networks that calculate the weights of the behavior features and context features respectively, with the sum of the weights being 1. Specifically, the behavior feature vector is input into the first gating network, which is a single-layer fully connected layer with a logistic function, outputting a behavior weight between 0 and 1; the context feature vector is input into the second gating network, outputting a context weight, which is equal to 1 minus the behavior weight. Then, the behavior feature vector and the context feature vector are summed element-wise with weights, meaning the value at each position equals the behavior weight multiplied by the behavior feature value at that position, plus the context weight multiplied by the context feature value at that position, resulting in the behavior-scene joint encoding.

[0031] Please see Figure 2 As shown, in S4, a differentiable temporal logic state machine is maintained based on the numerical trajectories of atomic events. During the process of converting the behavior-scene joint encoding into a sequence of inference text tokens, logical conflict detection is performed between each token to be generated and the event facts in the state machine. The probability distribution of the tokens is adjusted inversely through gating gradients to suppress token outputs that violate temporal physical laws. Specifically, this includes: First, three parallel numerical sequences are extracted from the obtained atomic event numerical trajectories. For the turn signal brightness pulse timing curve, a time window of 1 second in 0.1-second increments is used. Within each time window, points where the brightness value jumps from below 30 to above 150 are detected, and the corresponding time position is recorded as the rising edge. All rising edges are arranged in chronological order to form a rising edge set. For the brake light state duration sequence, the cumulative duration value at each time point is converted into a Boolean state flag: if the duration is greater than 0.1 seconds, the flag is true; otherwise, it is false. For the vehicle lateral offset velocity trajectory, the trapezoidal integral method is used to obtain the cumulative lateral displacement. That is, starting from the initial frame, the lateral displacement velocity of each frame is multiplied by the frame interval of 0.04 seconds (corresponding to 25 frames per second) to obtain the displacement increment of that frame. All increments are summed to obtain the cumulative value.

[0032] Secondly, the three variables registered internally in the state machine are updated based on the rising edge set, Boolean state flags, and cumulative lateral displacement: the blink counter, the braking timer, and the offset start flag. Specifically, the update method is as follows: iterate through the time interval between two adjacent rising edges in the rising edge set. If the interval is less than 0.4 seconds, the blink counter is incremented by 1; otherwise, it is reset to 0. For the Boolean state flag, when the flag is true, the braking timer is incremented by 0.04 seconds every frame (0.04 seconds). When the flag changes from true to false, the peak value of the braking timer at that moment is retained as the duration of a braking event, and the braking timer is reset to 0 to continue the next round of timing. For the cumulative lateral displacement, it is divided by the nominal vehicle width of 1.8 meters to obtain the lateral offset multiple. When this multiple changes from less than 1 to greater than 1, the current moment is recorded as the offset start flag, and this flag (with a value of 1) is continuously output until the multiple falls back to less than 1, at which point the offset start flag becomes 0. The updated blink counter, the current value of the brake timer, and the presence or absence (0 or 1) of the offset start marker are concatenated in sequence to form a numerical vector of length 3, which serves as the physical fact state vector at the current moment.

[0033] Then, in the process of converting the obtained behavior-scene joint encoding into a sequence of inference text tokens, each token (e.g., "turn signal not activated," "emergency braking") corresponds to a predicate embedding vector. This predicate embedding vector is obtained through a pre-trained word embedding table of dimension 256, where each predicate corresponds to a fixed 256-dimensional real-valued vector. The current physical fact state vector (of length 3) is mapped to the same 256-dimensional space through a trainable linear projection layer to obtain the state vector. The similarity between the predicate embedding vector and the state vector is calculated using the cosine similarity formula: ;in, The cosine similarity between the predicate embedding vector and the physical fact state vector is represented by... The first term representing the predicate embedding vector One portion, The state vector is represented by the first... The similarity score consists of several components, with the denominator being the product of the magnitudes of the two vectors. The similarity score ranges from -1 to 1, with a higher value indicating greater consistency between the predicate and the current physical fact. If the calculated similarity score is below a preset threshold of 0.5, it is determined that the predicate corresponding to the token has a logical conflict with the event fact in the state machine, meaning that the token violates the temporal physical laws.

[0034] Finally, for tokens identified as conflicting, a negative gradient is generated. This gradient is used to adjust the probability distribution of the token generation layer in reverse. The formula for calculating the negative gradient is: ;in, This represents the negative adjustment amount generated for tokens that violate the laws of temporal physics. This indicates that the maximum value is taken during the operation, with 0.5 as the similarity threshold and 0.2 as the adjustment coefficient. The formula for calculating the negative gradient is as follows: when the similarity is below 0.5, the gradient equals the difference between the threshold and the similarity multiplied by 0.2; when the similarity is greater than or equal to 0.5, the gradient is 0. The calculated gradient value is added to the original logical value corresponding to the token in the token generation layer in negative form, that is, the new logical value equals the original logical value minus the gradient. Subsequently, the token generation layer performs softmax normalization on the new logical values ​​of all tokens to obtain the adjusted probability distribution. Since the logical value of conflicting tokens is reduced, their probability of being selected is reduced accordingly, thereby suppressing the output of tokens that violate the temporal physical laws. No adjustment is made for tokens that do not conflict. The probability distribution adjusted above is used to sample and generate the next token, and this process is repeated until the complete inference text is generated.

[0035] In S5, the adjusted token sequence is reassembled into the final inference text, outputting structured analysis results that include driving behavior classification, risk level, causal explanation, and logical consistency confidence, specifically including: First, the adjusted token sequence is aligned with the timestamps of the original video frames according to the generation time of each token, meaning each token corresponds to a start and end time. These tokens are then concatenated chronologically to form a sequence of natural language phrases. The construction rules for each phrase are as follows: extract the subject (e.g., "this car" or "the car in front"), the action predicate (e.g., "sudden braking," "changing lanes without using turn signals"), and the corresponding time interval (start timestamp to end timestamp, in milliseconds) from the tokens. During concatenation, a space is inserted between the subject and the action predicate, and the word "at" is inserted between the action predicate and the time interval, ultimately resulting in phrases like "This car braked suddenly at 1500 milliseconds to 2200 milliseconds".

[0036] Secondly, the severity coefficient of each behavioral predicate is extracted from the natural language phrase sequence. The specific process consists of three steps. First, each behavioral predicate is mapped to a semantic embedding space: a pre-trained word embedding table is used, containing 256-dimensional real-valued vectors of 200 common driving behavioral predicates. For the input behavioral predicate, its 256-dimensional embedding vector is obtained directly from the table. A hazard level anchoring vector library is pre-established, containing the definitions of five anchoring vectors, corresponding to five levels: "extremely low hazard," "low hazard," "medium hazard," "high hazard," and "extremely high hazard." Each anchoring vector is obtained by collecting a large number of driving behavior samples labeled with hazard levels, averaging the embedding vectors of all behavioral predicates at the same level, and obtaining the anchoring vector for that level. The semantic distance between the behavioral predicate embedding vector and each anchoring vector is calculated using Euclidean distance, i.e., calculating the sum of squares of the differences between corresponding components of two 256-dimensional vectors and then taking the square root. The first step involves selecting the hazard level corresponding to the minimum distance and using its score (1 for extremely low hazard, 2 for low hazard, 3 for medium hazard, 4 for high hazard, and 5 for extremely high hazard) as the baseline severity of the predicate. The second step is to obtain the length of the time interval corresponding to the predicate, calculated by subtracting the start timestamp from the end timestamp, resulting in the interval length (in milliseconds). A short-term hazard threshold of 500 milliseconds is set. The interval length is compared to this threshold: when the interval length is less than 500 milliseconds, a time compression factor is calculated. This factor is obtained by dividing the short-term hazard threshold (500 milliseconds) by the interval length, then squaring the quotient. The compression factor equals the square of the quotient (threshold divided by the length). When the interval length is greater than or equal to 500 milliseconds, the compression factor is set to 1. The third step is to multiply the baseline severity by the time compression factor to obtain the severity coefficient of the predicate. Since a shorter interval length results in a larger compression factor, the severity coefficient is also larger, reflecting the higher severity of short-term hazard behaviors.

[0037] Next, a preset risk matrix is ​​obtained. This risk matrix is ​​a two-dimensional table with 5 rows and 5 columns. The rows correspond to the behavior type (e.g., sudden braking, sudden acceleration, lane changing without signaling, etc., a total of 5 typical behaviors), and the columns correspond to the road scenario in which the behavior occurs (e.g., urban roads, highways, rural roads, tunnels, intersections, a total of 5 scenarios). The weight value of each cell in the matrix is ​​obtained by statistically analyzing a large amount of accident data, and the value ranges from 0 to 1. For each behavior predicate, the corresponding weight is found in the risk matrix according to its behavior type and the current scenario. The severity coefficient of the behavior predicate is multiplied by the weight to obtain the risk contribution value of the behavior. The risk contribution values ​​of all behavior predicates are summed to obtain the risk level of the driving behavior classification. The risk level is a floating-point number between 0 and 100, with a larger value indicating a higher risk.

[0038] Next, the logical consistency confidence score is calculated. The absolute values ​​of the negative gradients generated during each logical conflict detection are summed to obtain the total absolute value of the conflict gradient. This total absolute value of the conflict gradient is then divided by the total length of the token sequence (i.e., the total number of tokens), yielding a value between 0 and 1, which is used as the logical consistency confidence score. A confidence score closer to 1 indicates greater consistency between the reasoning process and physical facts, while a score closer to 0 indicates more conflicts.

[0039] Finally, the calculated risk level and logical consistency confidence score are appended to the end of the natural language phrase sequence in text format: "Risk Level: Numerical, Logical Consistency Confidence Score: Numerical". Simultaneously, based on the logical conflict detection records, when a token is determined to be conflicted, a causal explanation phrase is inserted after the corresponding phrase, such as "Because the turn signal was actually flashing, the assertion of not using the turn signal is invalid". All content is then combined sequentially to output the structured analysis result. This result is a complete text containing the original phrase sequence, the appended risk level, confidence score, and conflict explanation.

[0040] The working principle of this invention is as follows: First, a continuous video frame sequence output by a dashcam and the timestamps corresponding to each frame are collected as raw input data. Then, a large visual model composed of multiple stacked spatiotemporal decomposition blocks is used to perform spatiotemporal feature decomposition on the raw input data to generate a high-dimensional visual feature sequence. Simultaneously, the turn signal brightness pulse timing curve, brake light state duration sequence, and vehicle lateral offset speed trajectory are extracted as atomic event numerical trajectories. Next, the high-dimensional visual feature sequence is mapped to preliminary driving behavior labels and fused with the road scene context embedding extracted from the temporal change saliency map to form a behavior-scene joint encoding. Finally, a differentiable temporal logic state is maintained based on the atomic event numerical trajectories. The system converts behavior-scenario joint encoding into a sequence of inference text tokens. It calculates the cosine similarity between each token to be generated and the physical fact state vector in the state machine. When the similarity is below a threshold, a negative gradient is generated. This negative gradient is used to suppress token outputs that violate temporal physical laws by adjusting the probability distribution of the token generation layer. Finally, the adjusted token sequence is aligned by timestamps and reassembled into a natural language phrase sequence. The severity coefficients of the behavior predicates are extracted from this sequence and accumulated using a preset risk matrix to obtain the risk level. Simultaneously, the logical consistency confidence level is calculated based on the accumulated negative gradient value. The risk level, confidence level, and causal explanation phrases are appended to the end of the phrase sequence, outputting the structured analysis results.

[0041] To further illustrate the beneficial effects achieved by combining the visual large model with atomic event numerical trajectories and differentiable temporal logic state machines in this invention, this embodiment uses multi-scene road videos collected by a dashcam to verify the method of this invention.

[0042] The test videos were captured by a front-view dashcam with a resolution of 1920×1080 and a frame rate of 30 frames per second. Each video segment ranged in length from 8 to 20 seconds. A total of 240 video segments were tested, including 60 segments in normal daylight, 60 segments in low-light nighttime, 60 segments in rainy weather, and 60 segments with slightly damaged or glare lenses. Each video segment was manually annotated with the actual driving behavior, including normal following, deceleration and braking, emergency braking, lane changing with turn signals, lane changing without turn signals, driving over lane lines, lateral deviation, and abnormal braking by the vehicle in front. The start and end times of turn signal flashing, the duration of brake light illumination, and the lateral displacement change of the vehicle's detection frame center point were also annotated as evaluation benchmarks.

[0043] The large-scale visual model used in this embodiment includes a visual encoder, five stacked spatiotemporal decomposition blocks, a behavior-scene joint encoding module, and a text token generation module. The visual encoder first converts the input video frames into high-dimensional visual feature maps, generating a 768-dimensional visual feature vector for each frame. The spatiotemporal decomposition blocks further decompose this visual feature vector into low-frequency scene structure features and high-frequency motion change features. The low-frequency scene structure features are mainly used to identify road boundaries, lane lines, vehicles, pedestrians, and the traffic environment. The high-frequency motion change features are input to three numerical projection heads to output turn signal brightness pulses, brake light duration, and lateral offset speed. Therefore, the large-scale visual model does not rely solely on the visual impression of a single frame to judge driving behavior, but simultaneously obtains the trajectory of physical quantities with temporal continuity, providing a verifiable factual basis for subsequent inference.

[0044] In a specific test segment, the video length was 10 seconds, with a total of 300 frames. In this segment, the vehicle in front started flashing its right turn signal at 2.1 seconds, began changing lanes to the right at 3.4 seconds, completed the lane crossing at 4.2 seconds, and experienced brief braking between 4.6 seconds and 5.3 seconds. The first numerical projection head of the visual large model output the brightness pulse intensity in the region of interest of the right turn signal, detecting obvious rising edges at 2.10 seconds, 2.52 seconds, 2.95 seconds, 3.37 seconds, and 3.79 seconds, with an average pulse peak value of 181.6. The second numerical projection head detected that the brake light was continuously illuminated for 0.72 seconds. The third numerical projection head tracked the center point of the detection frame of the vehicle in front and obtained a cumulative lateral offset of 1.92 meters, exceeding the nominal width of the vehicle of 1.80 meters. Therefore, the state machine identified this segment as "changing lanes to the right after the right turn signal was activated, accompanied by brief braking". The numerical trajectories of the relevant atomic events are shown in Table 1.

[0045] Table 1. Numerical Trajectory Table of Related Atomic Events

[0046]

[0047] Without the constraint of a differentiable sequential logic state machine, a purely visual large model is prone to misinterpreting the flashing right turn signal area as road surface reflection under nighttime glare conditions, generating a token sequence of "the vehicle in front suddenly changed lanes without using its turn signal" during the text inference stage. Using the method of this invention, the state machine writes "right turn signal flashing effectively" into the physical fact state vector based on the continuous rising edges of the right turn signal brightness pulse. When the text token generation module attempts to output the predicate "turn signal not used," the cosine similarity between this predicate embedding and the current physical fact state vector is only 0.28, lower than the preset threshold of 0.50. Therefore, a negative gradient is generated, reducing the output probability of this token. Before adjustment, the probability of the "turn signal not used" token is 0.41, and the probability of the "right turn signal used" token is 0.33; after negative gradient adjustment, the probability of the "turn signal not used" token decreases to 0.12, and the probability of the "right turn signal used" token increases to 0.62. The final output was corrected from the incorrect "The vehicle in front suddenly changed lanes without signaling, which is a high risk" to "The vehicle in front changed lanes to the right after signaling with its right turn signal and then braked briefly, which is a medium risk." This demonstrates that the present invention can use underlying physical signals to correct visual illusions generated by large-scale visual models under conditions such as low light, glare, and windshield wiper obstruction.

[0048] To verify the overall effectiveness, this embodiment compares the method of the present invention with two comparative methods. Comparison method one is a traditional video behavior recognition method using convolutional neural networks plus recurrent neural networks, which only outputs behavior categories. Comparison method two is a large-scale visual model method that does not introduce atomic event numerical trajectories and state machine constraints; it can generate natural language explanations, but the inferred text is not explicitly constrained by turn signal, brake light, and lateral offset trajectories. The statistical results of the three methods on the same test set are shown in Table 2.

[0049] Table 2 Comparison of Statistical Results

[0050]

[0051] As shown in the table above, under normal daylight conditions, the recognition accuracy of the method of the present invention is improved from 91.2% to 93.5% compared to the unconstrained visual large model method; under low light conditions at night, the accuracy is improved from 78.4% to 88.6%; under rainy conditions, the accuracy is improved from 73.1% to 85.9%; and under conditions of lens damage or glare, the accuracy is improved from 69.8% to 83.7%. The improvement is particularly significant in scenarios with obvious visual feature degradation, indicating that the visual large model, by combining with atomic event numerical trajectories, can avoid subjective inferences based solely on blurred image textures, thereby improving reliability in harsh environments. Simultaneously, the logic conflict rate of the method of the present invention decreases from 12.6% in the unconstrained visual large model method to 3.1%, indicating that the differentiable temporal logic state machine can effectively suppress text token outputs inconsistent with physical facts.

[0052] Furthermore, this embodiment statistically analyzed the detection errors of different atomic events to illustrate the role of the parallel numerical projection head in the large-scale visual model. The results are shown in Table 3.

[0053] Table 3 Comparison of Detection Errors

[0054]

[0055] As shown in Table 3, this invention does not simply increase the size of the visual classification network, but rather sets up a spatiotemporal decomposition structure oriented towards driving behavior within the large visual model, enabling it to simultaneously output semantic features and physical quantity trajectories. The turn signal brightness pulse timing curve can reflect whether the turn signal flashes periodically; the brake light state duration sequence can reflect whether the braking behavior actually occurs and its duration; and the lateral deviation speed trajectory can reflect whether the vehicle has completed an actual lane departure.

[0056] After the numerical trajectories of the three types of atomic events mentioned above are input into a differentiable temporal logic state machine, temporal rules such as "signaling first, then lateral movement," "braking first, then deceleration," and "lateral deviation exceeding the width of the vehicle is considered a lane change" can be formed. Since these rules directly participate in the adjustment of the token probability distribution, the explanatory text generated by the visual big model is no longer just a visual semantic description, but a credible reasoning result constrained by physical facts.

[0057] This embodiment also validates the risk level output. Taking three high-risk behaviors—"changing lanes without signaling," "sudden braking," and "driving over lane lines"—as examples, manually labeled risk levels are used as reference values. In adverse weather conditions, the unconstrained visual large-scale model easily misinterprets raindrop reflections as brake light illumination or blurred lane lines as lane crossings, leading to excessively high risk levels. The method of this invention performs secondary verification using brake light duration and cumulative lateral offset. When the brake light illumination duration is less than 0.1 seconds and the cumulative lateral offset does not exceed the vehicle width threshold, probability suppression is applied to high-risk predicates such as "sudden braking" and "changing lanes over lane lines," thereby reducing risk level misjudgments. Statistical results show that the average risk level error of the unconstrained visual large-scale model is 10.8, while the method of this invention reduces it to 4.2, indicating that this invention can improve the stability and interpretability of risk quantification results while maintaining natural language reasoning capabilities.

[0058] In summary, this embodiment demonstrates, through specific video samples, atomic event numerical trajectories, and comparative test data, that the present invention utilizes a large visual model to extract high-dimensional visual semantic features and further introduces quantifiable physical trajectories such as turn signal brightness pulses, brake light durations, and vehicle lateral deviation speeds, thus subjecting the text reasoning process of the large visual model to a differentiable temporal logic state machine. The present invention can significantly reduce erroneous driving behavior assertions caused by visual illusions, improve the accuracy of driving behavior recognition under conditions of nighttime, rain, glare, and lens contamination, and reduce logical conflict rates and risk level errors. This proves that the present invention has the beneficial effects of improving the credibility of driving behavior analysis, enhancing adaptability to harsh environments, and achieving fine-grained risk quantification assessment.

[0059] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A method for analyzing and inferring driving behavior from dashcam videos based on a large visual model, characterized in that, Includes the following steps: S1: Collect the continuous video frame sequence output by the dashcam and the timestamp corresponding to each frame as the raw input data; S2: Use a large visual model to perform spatiotemporal feature decomposition on the original input data to generate a high-dimensional visual feature sequence. Simultaneously extract the turn signal brightness pulse timing curve, brake light state duration sequence, and vehicle lateral offset speed trajectory as atomic event numerical trajectories. S3: Map the high-dimensional visual feature sequence to preliminary driving behavior labels and road scene context embeddings to form behavior-scene joint encoding; S4: Maintain a differentiable temporal logic state machine based on the numerical trajectory of atomic events, and in the process of converting behavior-scene joint encoding into a sequence of inference text tokens, perform logical conflict detection between each token to be generated and the event facts in the state machine, and adjust the probability distribution of the tokens in reverse through gating gradients to suppress token outputs that violate the temporal physical laws. S5: Reassemble the adjusted token sequence into the final inference text, and output a structured analysis result that includes driving behavior classification, risk level, causal explanation and logical consistency confidence.

2. The method for analyzing and inferring driving behavior from dashcam videos based on a large visual model as described in claim 1, characterized in that, The original input data is decomposed into spatiotemporal features using a large visual model, specifically including: A continuous sequence of video frames is input into a large visual model consisting of multiple stacked spatiotemporal decomposition blocks in chronological order. Each spatiotemporal decomposition block first separates the low-frequency components in the temporal domain through a local sliding window attention layer, and then extracts the high-frequency changes in the temporal domain through an adaptive gated recurrent unit. The low-frequency components output by each spatiotemporal decomposition block are accumulated frame by frame to generate a high-dimensional visual feature sequence. At the same time, the high-frequency variation features are fed into three parallel numerical projection heads. The first projection head outputs the brightness pulse intensity value of the region of interest of the turn signal, the second projection head outputs the cumulative value of the binary state duration of the brake light region, and the third projection head outputs the lateral displacement velocity of the center point of the vehicle detection box. The values ​​output from the three projectors are arranged in chronological order to obtain the turn signal brightness pulse timing curve, the brake light state duration sequence, and the vehicle lateral offset velocity trajectory, which are used as atomic event numerical trajectories.

3. The method for analyzing and inferring driving behavior from dashcam videos based on a large visual model as described in claim 2, characterized in that, The separation of time-domain low-frequency components through a local sliding window attention layer specifically includes: The features of consecutive video frames are divided into multiple overlapping windows along the time sequence. Each window contains a fixed number of frames, and adjacent windows share half of the frames. Within each window, an attention weight that decays exponentially with time offset is calculated for each frame, and the features of all frames within the window are weighted and summed to obtain the time-domain low-frequency compressed vector of the window. The temporal low-frequency compression vectors of adjacent windows are linearly interpolated and fused according to the number of overlapping frames to generate a temporal low-frequency component sequence that maintains the same frame rate as the original.

4. The method for analyzing and inferring driving behavior from dashcam videos based on a large visual model as described in claim 1, characterized in that, S3 specifically includes: The high-dimensional visual feature sequence is divided into multiple non-overlapping segments along the time dimension. The temporal maximum entropy pooling value is calculated in each segment. The key frame features with the most information in the non-overlapping segments are extracted. The key frame features are then matched with a predefined behavioral primitive feature library using cosine similarity to output preliminary driving behavior labels. Differential attention is performed on the high-dimensional visual feature sequence and the scene context embedding output from the previous time step to obtain the temporal change saliency map of each pixel position. Then, the spatial relative positional relationship and topological connection order between road participants are extracted from the saliency map to output the road scene context embedding. The feature vectors corresponding to the initial driving behavior labels and the feature vectors embedded in the road scene context are input into a trainable dual-gated fusion unit. The two are then summed element-wise using adaptive weights to generate a behavior-scene joint encoding.

5. The method for analyzing and inferring driving behavior from dashcam videos based on a large visual model as described in claim 4, characterized in that, The process of obtaining a temporal variation saliency map for each pixel location specifically includes: The original difference feature map is obtained by performing a pixel-by-pixel subtraction operation between the feature map of the current frame in the high-dimensional visual feature sequence and the corresponding spatial location feature map embedded in the scene context of the previous time step. For each pixel position in the original difference feature map, take a local neighborhood window centered on itself, calculate the standard deviation of the difference values ​​of all pixels within the window, and then multiply the standard deviation by the difference value of the center pixel to obtain the difference value for local contrast enhancement. The local contrast enhancement difference value at each pixel location is sequentially passed through a three-segment nonlinear mapping. The first segment compresses negative values ​​to zero, the second segment linearly amplifies small values ​​near zero, and the third segment saturates large values ​​to a fixed upper limit, outputting a saliency map of the temporal changes at each pixel location.

6. The method for analyzing and inferring driving behavior from dashcam videos based on a large visual model as described in claim 1, characterized in that, S4 specifically includes: The turn signal brightness pulse timing curve in the atomic event numerical trajectory is divided into rising edge sets according to time windows. At the same time, the brake light state duration sequence is converted into Boolean state flags, and the lateral displacement cumulative amount is obtained by integrating the vehicle lateral offset velocity trajectory. The flash counter, brake timer, and offset start flag stored inside the state machine are updated based on the rising edge set, Boolean state flags, and lateral displacement accumulation to obtain the physical fact state vector at the current moment. When generating each reasoning text token, the predicate embedding corresponding to the reasoning text token is used to calculate the dot product similarity with the physical fact state vector. If the similarity is lower than the threshold, a negative gradient is generated. The negative gradient is backpropagated to the probability distribution of the token generation layer to reduce the probability of selecting tokens that violate the temporal physical laws.

7. The method for analyzing and inferring driving behavior from dashcam videos based on a large visual model as described in claim 6, characterized in that, The process of obtaining the physical fact state vector specifically includes: The time interval between two adjacent rising edges in the rising edge set is compared with a preset flashing period threshold. If the interval is less than the threshold, the flashing counter is incremented; otherwise, the flashing counter is reset to zero. The braking timer is updated based on the duration of the Boolean state flag being true, and the peak value of the timer is retained as the duration of a braking event when the state flag changes from true to false. The lateral offset multiple is obtained by dividing the cumulative lateral displacement by the nominal vehicle width. When the lateral offset multiple changes from less than 1 to greater than 1, the current moment is recorded as the offset start mark, and the offset start mark is continuously output until the multiple falls back to less than 1. The updated blink counter, the current value of the brake timer, and the presence or absence of the offset start marker are combined into a fixed-length numerical vector, which serves as the physical fact state vector at the current moment.

8. The method for analyzing and inferring driving behavior from dashcam videos based on a large visual model according to claim 1, characterized in that, S5 specifically includes: The adjusted token sequence is aligned with the timestamp of the original video frame according to the generation time, and then concatenated into a natural language phrase sequence, where each phrase contains the subject of the behavior, the action predicate, and the time interval. The severity coefficients of behavioral predicates are extracted from natural language phrase sequences, multiplied by the weights of the corresponding behaviors in a preset risk matrix, and summed to obtain the risk level of driving behavior classification. The absolute values ​​of the negative gradients generated by each logical conflict detection are accumulated and then divided by the total length of the token sequence to obtain the logical consistency confidence level between 0 and 1. The risk level and logical consistency confidence level are appended to the end of the natural language phrase sequence, and causal explanatory phrases are inserted based on the conflict detection records to output structured analysis results.

9. The method for analyzing and inferring driving behavior from dashcam videos based on a large visual model as described in claim 8, characterized in that, The calculation process for the severity coefficient is as follows: Each behavioral predicate in the natural language phrase sequence is mapped to the semantic embedding space. The semantic distance between the corresponding embedding and each anchor vector in the predefined danger level anchoring vector library is calculated. The danger level score corresponding to the minimum distance is selected as the baseline severity of the predicate. Obtain the time interval length corresponding to the behavior predicate, compare the time interval length with the short-term danger threshold, and calculate the time compression factor when the length is less than the threshold. The time compression factor is equal to the square of the threshold divided by the length. Multiplying the baseline severity by the time compression factor yields the severity coefficient of the behavioral predicate, where a shorter time interval results in a larger severity coefficient.

Images