Vehicle driving abnormal behavior safety detection method and system based on vision intelligence

Abstract

The invention discloses a vehicle driving abnormal behavior safety detection method and system based on visual intelligence, and the method comprises the steps: firstly carrying out the posture estimation of an original video stream, and converting the motion of a driver into a kinematics feature sequence containing a relative angle and a normalized kinetic parameter; aiming at semantic ambiguity of fine-grained behaviors, a feature importance weight guide algorithm is focused on key micro-action dimensions, and priori knowledge of a cockpit functional area is fused to carry out space pruning and space-time joint cost calculation, so that behaviors with similar forms but different semantics are accurately distinguished. Afterwards, a distance vector sequence obtained through matching is converted into probability distribution, the probability distribution is input into a time sequence grammar model containing an attention mechanism, accurate reasoning of the intention of a driver is achieved through deep decoding and key frame focusing of action evolution logic, and a self-adaptive graded safety alarm is generated according to the accurate reasoning of the intention of the driver. And the accuracy and the real-time performance of abnormal behavior detection in a complex scene are improved.

Description

Technical Field

[0001] This application relates to the fields of intelligent transportation systems and vehicle active safety technology, and more specifically, to a method and system for detecting abnormal driving behavior of vehicles based on visual intelligence. Background Technology

[0002] As autonomous driving technology evolves towards Level 2+ / Level 3, driver monitoring systems have become a core component for ensuring driving safety. Visual intelligence-based technologies are widely used to capture the driver's state in real time, and the detection of obvious fatigue characteristics such as closed eyes and yawning is relatively mature, aiming to prevent traffic accidents caused by distraction or fatigue.

[0003] However, when faced with complex industrial-grade application scenarios, existing technologies face severe challenges in handling highly similar fine-grained interactive behaviors, namely the semantic ambiguity of these behaviors. For example, a driver moving their hand to their mouth could visually be drinking water, smoking, or simply coughing. Existing methods based on convolutional neural networks or simple skeletal point detection often struggle to accurately distinguish such behaviors. This stems from the dilution of feature information and the lack of spatiotemporal modeling: on the one hand, crucial discriminative information is easily drowned out by background noise during deep feature extraction of the entire image or large regions; on the other hand, existing technologies lack a deep understanding of the temporal evolution logic of actions, making it difficult to capture the subtle differences between different behaviors in a dynamic process, and relying solely on instantaneous states cannot accurately define behavioral intentions. Furthermore, even when some existing technologies attempt to introduce dynamic time warping algorithms for action sequence matching, profound inherent flaws remain. Traditional DTW algorithms homogenize all kinematic feature dimensions when calculating costs, exhibiting semantic blindness. This means they fail to automatically focus on the most discriminative feature dimensions based on specific behaviors, causing crucial signals to be masked by noise in secondary dimensions. Simultaneously, existing matching mechanisms often ignore the spatial functional context of the action (e.g., whether the action originated in the cup holder area or the steering wheel area), making it impossible to accurately distinguish semantically disparate behaviors based solely on action morphology. These limitations collectively result in a persistently high false alarm rate for existing DMS systems in complex scenarios, failing to meet the demands of high-precision security detection.

[0004] Therefore, an optimized safety detection scheme for abnormal vehicle driving behavior is desired. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a visual intelligence-based method and system for detecting abnormal vehicle driving behavior.

[0006] According to one aspect of this application, a vehicle driving abnormal behavior safety detection method based on visual intelligence is provided, comprising: Acquire the raw video stream captured by the vehicle-mounted DMS camera; Pose sequence extraction and kinematic feature transformation are performed on the original video stream to obtain a kinematic feature sequence; Dynamic matching is performed between the kinematic feature sequence and each primitive template in the action primitive dictionary to obtain the distance vector; Temporal syntax recognition is performed on the distance vector sequence over the past N time steps to obtain behavior probabilities; Adaptive risk assessment and graded alarms are performed on the probability of behavior to obtain alarm instructions.

[0007] According to another aspect of this application, a vehicle driving abnormal behavior safety detection system based on visual intelligence is provided, comprising: The raw video stream acquisition module is used to acquire the raw video stream captured by the vehicle-mounted DMS camera; The pose sequence extraction and kinematic feature transformation module is used to extract pose sequences and transform kinematic features from the original video stream to obtain kinematic feature sequences. The dynamic matching module is used to dynamically match the kinematic feature sequence and each primitive template in the action primitive dictionary to obtain the distance vector; The temporal syntax recognition module is used to perform temporal syntax recognition on the distance vector sequence of the past N time steps to obtain the behavior probability; The adaptive risk assessment and graded alarm module is used to perform adaptive risk assessment and graded alarms on the probability of behavior in order to obtain alarm instructions.

[0008] Compared with existing technologies, this invention proposes a visual intelligence-based method for detecting abnormal driving behavior in vehicles. It transforms in-vehicle video streams into high-dimensional kinematic feature sequences and uses a dictionary of action primitives containing rich prior knowledge for dynamic matching. During the matching phase, a template-specific feature importance weight vector is introduced to automatically focus on key discriminative feature dimensions, overcoming recognition interference caused by feature homogenization. Simultaneously, spatial prior pruning and spatiotemporal joint cost calculation are performed using a cockpit functional area map, eliminating behavioral ambiguity due to similar action forms but different semantics through physical spatial constraints. Subsequently, the distance vector generated by matching is transformed into a probability sequence, and a temporal grammar model incorporating a self-attention mechanism is used to deeply analyze the evolution logic and key temporal patterns between action primitives, accurately identifying fine-grained driving behavior intentions such as drinking water and smoking. Finally, an adaptive risk assessment strategy is combined to generate tiered alarm commands, achieving driving safety detection with high robustness and context awareness. Attached Figure Description

[0009] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 This is a flowchart of a vehicle driving abnormal behavior safety detection method based on visual intelligence according to an embodiment of this application; Figure 2 This is a schematic diagram of data flow for a visual intelligence-based vehicle driving abnormal behavior safety detection method according to an embodiment of this application; Figure 3 The flowchart shows the dynamic matching of action primitives based on a weighted cost matrix in the vehicle driving abnormal behavior safety detection method based on visual intelligence according to the embodiments of this application. Figure 4 This is a flowchart of the template-pre-screening-based dynamic matching of action primitives in the visual intelligence-based vehicle driving abnormal behavior safety detection method according to an embodiment of this application; Figure 5 The flowchart illustrates the process of performing temporal syntax recognition on a distance vector sequence over the past N time steps to obtain the behavior probability in a vehicle driving abnormal behavior safety detection method based on visual intelligence, according to an embodiment of this application. Figure 6 This is a flowchart illustrating the process of using self-attention focusing on key action frames of a hidden state sequence to obtain a global context vector in a vehicle driving abnormal behavior safety detection method based on visual intelligence, according to an embodiment of this application. Figure 7 This is a block diagram of a visual intelligence-based vehicle driving abnormal behavior safety detection system according to an embodiment of this application. Detailed Implementation

[0011] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0012] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0013] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0014] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0015] Existing vision-based driver monitoring technologies often suffer from severe semantic confusion and false alarms when faced with fine-grained interactive behaviors that are highly similar in appearance, such as drinking water, smoking, and coughing with a hand covering the mouth. This is because key micro-motion features are often obscured by background noise and there is a lack of perception of the spatial region where the action occurs. Therefore, this application proposes a visual intelligence-based method for detecting abnormal driving behavior in vehicles. This method generally adopts a progressive processing logic from micro-kinematic feature extraction to macro-temporal grammar reasoning to solve the challenge of intent understanding in complex scenarios. Specifically, the original video stream is first transformed into a kinematic feature sequence robust to translation and scaling. Then, in the action sequence matching stage, blind global traversal is abandoned, and instead, spatial prior pruning is performed using a cockpit functional area map. Feature importance weights for specific behavioral templates are loaded, and the spatiotemporal joint cost function dynamically focuses on the most discriminative key dimensions (such as subtle interactions between fingers and mouth), thereby obtaining distance vectors rich in semantic information. Finally, a recurrent neural network with a self-attention mechanism is used to perform temporal grammar modeling on these distance vector sequences to identify the implicit logical combination rules between action primitives, thereby accurately inferring the driver's true intention and triggering corresponding adaptive safety warnings.

[0016] Figure 1 This is a flowchart of a visual intelligence-based vehicle driving abnormal behavior safety detection method according to an embodiment of this application. Figure 2 This is a schematic diagram of the data flow of a vision-based intelligent vehicle driving abnormal behavior safety detection method according to an embodiment of this application. Figure 1 and Figure 2As shown, the vehicle driving abnormal behavior safety detection method based on visual intelligence according to an embodiment of this application includes the following steps: S100, acquiring the original video stream captured by the vehicle-mounted DMS camera; S200, extracting the pose sequence and transforming the kinematic features of the original video stream to obtain a kinematic feature sequence; S300, dynamically matching the kinematic feature sequence with each primitive template in the action primitive dictionary to obtain a distance vector; S400, performing temporal grammar recognition on the distance vector sequence of the past N time steps to obtain the behavior probability; S500, performing adaptive risk assessment and graded alarm on the behavior probability to obtain an alarm command.

[0017] Specifically, in step S100, the raw video stream captured by the vehicle-mounted DMS camera is acquired. It is understood that due to the drastic changes in lighting conditions within the cockpit during vehicle operation, and the instantaneous and highly dynamic nature of abnormal behaviors such as smoking and drinking by the driver, a single static image or low-frequency sampling cannot capture the spatiotemporal details required for resolving fine-grained movements. Therefore, in the technical solution of this application, the vehicle-mounted DMS camera is used to continuously acquire raw video streams containing dynamic images of the driver's upper body, thereby constructing a data channel for continuous, uninterrupted observation of the driver's state. This provides high-fidelity and time-series-aligned source data for subsequent pose estimation and kinematic analysis, ensuring that the algorithm model does not miss crucial behavioral features such as finger micro-movements or mouth opening and closing due to data gaps.

[0018] More specifically, in one particular example of this application, this step first involves optically covering the physical space where the driver is located using an image acquisition device fixedly installed at a specific location within the vehicle's driver's compartment. This installation location is precisely set within a field of view capable of completely capturing the driver's head posture, facial expressions, and the range of their hands operating on the steering wheel and center console, such as the inside of the A-pillar or above the dashboard. The photoelectric sensor inside the image acquisition device converts the captured optical images into digital signals in real time and generates a continuous video frame sequence according to a preset high frame rate and high resolution standard. This video frame sequence is synchronously transmitted to the input buffer of the onboard computing platform via a high-speed vehicle communication bus or a dedicated video interface. During this process, the image acquisition device is configured to activate near-infrared sensing mode and work in conjunction with an infrared supplementary light lamp, thereby eliminating the impact of external ambient light fluctuations on image quality. This ensures that a clear, low-noise grayscale video stream can still be output even in complex lighting scenarios such as nighttime, tunnels, or backlighting, achieving all-weather visual data acquisition.

[0019] Specifically, in step S200, the original video stream undergoes pose sequence extraction and kinematic feature transformation to obtain a kinematic feature sequence. It is understood that background clutter, lighting changes, and non-behavioral elements such as driver clothing texture in the vehicle driving environment introduce a large amount of redundant information and interference noise into the original pixel-level data. Directly performing fine-grained behavior analysis based on pixel features easily leads to the algorithm wasting computational resources on irrelevant visual details, making it difficult to accurately capture the essential differences in limb geometry and dynamic evolution of behaviors such as smoking or drinking. Therefore, in the technical solution of this application, the original video stream is further subjected to pose sequence extraction and kinematic feature transformation to obtain a kinematic feature sequence. The specific execution process involves using human pose estimation technology to locate the pixel coordinates of key points on the driver's upper body from consecutive video frames, performing temporal smoothing filtering to eliminate detection jitter, and calculating the relative angles between key points and the velocity and acceleration vectors normalized to torso length. This approach reduces the dimensionality of high-dimensional visual images to a purely geometric motion representation that is highly invariant to translation, scaling, and appearance changes. It forces subsequent processing logic to focus on the spatial configuration changes and movement rate rhythm of the driver's limbs. This effectively isolates the interference of environmental and appearance factors, providing a set of high signal-to-noise ratio, dynamically rich feature descriptions for distinguishing complex interactive behaviors that are similar in appearance but have vastly different motion logics. This enhances the robustness of abnormal behavior detection in dynamic and complex scenarios.

[0020] More specifically, in a specific example of this application, performing pose sequence extraction and kinematic feature transformation on the original video stream to obtain a kinematic feature sequence includes: extracting pose sequence from the original video stream to obtain a smooth pose data sequence; and performing kinematic feature transformation on the smooth pose data sequence to obtain a kinematic feature sequence.

[0021] Accordingly, pose sequence extraction is performed on the original video stream to obtain a smooth pose data sequence. It is understandable that due to the inevitable vibrations generated during vehicle operation and the complex and variable lighting environment inside the driver's cabin, the human body key point coordinates detected directly based on a single frame image often contain unnatural jitter or abrupt noise. This high-frequency noise severely interferes with the accurate calculation of minute movement amplitudes. Therefore, in the technical solution of this application, pose sequence extraction is further performed on the original video stream to obtain a smooth pose data sequence, which is used to construct a geometric skeleton model of the driver's body parts, and a temporal filtering mechanism is used to correct detection biases. This outputs a set of key point coordinates that are temporally continuous, have smooth trajectories, and conform to the laws of human kinematics, laying a solid data foundation for subsequent extraction of high signal-to-noise ratio kinematic features.

[0022] More specifically, in a concrete example of this application, the processing flow first preprocesses each frame of image decoded from the original video stream, including scaling and normalization, to adapt it to the input layer requirements of a high-precision two-dimensional human pose estimation network. Then, the preprocessed image is input into the pose estimation network, which uses deep convolutional structures to extract visual features of the driver's upper body and outputs a set of heatmaps corresponding to key areas such as the wrist, elbow, shoulder, and mouth. Next, the heatmaps are decoded, and the original pixel coordinates and confidence levels of each key point are determined by finding the response peak locations. Finally, a Kalman filter is used to perform temporal smoothing on the obtained sequence of original key point coordinates. This filter predicts the current position based on the state of the previous moment and performs weighted corrections based on the current observations, effectively filtering out random jitter caused by detection instability and generating the final smooth pose data sequence.

[0023] Accordingly, kinematic feature transformation is performed on the smooth posture data sequence to obtain a kinematic feature sequence. It is understandable that, since the original pixel coordinate data heavily depends on the physical distance between the camera and the driver, as well as the driver's own body size, simple absolute position information cannot extract the essence of the action independent of the imaging perspective, and it is difficult to characterize the fine-grained dynamic features implied in the temporal rate of change, such as the rapid shaking when smoking or the gentle tilting of the head when drinking water, using static coordinates. Therefore, in the technical solution of this application, kinematic feature transformation is further performed on the smooth posture data sequence to obtain a kinematic feature sequence, thereby elevating the low-level geometric position information to a semantic feature space that includes relative structural relationships and higher-order dynamic attributes. This effectively eliminates recognition bias caused by different shooting angles or occupant body sizes, and accurately captures the specific fingerprints of different abnormal behaviors in terms of movement rhythm and force using higher-order quantities such as velocity and acceleration.

[0024] More specifically, in a concrete example of this application, the conversion process first selects key points such as the shoulder, elbow, and wrist based on the topological structure of the human skeleton to construct vector connections. Relative angle features are obtained by calculating the angles between adjacent vectors. For example, the elbow bending angle is obtained by taking the dot product of the vector connecting the elbow and wrist and the vector connecting the shoulder and elbow, thus representing the folding state of the limb. Simultaneously, to address scale uncertainty, the computational logic obtains the Euclidean distance between the left and right shoulder key points in real time as the baseline torso length. The displacement difference between each key point in adjacent frames is divided by this baseline torso length to obtain a normalized velocity vector. This metric objectively reflects the speed of the movement regardless of image size. Based on this, a temporal difference operation is performed on the normalized velocity vector to obtain normalized acceleration, thereby quantifying the force changes and explosive characteristics during the movement. Finally, the relative angles, normalized velocities, and normalized accelerations calculated for each frame are concatenated into a high-dimensional feature vector in a predetermined order and arranged continuously according to timestamps to form a kinematic feature sequence for subsequent matching analysis. Meanwhile, the original pixel coordinate sequence of the smoothed key points corresponding to the frame is preserved and output synchronously as auxiliary data input for subsequent spatial semantic analysis.

[0025] Specifically, in step S300, the kinematic feature sequence and each primitive template in the action primitive dictionary are dynamically matched to obtain a distance vector. It is understood that when drivers perform abnormal behaviors such as drinking water, smoking, or making phone calls, the speed, rhythm, and amplitude of their movements often exhibit significant individual differences and randomness, making it difficult to accurately capture the essence of the behavior based solely on feature comparisons of fixed durations. Therefore, in the technical solution of this application, the kinematic feature sequence and each primitive template in the action primitive dictionary are further dynamically matched to obtain a distance vector. The real-time extracted kinematic feature sequence is elastically aligned and similarity measured with the standard templates in the preset action primitive dictionary to eliminate the nonlinear distortion effect in the time dimension. This effectively accommodates changes in the action rate of different drivers or the same driver in different situations, generating a distance vector that robustly represents the similarity between the current action and various dangerous behaviors, thus providing a reliable quantitative basis for subsequent accurate identification. To balance the accuracy and real-time performance of the matching, this application provides two optimized matching paths in different embodiments: feature weighting and spatiotemporal pre-screening, which are described in detail below.

[0026] In one specific embodiment, to improve matching accuracy, a template-guided, dynamically weighted feature importance DTW matching mechanism can be constructed to assign different importances to different dimensions of kinematic features (such as relative angle, normalized velocity, and normalized acceleration). That is, before matching begins, the action primitive template to be matched is loaded with its own specific feature importance weights, injecting prior domain knowledge about which features are most critical for recognizing this specific action into the subsequent cost calculation, thereby overcoming the semantic blindness of traditional methods. Understandably, standard dynamic time warping algorithms assume, by default, that each physical dimension in the kinematic feature vector contributes equally to the semantic discrimination of the behavior when calculating the similarity between two action sequences. This results in homogenized distance measurements for heterogeneous features such as relative angle, normalized velocity, and normalized acceleration. However, this completely deviates from the physical essence of driving behavior. Different abnormal behaviors are often physically defined by only a few specific types of features. For example, nodding while dozing off is mainly defined by abrupt changes in the normalized acceleration features of key points in the head, and is not sensitive to the relative angles of the limbs. Conversely, smoking is highly dependent on the relative angle features of the hands and mouth, and has a high tolerance for changes in action speed.

[0027] Therefore, in the technical solution of this application, a template-guided, dynamically weighted DTW matching mechanism is constructed to assign different importances to different dimensions of kinematic features. This establishes a focus allocation system driven by behavioral semantics in reverse computational logic. Utilizing pre-built expert prior knowledge in the primitive templates, the weight coefficients of each feature channel are adjusted in real time during the matching process. This allows the distance calculation process to adaptively amplify the differences in decisive features and mask fluctuations in irrelevant features. This effectively solves the problem of weight imbalance in the semantic expression of multi-dimensional motion features, ensuring the algorithm remains robust when facing speed noise caused by vehicle vibration or random angular deviations of non-critical limbs. It performs precise similarity measurement only on the core physical dimensions that truly represent the specific abnormal behavior, thereby improving the signal-to-noise ratio and semantic accuracy of behavior recognition under complex driving conditions. Figure 3 As shown, dynamic matching is performed on the kinematic feature sequence and each primitive template in the action primitive dictionary to obtain the distance vector, including: S311, loading the feature importance weight vector of the primitive template; S312, calculating the matching weighted cost matrix between the kinematic feature sequence and the primitive template based on the feature importance weight vector; S313, performing cumulative path optimization on the matching weighted cost matrix to obtain the final matching distance between the primitive template and the kinematic feature sequence.

[0028] In step S311, the feature importance weight vector of the primitive template is loaded. It is understandable that standard distance metric algorithms, when calculating the similarity of action sequences, treat all dimensions within the kinematic feature vector with indiscriminate homogenization, failing to reflect the inherent semantic hierarchy and discriminative differences between different feature dimensions in distinguishing specific driving behaviors. This results in noise from secondary features, such as shoulder swaying caused by vehicle bumps, easily drowning out the micro-motion signals of the hands that truly define the nature of the behavior. Therefore, in the technical solution of this application, the feature importance weight vector of the primitive template is further loaded to inject exclusive prior discriminative knowledge into each action primitive template to be matched, clearly indicating which key limb movements the algorithm should focus on during the matching process. This overcomes the semantic blindness of traditional matching methods, ensuring that subsequent cost calculations can automatically amplify the differences in highly discriminative feature dimensions while effectively suppressing environmental noise interference from non-critical dimensions, thereby achieving precise focusing on the driver's fine-grained intentions at the feature level.

[0029] More specifically, in a concrete example of this application, this step relies on a pre-built action primitive repository containing rich semantic information. During execution, when selecting the first action primitive from the pre-stored action primitive dictionary... Template At the same time, the feature importance weight vector paired with it will be retrieved from storage. The weight vector In the offline phase, a large amount of sample data of this type of behavior is collected, and the variance statistics of each feature dimension in the sample set are calculated. Dimensions with smaller variance (i.e., stable feature performance) are assigned higher weights, while dimensions with larger variance (i.e., high randomness) are assigned lower weights. This is then normalized to generate the weight vector. This weight vector has the same dimension as the kinematic feature vector, and each value within it corresponds to a specific kinematic feature channel (such as relative angle, normalized velocity, etc.). It acts like a filter or lens tailored to a specific template, explicitly defining the importance of the corresponding feature dimension through its numerical value. For example, when matching a smoking template, this weight vector is pre-set to make the system pay more attention to feature dimensions related to slight wrist rotation and the distance between the mouth and nose, assigning them higher weight values, while assigning lower weight values ​​to torso background swaying or non-interactive hand displacement features that are less coupled to the smoking action. The feature importance weight vector... This provides precise, context-aware guidance for subsequent matching calculations, in which That is, template The corresponding weight vector of the first The importance weight value of each dimension will directly participate in the subsequent weighted cost calculation, thereby amplifying the key micro-motion signals and suppressing environmental noise at the mathematical level.

[0030] In step S312, a matching weighted cost matrix between the kinematic feature sequence and the primitive template is calculated based on the feature importance weight vector. It is understandable that, because all dimensions of the kinematic feature vector are assigned the same weight in the standard Euclidean distance calculation, under complex conditions such as vehicle bumps, secondary feature noise generated by shoulder or torso swaying often numerically masks the differences in hand micro-movements that truly define fine-grained behavior, making it impossible for the system to effectively distinguish behaviors with similar motion patterns but vastly different safety risks. Therefore, in the technical solution of this application, a matching weighted cost matrix between the kinematic feature sequence and the primitive template is further calculated based on the feature importance weight vector. This constructs a non-uniform cost space that dynamically reflects the semantic concerns of specific behaviors; that is, in this space, differences in key feature dimensions are amplified, while noise in irrelevant dimensions is suppressed. In this way, simple geometric distance calculation can be transformed into a semantic similarity metric with scene context awareness, ensuring that the final generated cost matrix is ​​like a pre-carved semantic terrain map, providing a high signal-to-noise ratio decision basis for subsequent accurate capture of actions such as the slight finger movement when smoking or the swallowing action when drinking water.

[0031] More specifically, in a concrete example of this application, the calculation process first creates a two-dimensional matrix container with dimensions equal to the length of the real-time action sequence multiplied by the length of the primitive template sequence to store the local matching cost. Then, a double-loop traversal logic is initiated, targeting the first... The first frame in the selected primitive template sequence Frames, utilizing pre-loaded feature importance weight vectors As a moderating factor, the sum of weighted squared errors of both factors across each feature dimension is calculated one by one. This calculation logic is rigorously defined by the following formula:

[0032] in, The value located at the th position in the constructed matching weighted cost matrix represents the position of the th position. line, number The cost element of the column quantifies the degree of semantic mismatch between the action states at two time points. It represents the total number of dimensions of the kinematic feature vectors, encompassing all feature channels from macroscopic limb position to microscopic joint angles. As an index variable, it is used to iterate through each dimension of the feature vector. That is, the first matching number. The weight vector specific to each primitive template is the first one. Each dimension has an importance weight value; the larger the value, the more critical that dimension is to determining the current behavior. Represents the first action in the real-time action sequence The first frame feature vector Measurement values ​​in each dimension, and Then it represents the first In the primitive template sequence, the th The first frame feature vector Standard values ​​for each dimension. Under this calculation mechanism, for example when matching a smoking template, the weight vector... It assigns extremely high weights to the wrist rotation angle and the distance between the index finger and the mouth, while assigning extremely low weights to the vertical head displacement dimension. At this point, even if the driver's head experiences significant jolting due to uneven road surfaces (i.e.,...), and (where the difference in head displacement dimension is large), due to the corresponding Minimal, its effect on the final The contribution remains negligible; conversely, if the driver's hand does not make the rotational motion characteristic of smoking (i.e., a large difference in wrist rotation dimension), due to the corresponding It's very large, which will lead to The cost increases dramatically, creating a high barrier in the cost matrix and clearly indicating a mismatch between actions. This mechanism, through a spotlight effect at the computational level, forces the model to ignore vehicle vibration noise and focus more on the subtle physical interactions that determine the nature of behavior.

[0033] In step S313, cumulative path optimization is performed on the matching weighted cost matrix to obtain the final matching distance between the primitive template and the kinematic feature sequence. It is understandable that the speed at which a driver performs specific actions is highly random and nonlinear. For example, completing a drinking action might only require two seconds of rapid head tilting or five seconds of slow swallowing. This elastic scaling in the time dimension often results in the real-time action sequence and the primitive template not being directly aligned temporally. If only rigid linear matching is relied upon, even if the spatial forms of the two are highly consistent, a large cumulative error will occur due to temporal misalignment, leading to recognition failure. Therefore, in the technical solution of this application, cumulative path optimization is further performed on the matching weighted cost matrix to obtain the final matching distance between the primitive template and the kinematic feature sequence. This allows a dynamic programming algorithm to find a nonlinear alignment path with the minimum total cost within the semantic terrain map constructed by the weighted cost. This effectively avoids interference caused by differences in action duration and speed, transforming temporal uncertainty into an optimization problem of path search, ensuring that the final output matching distance simply reflects the similarity of actions at the behavioral semantic level.

[0034] More specifically, in a particular example of this application, this step is based on the matching weighted cost matrix generated by the preceding steps. Perform dynamic programming recursive calculations. The calculation process follows the boundary condition initialization and state transition equations, starting from the initial point of the matrix and calculating the minimum cumulative cost to reach the current position for each node. This cumulative path optimization process is strictly executed according to the following formula:

[0035] in, Represents the th element in the cumulative cost matrix line, number The cumulative value of the column represents the real-time action sequence up to the [number]th [action]. Frames and primitive templates up to the [number]th [number]. The total cost of achieving optimal timing alignment for frames. It is the local weighted matching cost at the current position, that is, the semantic distance calculated in the previous step that takes into account the importance of features. The function selects the node with the minimum cumulative cost from three neighboring predecessor nodes: the one to the left (representing the stagnation of the primitive template and the extension of the real-time sequence), the one above (representing the stagnation of the real-time sequence and the extension of the primitive template), and the one to the upper left (representing the synchronous advancement of both). Using this formula, the algorithm constructs a valley path through the low-potential region in the cost matrix. Taking a driver drinking water scenario as an example, when the driver completes the drinking action at an extremely fast speed, the real-time sequence is shorter compared to the template at standard speed. The optimization mechanism in the above formula will automatically tend to choose an upward or leaping path, i.e., quickly crossing the template axis while spending less time on the real-time axis; conversely, if the driver's action is slow, the path will tend to advance along the left. Regardless of the speed of the action, as long as the interaction characteristics of the hand and mouth during the action (determined by…) are considered… This aligns with the high-weighted dimension guarantee in the algorithm, allowing the optimization path to avoid high-cost regions and ultimately calculate the normalized cumulative distance (i.e., ...). The final value (divided by the path length) will remain in the low position, thus correctly identifying it as drinking behavior without being misled by time differences.

[0036] In another specific embodiment, as an optional or additional optimization, to reduce the computational overhead of performing one-to-one matching on a large-scale primitive dictionary and to reduce mismatches caused by context-independent templates, a dynamic matching method based on functional area prior pruning and spatiotemporal joint cost is adopted. This method introduces spatial dimension contextual information to fundamentally reconstruct the original matching process. It is understandable that existing matching mechanisms based on dynamic time warping suffer from severe semantic blind spots and computational bottlenecks when processing driving behavior. Their cost function, which relies solely on kinematic morphological similarity, cannot perceive the physical spatial context of the action, making it difficult for the system to distinguish between actions with similar trajectories but vastly different safety implications, such as taking water from a cup holder and drinking versus covering one's mouth and yawning at the steering wheel. Furthermore, the brute-force matching method overwhelms the onboard computing unit.

[0037] Therefore, in the technical solution of this application, a dynamic matching method based on functional area prior pruning and spatiotemporal joint cost is adopted. This method introduces spatial contextual information to fundamentally reconstruct the original matching process, establishing a new matching paradigm that prioritizes spatial filtering followed by spatiotemporal joint measurement. It first eliminates spatially unreasonable candidate primitives using the inherent functional area layout logic within the cockpit, and then forcibly adds spatial trajectory conformity verification constraints during the fine matching of the remaining templates. This upgrades simple geometric shape comparison to intent understanding with environmental semantic awareness, ensuring that the final output matching result not only matches the actions but also the positions. This reduces system computational overhead while eliminating action ambiguity caused by missing spatial context, achieving precise targeting of the driver's complex interactive intents. Figure 4 As shown, the process of dynamically matching the kinematic feature sequence and each primitive template in the action primitive dictionary to obtain the distance vector also includes: S321, performing template pre-screening on the action primitive dictionary based on the kinematic feature sequence, functional area map, and real-time keypoint original pixel coordinate sequence to obtain a pre-screened primitive dictionary; S322, performing dynamic matching on the kinematic feature sequence, real-time keypoint original pixel coordinate sequence, and each primitive template in the pre-screened primitive dictionary to obtain the spatiotemporal DTW distance set; S323, vectorizing the spatiotemporal DTW distance set to obtain the distance vector.

[0038] In step S321, based on the kinematic feature sequence, functional area map, and real-time keypoint raw pixel coordinate sequence, template pre-screening is performed on the action primitive dictionary to obtain a pre-screened primitive dictionary. It is understandable that, because this mechanism requires traversing and matching all K templates in the action primitive dictionary for each real-time input action sequence when performing driving behavior sequence matching based on dynamic time warping, its overall time complexity is as high as... This becomes a performance bottleneck in automotive embedded systems with stringent real-time requirements as the dictionary size increases. Furthermore, the original DTW cost function only measures the morphological similarity of action sequences at the kinematic feature level, failing to perceive where the actions occur. This makes it impossible to effectively distinguish between semantically disparate but kinematically highly similar behaviors. Therefore, in this application's technical solution, a pre-filtered primitive dictionary is obtained by pre-selecting templates based on kinematic feature sequences, functional area maps, and real-time keypoint original pixel coordinate sequences. This utilizes low-cost spatial prior knowledge to effectively prune the search space, simulating an expert judgment logic that preemptively eliminates many spatially unreasonable possibilities based on the location of the action. This significantly reduces the number of templates that need to be processed in subsequent steps while ensuring no potentially correct matching templates are lost, thereby reducing the overall computational load and ensuring real-time response capabilities.

[0039] More specifically, in a concrete example of this application, this step first invokes the localization logic to extract the hand position coordinates of the action's starting frame based on the real-time keypoint raw pixel coordinate sequence synchronized with the kinematic feature sequence. Then, these starting frame coordinates are mapped onto a pre-defined cockpit functional area geometry map, which divides the cockpit space into semantically defined physical areas such as the steering wheel area, gear shift area, cup holder area, and window area. Next, the specific functional area ID initiating the current action is determined through spatial inclusion relation operations; this process is formally expressed as a formula:

[0040] in, This represents the identifier of the starting functional area associated with the calculated real-time motion trajectory. It is a positioning function that performs geometric matching. This represents the first data point in the coordinate sequence, i.e., the pixel coordinates of the starting frame of the action. This represents a pre-defined geometric map of the cockpit functional areas. After determining the starting region, a conditional filtering operation is performed on the action primitive dictionary, retaining only those templates marked as originating from that specific region during the offline construction phase. This filtering logic uses the formula:

[0041] in, It is a candidate primitive dictionary that contains only highly relevant templates after pre-screening. Represents the original dictionary The first in A primitive template, This represents the logical AND operation. It is to obtain the template A function that presets the initial ribbon attribute value; this attribute value must be consistent with the real-time calculated value. Strict equality is applied. For example, in a real-world scenario, if the detected driver's hand movement trajectory originates from the seatbelt anchor point area near the B-pillar and moves towards the chest, the localization function will classify the starting area as the seatbelt functional area. Based on the above filtering logic, the pre-screening process will automatically eliminate templates such as drinking water originating from the cup holder area or yawning originating from the steering wheel area, retaining only compliant templates such as pulling the seatbelt or adjusting the seat back. This spatial constraint-based filtering mechanism ensures that even if the action of pulling the seatbelt is visually similar to the action of raising a cup at a specific angle, the algorithm can fundamentally eliminate the possibility of misidentifying it as drinking water behavior before entering the time-consuming matching phase, thereby significantly improving the accuracy and efficiency of recognition.

[0042] In step S322, the kinematic feature sequence, the original pixel coordinate sequence of real-time keypoints, and each primitive template in the pre-screened primitive dictionary are dynamically matched to obtain the spatiotemporal DTW distance set. It is understandable that although the preceding pre-screening steps eliminate templates with obviously incorrect starting positions, the remaining candidate actions are often still highly similar in terms of pure kinematic morphology. For example, waving a hand in mid-air and waving a hand near the mouth have almost identical rates of change in joint angles. Standard dynamic time warping algorithms, when calculating the similarity between two sequences, simply accumulate the differences in motion feature values, lacking the ability to continuously verify whether the real-time trajectory remains within the correct physical region during the action. This results in behaviors with different positions but identical actions still being judged as a match. Therefore, in the technical solution of this application, the kinematic feature sequence, the original pixel coordinate sequence of real-time keypoints, and each primitive template in the pre-screened primitive dictionary are further dynamically matched to obtain the spatiotemporal DTW distance set. This forces the introduction of spatial position constraints in the cost calculation of each frame, upgrading the simple action morphology comparison to a dual verification of action morphology and spatial trajectory. In this way, the calculated distance value can not only reflect whether the action is performed correctly, but also whether the path of the action is correct, thus effectively distinguishing ambiguous behaviors that have similar starting points but whose intermediate trajectories deviate from the preset functional area.

[0043] More specifically, in a concrete example of this application, the process first iterates through each remaining candidate template in the pre-screened primitive dictionary. For each template, the algorithm constructs a dynamic programming matrix and calculates the alignment cost between the real-time sequence and the template sequence frame by frame. In this process, instead of solely relying on differences in motion features, a weighted fusion calculation logic is executed, which is mathematically expressed as the formula:

[0044] In this formula, The first representing the real-time sequence The first frame and template sequence The spatiotemporal joint cost between frames, It is a hyperparameter between 0 and 1, used to balance the importance of kinematic cost and spatial cost. It partially inherits the original kinematic morphology matching and measures real-time motion characteristics. With template motion characteristics The Euclidean distance between them, and This is the newly added spatial trajectory conformity cost component, which receives real-time keypoint coordinates. Coordinates corresponding to the template As input, the spatial cost function is designed as a region penalty logic. When the spatial positions of the two do not conform to the expected functional area logic, the function will return a high penalty value. Taking a driver smoking scenario as an example, the smoking action requires the hand to hover and move back and forth briefly in the mouth area. If, in real-time, the hand exhibits a similar reciprocating frequency, but its real-time coordinates remain hovering in the chin or neck area (possibly a scratching motion), then the formula... The item will continuously output high costs due to regional mismatch, resulting in a larger cumulative DTW distance value, thus determining that the behavior does not belong to smoking, avoiding misjudgment caused by simply similar action frequencies.

[0045] In step S323, the spatiotemporal DTW distance set is vectorized to obtain a distance vector. It is understandable that, because the spatiotemporal DTW distance values ​​calculated for each primitive template in the preceding steps exist in isolation as discrete scalars, lacking a unified linear algebraic structure, they cannot directly adapt to the stringent format requirements of the tensor-based temporal syntax recognition module based on deep neural networks. Furthermore, simple discrete numerical values ​​are difficult to represent the relative affinity distribution of the current action in the entire candidate primitive space through parallel matrix operations. Therefore, in the technical solution of this application, the spatiotemporal DTW distance set is further vectorized to obtain a distance vector, thereby encapsulating these loosely defined distance metrics for different templates into mathematical vector objects with strict index correspondence and fixed dimensions. This enables standardized interface connection from the signal output of the geometric matching layer to the data input of the semantic reasoning layer, providing the necessary structured data foundation for the subsequent model to simultaneously evaluate the matching degree between the current action and all potential intentions through efficient matrix multiplication.

[0046] More specifically, in a concrete example of this application, the vectorization process performs a serialized data normalization operation. This operation strictly follows the index order of candidate templates in the pre-selected primitive dictionary, traversing the computation result corresponding to each candidate template. Specifically, the previous step is applied to each candidate template... The calculated spatiotemporal DTW distance value, which integrates kinematics and spatial semantics. The templates are extracted sequentially and concatenated according to their inherent order in the dictionary, thus assembling a one-dimensional feature vector. This process and the final generated vector are mathematically formalized as the formula:

[0047]

[0048] in, This represents the final spatiotemporal refined distance vector, which is essentially a semantic distance spectrum containing the current real-time action sequence relative to all candidate primitive templates. It is aimed at the first The specific spatiotemporal DTW distance value calculated by each template This represents the total number of candidate templates remaining after spatial prior pruning, and determines the length of the vector. This serves as a pre-selected primitive dictionary. Through this mathematical encapsulation, the original distance information is transformed into a high-density feature representation, where each dimension of the vector precisely corresponds to a specific action semantic hypothesis, and the numerical value directly reflects the confidence level of that hypothesis. Taking the action of recognizing a hand approaching the mouth as an example, if index 1 in the pre-selected dictionary corresponds to the drinking template and index 2 corresponds to the smoking template, the calculated vector is... The low value of the first dimension of the vector directly indicates to the subsequent neural network that the current action is semantically inclined to drinking water rather than smoking, thus realizing an effective transformation from geometric calculation results to semantic feature input.

[0049] It should be understood that the matching method based on the weighted cost matrix described in the above paragraphs, and the optimization method based on template pre-screening described in the paragraphs, can be implemented independently or in combination. For example, a pre-screening step can be performed first to narrow down the matching range, and then the selected primitive templates can be matched more accurately using the weighted cost matrix method.

[0050] Specifically, in step S400, temporal grammar recognition is performed on the distance vector sequence of the past N time steps to obtain the behavior probability. It is understandable that, because the distance vector sequence generated in the preceding steps only quantifies the geometric and spatial similarity between the driver's action segments at each instant and the discrete primitive template, it lacks a holistic model of the inherent long-term temporal logic and combination rules of complex behaviors. This results in the inability to effectively distinguish between real interactive behaviors that conform to a specific process (such as drinking water consisting of a strict sequence of raising a hand, hovering, and tilting the head) and disordered random limb movements, simply relying on the accumulation of instantaneous matching values. Therefore, in the technical solution of this application, temporal grammar recognition is further performed on the distance vector sequence of the past N time steps to obtain the behavior probability. This maps this series of discrete, physical-level distance metrics to a continuous, semantic-level intent space, utilizing a sequence model with memory capabilities to capture and verify the implicit grammatical structure during the action evolution process. In this way, the probability value of the final output can not only reflect the similarity of the actions in appearance, but also confirm the rationality of the actions in execution logic, thereby achieving accurate intention reasoning and confirmation of complex and continuous abnormal behaviors such as drinking water and smoking at extremely high time resolution.

[0051] Figure 5 This is a flowchart illustrating the process of performing temporal grammar recognition on a distance vector sequence over the past N time steps to obtain behavior probabilities, according to an embodiment of the visual intelligence-based vehicle driving abnormal behavior safety detection method of this application. Figure 5 As shown, step S400 includes: S410, performing a primitive probability transformation based on softened distance on the distance vector sequence of the past N time steps to obtain a primitive probability vector sequence; S420, performing action syntax sequence encoding based on GRU on the primitive probability vector sequence to obtain a hidden state sequence; S430, performing self-attention focusing on key action frames on the hidden state sequence to obtain a global context vector; S440, inputting the global context vector into a classifier to obtain behavior probabilities.

[0052] In step S410, the distance vector sequence from the past N time steps is transformed into a primitive probability vector sequence based on softened distance. It is understandable that the spatiotemporal DTW distance calculated in the preceding steps is merely an unbounded physical quantity representing the geometric difference between the action form and the template. Its value fluctuates significantly due to the influence of the action amplitude and duration. Furthermore, the logic that smaller values ​​indicate greater similarity contradicts the input characteristic typically required by neural networks, where larger values ​​indicate higher confidence. Directly inputting such unnormalized raw distances into subsequent temporal models leads to unstable gradient updates and an imbalance in semantic weight allocation. Therefore, in the technical solution of this application, the distance vector sequence from the past N time steps is further transformed into a primitive probability vector sequence based on softened distance. This allows the use of a nonlinear activation function with a temperature parameter to map the distance metric in geometric space to a confidence distribution in probability space, thereby transforming the absolute difference value into a relative probability representing the current action belonging to each primitive category. This effectively eliminates the influence of dimensions and normalizes the data to the range of 0 to 1, while preserving the competitive relationship and ambiguity information between different behavioral hypotheses, providing standardized probabilistic feature inputs for subsequent models to perform accurate logical reasoning in the time-series dimension.

[0053] More specifically, in a concrete example of this application, the transformation process is performed independently for each time step in the distance vector sequence, aiming to reconstruct a distance vector containing K dimensions (corresponding to K primitive templates) into a probability vector of the same dimension. The operation first performs a negative operation on each element of the distance vector, which reverses the original distance minimization optimization objective into a probability maximization expression. Subsequently, a learnable temperature coefficient is introduced. The negative vector is then scaled by division; this coefficient serves as a key hyperparameter to adjust the entropy or sharpness of the probability distribution. Next, a Softmax normalization operation is performed on the scaled vector, which calculates the exponential function of each dimension's value and divides it by the sum of all exponential function values, ensuring that the cumulative value of all elements in the output vector is 1. Under this mechanism, if the current driver's hand movement trajectory is very close to the smoking primitive in terms of spatiotemporal features, but relatively far from the normal driving primitive, the transformed probability vector will exhibit a high probability value close to 1 in the corresponding smoking dimension, and a low probability value close to 0 in other dimensions. When the action is in an ambiguous intermediate state, such as when the hand has just left the steering wheel but has not yet reached the mouth, the probability vector will retain non-zero probabilities for multiple possibilities such as smoking and drinking with a relatively flat distribution, leaving the final decision to the subsequent temporal context model for comprehensive adjudication.

[0054] In step S420, the primitive probability vector sequence is encoded using GRU-based action grammar sequence encoding to obtain the hidden state sequence. It is understandable that, since the primitive probability vector sequence obtained in the preceding steps is merely a set of discrete, instantaneous state assumptions, the isolated probability distribution cannot reflect the evolutionary logic of the action over time. For example, the act of drinking water necessarily follows a specific temporal grammar: hand raised – mouth lingering – head tilted back to swallow – hand lowered. Furthermore, the simple instantaneous probability peak is easily affected by accidental false detections, making it difficult to judge the completeness and authenticity of the action based on single-point information. Therefore, in the technical solution of this application, the primitive probability vector sequence is further encoded using GRU-based action grammar sequence encoding to obtain the hidden state sequence. This utilizes the unique gating mechanism and memory units of recurrent neural networks to recursively fuse the primitive observation probability at the current moment with the hidden state at historical moments, thereby explicitly modeling the temporal dependencies and state transition rules between action primitives. In this way, the originally fragmented probability signals can be integrated into hidden state representations with long-term contextual memory, enabling the model to automatically suppress random noise interference that does not conform to specific behavioral grammar rules, and to keenly capture coherent action patterns that conform to specific semantic logic.

[0055] More specifically, in a concrete example of this application, the encoding process sequentially inputs a sequence of primitive probability vectors with a time dimension of N into a pre-defined gated recurrent unit (GRU) network. In the computation at each time step, the update gate within the GRU unit dynamically decides how much historical action information (such as a raised hand) from the previous hidden state needs to be retained and passed to the current time step, while the reset gate determines how much redundant historical information irrelevant to the current behavior evolution needs to be ignored when calculating the current candidate hidden state. Subsequently, the network combines the gated historical states with the currently input primitive probability vectors to calculate and update the hidden state vector at the current time step. This hidden state vector not only encodes the probability that the driver's action at the current time step belongs to various primitives, but more importantly, it compresses and stores the complete evolution path information from the start of the action to the present. Through this recursive processing, if the input sequence presents a cyclical pattern of raising the hand, inhaling, lowering the hand, and raising the hand again, which conforms to the characteristics of smoking, the GRU network will output a hidden state sequence that evolves along a specific trajectory in a high-dimensional space. This sequence can strongly represent this periodic temporal grammar structure in terms of numerical distribution, providing high-level semantic features containing complete dynamic logic for subsequent intent classification.

[0056] In step S430, the hidden state sequence undergoes self-attention focusing on key action frames to obtain a global context vector. It is understandable that the driver's continuous action sequence is filled with numerous transitional states and redundant frames lacking discriminative value, such as the simple raising of the arm or the resetting process after an action. The core information truly defining the semantic category of the action is often concentrated in only a very few discrete key frames, such as the instant when the fingers touch the lips. If all time steps in the hidden state sequence are averaged indiscriminately, these high-value semantic signals will be diluted by massive background information, thus reducing the discriminative power of the final feature representation. Therefore, in the technical solution of this application, the hidden state sequence is further subjected to self-attention focusing on key action frames to obtain a global context vector, thereby introducing an adaptive weight allocation mechanism to automatically evaluate the contribution rate of the hidden state at each time step to the final intent determination. In this way, non-essential temporal noise can be efficiently filtered out, and scattered key feature fragments can be aggregated into a highly condensed global context vector, ensuring that the final classification decision is dominated by the most representative moment of action, thereby achieving accurate locking of the driver's intention in driving scenarios containing complex background actions.

[0057] Figure 6 This is a flowchart illustrating the process of using self-attention focusing on key action frames in a hidden state sequence to obtain a global context vector, according to an embodiment of the visual intelligence-based vehicle driving abnormal behavior safety detection method of this application. Figure 6 As shown, step S430 includes: S431, calculating the attention score of each hidden state in the hidden state sequence to obtain the attention score sequence; S432, performing weight normalization on the attention score sequence to obtain the attention weight sequence; S433, performing weighted aggregation on the hidden state sequence based on the attention weight sequence to obtain the global context vector.

[0058] In step S431, the attention score of each hidden state in the hidden state sequence is calculated to obtain an attention score sequence. It is understandable that when a driver performs complex interactive behaviors such as drinking water or smoking, the information density distribution of their action sequence in the time dimension is extremely uneven. What truly defines the nature of the behavior is often only a very brief, crucial segment, such as a finger touching the lips or tilting the head back to swallow. Numerous transitional actions, such as raising, lowering, or hovering, do not possess significant class discrimination. If all hidden states are treated equally without distinction, key micro-motion features will be overwhelmed by redundant temporal background noise. Therefore, in the technical solution of this application, the attention score of each hidden state in the hidden state sequence is further calculated to obtain an attention score sequence. This allows for the automatic evaluation of the semantic relevance strength between each instantaneous state and the final behavioral intention using a parameterized scoring mechanism. This provides accurate quantitative weighting for subsequent feature aggregation, ensuring that the model can keenly capture and lock onto the most discriminative core moments from the lengthy action sequence.

[0059] More specifically, in a concrete example of this application, the computation process relies on a nonlinear transformation network with learnable parameters to be executed independently for each time step. First, the high-dimensional hidden state vector at a certain time step, output by the gated recurrent unit, is input into a fully connected layer and projected onto a hidden feature space through matrix multiplication to extract the salient features of the action state at that time step. Subsequently, nonlinear activation functions such as hyperbolic tangent are applied to the projected vector to increase the model's expressive power and normalize the feature values. Finally, the activated vector is subjected to a dot product operation with a globally shared context query vector automatically learned during training. The scalar result of this operation is defined as the original attention score of the hidden state at that time step. This process is repeated for each hidden state in the sequence, ultimately generating a scalar sequence of the same length as the original action sequence, where the time steps with higher values ​​directly correspond to the keyframe positions with the most semantic value in the driver's actions.

[0060] In step S432, the attention score sequence is weighted and normalized to obtain the attention weight sequence. It is understandable that the original attention scores calculated in the preceding steps only reflect the absolute matching values ​​between the hidden state and the query vector at each independent time step. Their value range is unconstrained and lacks probabilistic statistical significance. Directly using these values ​​for feature aggregation leads to numerical divergence and fails to accurately measure the relative importance competition between different time steps. Therefore, in the technical solution of this application, the attention score sequence is further weighted and normalized to obtain the attention weight sequence, thereby mapping this set of unbounded scalar values ​​to a probability distribution vector whose sum is strictly equal to one. This transforms absolute score differences into relative weight proportions, forcing the model to make a binary choice under limited attention resource constraints. Specifically, it significantly amplifies the signal strength of key action frames such as hand touching lips, while compressing the weights of non-interactive redundant frames to near zero.

[0061] More specifically, in a concrete example of this application, the normalization process utilizes the Softmax function to perform a comprehensive operation on the attention score sequence containing N time-step values. First, an exponential operation is performed on each original score in the sequence. The non-linear amplification property of the exponential function is used to widen the gap between high and low scores, significantly enhancing the scores of keyframes that are only slightly higher than the background noise. Then, the sum of all exponentialized scores is calculated as a normalization factor. Finally, the exponentialized score at each time step is divided by this normalization factor to generate the final attention weight sequence. In this process, if the driver's action sequence includes a drinking action, the score corresponding to the moment of tilting the head back to swallow will be converted into a high weight value close to 1 after this processing, while the score corresponding to the forward and backward arm movements will be converted into a very small weight value, ensuring that the subsequent feature fusion process strictly follows this non-uniform semantic distribution.

[0062] In step S433, it is understood that although the hidden state sequence output by the recurrent neural network retains the temporal information of the entire process, the information of each frame is of equal length and dispersed in the temporal domain. Directly using the last frame state or simple average pooling would cause the key micro-action features to be diluted by the long-term background actions, resulting in the final generated feature vector failing to accurately reflect the core semantics of the behavior. Therefore, in the technical solution of this application, the hidden state sequence is further weighted and aggregated based on the attention weight sequence to obtain a global context vector, thereby performing a feature filtering and reorganization mechanism. Based on the importance distribution calculated in the previous steps, the dispersed temporal features are compressed into a fixed-dimensional global representation. In this way, it can be ensured that the final generated context vector highly matches the key moments that define the nature of the behavior in terms of numerical distribution, thereby maximizing the retention of discriminative information used to distinguish fine-grained behaviors such as drinking water and smoking.

[0063] More specifically, in a concrete example of this application, the aggregation process performs a linearly weighted vector synthesis operation. A zero vector with the same dimension as the hidden state vector is initialized as an accumulator, and then the time-step traversal logic is initiated. For each time step, the normalized attention weight scalar value and the corresponding hidden state feature vector are read, and a scalar-vector multiplication operation is performed, directly mapping the magnitude of the weight value to a scaling factor of the feature vector. The scaled feature vector is then accumulated into the global context vector. Through this mathematical process, hidden states corresponding to keyframes such as a finger touching lips dominate the final vector synthesis due to their larger weight coefficients, determining the direction and magnitude of the global context vector; conversely, hidden states corresponding to invalid transition frames have almost no impact on the final result due to their extremely small weight coefficients. The resulting global context vector, as a highly refined semantic carrier, is transmitted to the subsequent fully connected layer for final intent classification.

[0064] In step S440, the global context vector is input into the classifier to obtain the behavior probability. It is understandable that while the global context vector extracted in the preceding steps highly condenses the spatiotemporal semantic information of the driver's actions, it is essentially still an abstract numerical expression in a high-dimensional feature space. It cannot be directly used as an intuitive basis for determining the driver's specific intentions, and ultimately, a clear confidence index for specific abnormal behavior categories is needed to support subsequent decision-making logic. Therefore, in the technical solution of this application, the global context vector is further input into the classifier to obtain the behavior probability, thereby constructing a mapping relationship from the feature space to the category space. A fully connected layer projects the abstract feature semantics onto the predefined behavior category dimension, and a normalization function outputs a numerical distribution that conforms to the probability axiom. In this way, the internal representation of the deep neural network can be transformed into a classification result with clear physical meaning, quantifying the confidence level of whether the driver's current action belongs to various preset behaviors such as drinking water, smoking, or normal driving, providing a standardized mathematical basis for subsequent safety threshold judgments.

[0065] More specifically, in a concrete example of this application, the classification process first feeds a global context vector of dimension D into a fully connected neural network layer. This layer multiplies the input vector with a pre-trained weight matrix and adds a bias term through matrix multiplication, thereby linearly transforming the D-dimensional feature information into an M-dimensional logistic vector, where M strictly corresponds to the total number of driving behavior categories preset by the system, such as normal driving, smoking, drinking water, and telephone calls. Next, this logistic vector is input into the Softmax activation function module for non-linear processing. This function performs an exponential operation on each element in the vector and divides it by the sum of the exponents of all elements, thereby forcibly compressing the unbounded range of logistic values ​​to the interval between 0 and 1, and ensuring that the sum of the output values ​​of all categories is strictly equal to 1. Through this calculation process, an M-dimensional probability distribution vector is finally output, where the value of each dimension precisely represents the probability that the driver's current behavior belongs to the corresponding category. For example, if the output value of the dimension corresponding to the smoking category is 0.85, it clearly indicates that the system determines that there is a high-risk smoking behavior.

[0066] Specifically, during the model training phase, a time-series sample set containing labeled behavior categories is constructed. The cross-entropy loss function is used as the objective function, and the Adam optimizer is used to perform end-to-end backpropagation training on the overall model, including GRU network parameters, attention mechanism parameters, and classifier fully connected layer weights, until the loss function converges.

[0067] Specifically, in step S500, adaptive risk assessment and graded alarm are performed on the behavior probability to obtain an alarm command. It is understood that since the behavior probability value output by the deep neural network is essentially a continuous confidence signal that dynamically fluctuates with video frames, it is affected by ambient light flicker or the driver's instantaneous posture adjustment. A high probability peak in a single frame does not directly equate to a confirmed dangerous event, and different types of abnormal behaviors, such as smoking and closing one's eyes, have significant differences in safety threat levels. A single threshold judgment logic cannot adapt to differentiated intervention needs. Therefore, in the technical solution of this application, adaptive risk assessment and graded alarm are further performed on the behavior probability to obtain an alarm command. This is used to perform integral verification and smoothing filtering of the instantaneous probability in the time dimension, and to map the confirmed risk event to a preset graded intervention strategy system. This effectively eliminates false alarms caused by model prediction jitter, prevents driver fatigue caused by frequent alarm state switching, and ensures that the output alarm command strictly matches the actual risk urgency of the current driving behavior in terms of intensity and form.

[0068] More specifically, in a concrete example of this application, the implementation process first performs time-domain integration or moving average smoothing on the M-dimensional behavior probability vector output from the classifier in real time. This processing logic forces that the predicted probability of a specific abnormal behavior (such as smoking) must remain above the baseline confidence level within a set time window in order to pass the first level of validity verification, thereby filtering out occasional impulse interference. Subsequently, the smoothed probability value is input into a dual-threshold comparator with hysteresis characteristics. This comparator sets a higher trigger threshold and a lower clear threshold for each behavior. The alarm state is activated only when the probability curve rises above the trigger threshold, and the alarm is deactivated only when the probability falls back below the clear threshold, thus constructing a stability protection mechanism for the alarm state. Finally, based on the confirmed behavior category index, a preset risk strategy mapping table is consulted to generate differentiated control instructions for different behaviors. For example, for the high-risk handheld phone call behavior, a secondary alarm instruction including a buzzer sound and dashboard highlighting is generated, while for the low-risk drinking water behavior, only a background log recording instruction is generated. These structured alarm instructions are sent to the human-machine interaction controller for execution via the vehicle communication bus.

[0069] Furthermore, a vehicle driving abnormal behavior safety detection system based on visual intelligence is also provided.

[0070] Figure 7 This is a block diagram of a vision-based intelligent vehicle driving abnormal behavior safety detection system according to an embodiment of this application. Figure 7 As shown, the vehicle driving abnormal behavior safety detection system 100 based on visual intelligence according to an embodiment of this application includes: a raw video stream acquisition module 110, used to acquire raw video streams collected by an in-vehicle DMS camera; a posture sequence extraction and kinematic feature conversion module 120, used to extract posture sequences and convert kinematic features from the raw video stream to obtain kinematic feature sequences; a dynamic matching module 130, used to dynamically match the kinematic feature sequences with each primitive template in the action primitive dictionary to obtain distance vectors; a temporal syntax recognition module 140, used to perform temporal syntax recognition on the distance vector sequence over the past N time steps to obtain behavior probabilities; and an adaptive risk assessment and graded alarm module 150, used to perform adaptive risk assessment and graded alarms on the behavior probabilities to obtain alarm commands.

[0071] As described above, the visual intelligence-based vehicle driving abnormal behavior safety detection system 100 according to the embodiments of this application can be implemented in various types of computing devices or control units. For example, it can be deployed in the cockpit domain controller of an intelligent connected vehicle, a standalone DMS processing unit installed behind the dashboard, or a high-performance AI inference module integrated into an in-vehicle central computing platform. In one possible implementation, the visual intelligence-based vehicle driving abnormal behavior safety detection system 100 according to the embodiments of this application can be integrated into the computing device as a software module and / or hardware module. For example, the visual intelligence-based vehicle driving abnormal behavior safety detection system 100 can be a resident data processing service in the operating system of the computing device. This software module is configured to perform high-precision extraction of driver posture sequences, spatiotemporal transformation of kinematic features, dynamic matching of action primitives and real-time sequences, and behavioral intent reasoning based on temporal grammar. Alternatively, it can be a dedicated driver state monitoring algorithm program developed for the computing device. Of course, the vehicle driving abnormal behavior safety detection system 100 based on visual intelligence can also be one of the many hardware modules of the computing device or control unit, or it can be embedded in a field-programmable gate array circuit to accelerate the DTW operation and neural network inference with spatiotemporal joint cost in parallel, or it can be an in-vehicle vision processing integrated circuit for a specific application.

[0072] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A safety detection method for abnormal driving behavior in vehicles based on visual intelligence, characterized in that, include: Acquire the raw video stream captured by the vehicle-mounted DMS camera; Pose sequence extraction and kinematic feature transformation are performed on the original video stream to obtain a kinematic feature sequence; Dynamic matching is performed between the kinematic feature sequence and each primitive template in the action primitive dictionary to obtain the distance vector; Temporal syntax recognition is performed on the distance vector sequence over the past N time steps to obtain behavior probabilities; Adaptive risk assessment and graded alarms are performed on the probability of behavior to obtain alarm instructions.

2. The vehicle driving abnormal behavior safety detection method based on visual intelligence according to claim 1, characterized in that, The original video stream is subjected to pose sequence extraction and kinematic feature transformation to obtain a kinematic feature sequence, including: Pose sequence extraction is performed on the original video stream to obtain a smooth pose data sequence; Kinematic feature transformation is performed on the smooth attitude data sequence to obtain the kinematic feature sequence.

3. The vehicle driving abnormal behavior safety detection method based on visual intelligence according to claim 2, characterized in that, Kinematic characteristics include relative angles, normalized velocities at key points, and accelerations.

4. The vehicle driving abnormal behavior safety detection method based on visual intelligence according to claim 1, characterized in that, Dynamic matching is performed on the kinematic feature sequence and each primitive template in the action primitive dictionary to obtain the distance vector, including: Load the feature importance weight vector of the primitive template; Based on the feature importance weight vector, calculate the matching weighted cost matrix between the kinematic feature sequence and the primitive template; Accumulated path optimization is performed on the matching weighted cost matrix to obtain the final matching distance between the primitive template and the kinematic feature sequence.

5. The vehicle driving abnormal behavior safety detection method based on visual intelligence according to claim 1, characterized in that, Temporal grammar recognition is performed on the distance vector sequence over the past N time steps to obtain behavior probabilities, including: The primitive probability vector sequence is obtained by performing a soft distance-based primitive probability transformation on the distance vector sequence of the past N time steps; The primitive probability vector sequence is encoded using GRU-based action syntax sequence encoding to obtain the hidden state sequence. Self-attention focusing on key action frames is performed on the hidden state sequence to obtain the global context vector; Input the global context vector into the classifier to obtain the behavior probability.

6. The vehicle driving abnormal behavior safety detection method based on visual intelligence according to claim 5, characterized in that, Self-attention focusing on key action frames is performed on the hidden state sequence to obtain the global context vector, including: Calculate the attention score of each hidden state in the hidden state sequence to obtain the attention score sequence; The attention score sequence is weighted and normalized to obtain the attention weight sequence; The hidden state sequence is weighted and aggregated based on the attention weight sequence to obtain the global context vector.

7. The vehicle driving abnormal behavior safety detection method based on visual intelligence according to claim 1, characterized in that, Dynamic matching is performed on the kinematic feature sequence and each primitive template in the action primitive dictionary to obtain the distance vector, including: Based on kinematic feature sequences, functional area maps, and real-time keypoint raw pixel coordinate sequences, template pre-screening is performed on the action primitive dictionary to obtain a pre-screened primitive dictionary. Dynamic matching is performed on the kinematic feature sequence, the real-time keypoint original pixel coordinate sequence, and each primitive template in the pre-screened primitive dictionary to obtain the spatiotemporal DTW distance set. Vectorize the spatiotemporal DTW distance set to obtain the distance vector.

8. A vehicle driving abnormal behavior safety detection system based on visual intelligence, characterized in that, include: The raw video stream acquisition module is used to acquire the raw video stream captured by the vehicle-mounted DMS camera; The pose sequence extraction and kinematic feature transformation module is used to extract pose sequences and transform kinematic features from the original video stream to obtain kinematic feature sequences. The dynamic matching module is used to dynamically match the kinematic feature sequence and each primitive template in the action primitive dictionary to obtain the distance vector; The temporal syntax recognition module is used to perform temporal syntax recognition on the distance vector sequence of the past N time steps to obtain the behavior probability; The adaptive risk assessment and graded alarm module is used to perform adaptive risk assessment and graded alarms on the probability of behavior in order to obtain alarm instructions.

9. The vehicle driving abnormal behavior safety detection system based on visual intelligence according to claim 8, characterized in that, The dynamic matching module includes: Feature importance weight vector loading unit, used to load the feature importance weight vector of the primitive template; The matching weighted cost matrix calculation unit is used to calculate the matching weighted cost matrix between the kinematic feature sequence and the primitive template based on the feature importance weight vector; The cumulative path optimization unit is used to perform cumulative path optimization on the matching weighted cost matrix to obtain the final matching distance between the primitive template and the kinematic feature sequence.

10. The vehicle driving abnormal behavior safety detection system based on visual intelligence according to claim 8, characterized in that, The temporal grammar recognition module includes: The primitive probability transformation unit is used to perform a primitive probability transformation based on softened distance on the distance vector sequence of the past N time steps to obtain a primitive probability vector sequence. Action syntax sequence encoding unit, used to perform GRU-based action syntax sequence encoding on primitive probability vector sequence to obtain hidden state sequence; Self-attention focusing unit is used to perform self-attention focusing on key action frames of the hidden state sequence to obtain the global context vector; The behavior probability acquisition unit is used to input the global context vector into the classifier to obtain the behavior probability.

Images