A method and device for recognizing limb conflict behavior and a storage medium

Abstract

The present disclosure provides a method and device for identifying limb conflict behavior and a storage medium, comprising: arranging a monitoring module in a monitoring place, the monitoring module comprising at least a dynamic visual sensor, a depth camera and a voice acquisition unit; extracting limb motion speed features using event spatio-temporal pyramid convolution according to event streams collected by the dynamic visual sensor; extracting limb posture features using a three-dimensional posture estimation network model according to three-dimensional data collected by the depth camera; extracting emotion features using a voice recognition model and a large language model according to voice data collected by the voice acquisition unit; inputting the limb motion speed features, the limb posture features and the emotion features into a multi-modal attention fusion network to determine whether there is a limb conflict behavior. The present disclosure breaks through the limitation of traditional single visual mode, fuses dynamic vision, three-dimensional posture and voice emotion multi-dimensional information, reduces the interference of environmental factors such as strong light and shielding on single mode, and reduces the false negative and false positive rates.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence and monitoring technology, and particularly relates to a method and device for recognizing limb conflict behavior and a storage medium. BACKGROUND

[0002] With the wide application of artificial intelligence technology, behavior recognition systems based on computer vision and voice analysis play an important role in the field of monitoring. Especially in high-risk scenarios such as campuses, vehicle cabins, public transportation, and hospitals, how to accurately and efficiently recognize limb conflict behavior has become one of the key demands of intelligent security systems.

[0003] Traditional limb conflict behavior recognition methods mainly use RGB cameras to collect visual information and rely on image recognition technology of frame sequences, such as optical flow analysis, C3D, I3D, and other action classification networks. By capturing features such as human limb movement trajectories or speed changes, it is determined whether there is a limb conflict behavior. However, in strong light, occlusion, poor shooting angle, and other situations, the collected visual information is easily disturbed, and the image recognition technology is difficult to distinguish between normal limb behaviors such as hugging, and the recognition accuracy of limb conflict behavior is low. SUMMARY

[0004] Embodiments of the present disclosure provide a method and device for recognizing limb conflict behavior and a storage medium to solve the problem of low recognition accuracy of limb conflict behavior due to poor robustness of single modal recognition.

[0005] To solve the above problems, in a first aspect, a method for recognizing limb conflict behavior is provided, comprising:

[0006] A monitoring module is arranged in a monitoring place, and the monitoring module at least includes a dynamic visual sensor, a depth camera, and a voice acquisition unit;

[0007] Event stream collected by the dynamic visual sensor is used to extract limb motion speed features by event spatio-temporal pyramid convolution;

[0008] Three-dimensional data collected by the depth camera is used to extract limb posture features by a three-dimensional pose estimation network model;

[0009] Voice data collected by the voice acquisition unit is used to extract emotion features by a voice recognition model and a large language model;

[0010] The limb motion speed features, the limb posture features, and the emotion features are input into a multi-modal attention fusion network to determine whether there is a limb conflict behavior.

[0011] In a second aspect, a device for recognizing limb conflict behavior is provided, comprising:

[0012] The monitoring site arrangement module is configured to arrange a monitoring module at a monitoring site, the monitoring module comprising at least a dynamic visual sensor, a depth camera, and a voice acquisition unit;

[0013] The dynamic visual sensor acquisition module is configured to extract limb motion speed features from an event stream acquired by the dynamic visual sensor using event spatio-temporal pyramid convolution;

[0014] The depth camera acquisition module is configured to extract limb posture features from three-dimensional data acquired by the depth camera using a three-dimensional pose estimation network model;

[0015] The voice acquisition unit acquisition module is configured to extract emotion features from voice data acquired by the voice acquisition unit using a voice recognition model and a large language model;

[0016] The limb conflict behavior determination module is configured to input the limb motion speed features, the limb posture features, and the emotion features into a multi-modal attention fusion network to determine whether a limb conflict behavior exists.

[0017] In a third aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the limb conflict behavior recognition method according to the first aspect or any possible implementation manner of the first aspect are performed.

[0018] The beneficial effects of the embodiments of the present disclosure include:

[0019] This disclosure provides a method, apparatus, and storage medium for identifying physical conflict behavior, comprising: deploying a monitoring module in a monitored location, the monitoring module including at least: a dynamic visual sensor, a depth camera, and a voice acquisition unit; extracting limb movement speed features using event spatiotemporal pyramid convolution based on the event stream acquired by the dynamic visual sensor; extracting limb posture features using a 3D pose estimation network model based on the 3D data acquired by the depth camera; extracting emotional features using a speech recognition model and a large language model based on the voice data acquired by the voice acquisition unit; and inputting the limb movement speed features, limb posture features, and emotional features into a multimodal attention fusion network to determine whether physical conflict behavior exists. The physical conflict behavior identification method provided by this disclosure overcomes the limitations of traditional single visual modality by fusing dynamic vision, 3D pose, and voice emotion multi-dimensional information, reducing interference from environmental factors such as strong light and occlusion on a single modality, and lowering the false alarm and missed alarm rates. The dynamic visual sensor captures the speed features of rapid movements, and the depth camera provides 3D data, which can accurately distinguish between normal behaviors such as hugging and physical conflict; emotional features supplement semantic and emotional dimensions, and multi-feature fusion makes the judgment more comprehensive. The multimodal data acquisition and fusion mechanism significantly improves the accuracy, timeliness, and interpretability of physical conflict behavior recognition, making it particularly suitable for high-security scenarios such as campuses and public transportation, providing intelligent and automated behavior recognition capabilities for security systems. Attached Figure Description

[0020] Figure 1 A flowchart of a method for recognizing physical conflict behavior provided in an embodiment of this disclosure;

[0021] Figure 2 A flowchart for determining the speed characteristics of limb movements provided in this embodiment of the disclosure;

[0022] Figure 3 A flowchart for determining limb posture features provided in this embodiment of the disclosure;

[0023] Figure 4 A flowchart for determining whether physical conflict exists, provided as an embodiment of this disclosure;

[0024] Figure 5 A structural diagram of a device for recognizing physical conflict behavior provided in an embodiment of this disclosure. Detailed Implementation

[0025] This disclosure provides a method, apparatus, and storage medium for identifying physical conflict behavior. Preferred embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of this disclosure. Furthermore, the embodiments and features described herein can be combined with each other unless otherwise specified.

[0026] The embodiment of the present disclosure provides a method for identifying limb conflict behavior, as shown in the method for identifying limb conflict behavior, comprising: Figure 1

[0027] S101, a monitoring module is arranged in a monitoring place, and the monitoring module at least comprises a dynamic visual sensor, a depth camera and a voice acquisition unit;

[0028] S102, according to the event stream collected by the dynamic visual sensor, the limb motion speed feature is extracted by using the event space-time pyramid convolution;

[0029] S103, according to the three-dimensional data collected by the depth camera, the limb posture feature is extracted by using the three-dimensional posture estimation network model;

[0030] S104, according to the voice data collected by the voice acquisition unit, the emotion feature is extracted by using the voice recognition model and the large language model;

[0031] S105, the limb motion speed feature, the limb posture feature and the emotion feature are input into the multi-modal attention fusion network to determine whether there is limb conflict behavior.

[0032] With the popularization of artificial intelligence technology, the behavior recognition system based on computer vision and voice analysis occupies a pivotal position in the monitoring field. In particular, in the scenes with high risk coefficients such as campus, vehicle cabin, public transportation and hospital, accurately and efficiently identifying limb conflict behavior has become a key demand of the intelligent security system. The traditional limb conflict behavior recognition method mainly relies on RGB camera to collect visual information, and its technical core depends on image recognition technology of frame sequence, such as optical flow analysis and C3D, I3D action classification network. This method captures the moving track and speed change of human body limbs to infer whether there is limb conflict behavior. However, this method has obvious defects: when in strong light, shielding, poor shooting angle and other environments, the collected visual information is easily disturbed; moreover, the image recognition technology is insufficient in distinguishing between normal limb behavior such as hugging and limb conflict behavior, and the traditional camera is easy to blur the image when capturing fast action such as punching and pushing, which makes it difficult to accurately distinguish between hugging and limb conflict behavior, resulting in low recognition accuracy of limb conflict behavior. Therefore, it is necessary to provide a limb conflict behavior recognition method that integrates multi-modal data and large model reasoning to solve the problem of low recognition accuracy of limb conflict behavior.

[0033] ​In the embodiments of the present disclosure, a monitoring module including a dynamic vision sensor, a depth camera and a voice acquisition unit is arranged in a monitoring place to respectively collect event stream, three-dimensional data and voice data. The dynamic vision sensor (DVS, Dynamic Vision Sensor) can be an event-driven vision sensor inspired by the biological retina. Unlike traditional RGB cameras, the DVS generates event stream by capturing pixel brightness change events, can efficiently record dynamic information of fast body movements, and has strong anti-motion blur capability. The depth camera can be an imaging device that can simultaneously obtain a color image (RGB) of a scene and the distance (Depth) of each pixel to the camera. Unlike traditional RGB cameras that can only record colors, the depth camera additionally outputs a depth map (Depth Map), thereby expanding the 2D picture into 3D spatial information. The depth camera can collect three-dimensional data of the scene, accurately obtain the three-dimensional coordinates and spatial relationship of the human body, and make up for the errors in posture judgment of 2D images. The voice acquisition unit, such as a single microphone or a microphone array, is used to pick up voice signals in the monitoring area, including conversation content, tone, volume and other information, to provide data support for emotion feature extraction. Based on the event stream collected by the dynamic vision sensor, the time and space information of the pixel brightness change are recorded, and the event spatiotemporal pyramid convolution technology is used for feature extraction. The event spatiotemporal pyramid convolution (ESPC, Event-based Spatiotemporal Pyramid Convolution) can effectively capture the speed change (such as the speed of punching) of body movements in the time dimension and the movement range in the space dimension through multi-level convolution operation on the event stream in different time scales and space scales, accurately extract the body movement speed features reflecting the intensity of the movement, and solve the blur problem when the traditional RGB camera captures fast movements. The three-dimensional data collected by the depth camera, including the three-dimensional coordinates of each joint of the human body, are input into a three-dimensional pose estimation network model for processing. Through analysis of the three-dimensional coordinate data, the body posture features such as punching height, body contact angle, distance between people, etc. can be extracted, the spatial interaction relationship of the human body is accurately restored, the behaviors such as “hugging” and “pushing” which are easy to confuse are effectively distinguished, and the errors in posture judgment of 2D images are made up. Based on the voice data obtained by the voice acquisition unit, a speech recognition model is used to transcribe the collected voice data into text, and a large language model is input for context semantic and emotion analysis to identify potential threatening statements or emotion escalation behaviors and extract emotion features. For example, a speech recognition model such as Whisper is used to convert the voice data into a text sequence , , is input into a large language model (LLM) for semantic embedding to obtain emotion features, which are represented as:

[0034] ;

[0035] in, This represents the frozen or fine-tuned encoder of a large language model, with the input text sequence as input. , The pre-trained weights can be completely frozen, or they can be fine-tuned on sentiment data. Freezing means the parameters will no longer be updated, while fine-tuning means continuing backpropagation on downstream tasks. The encoder internally uses a multi-layer Transformer structure. After passing through a self-attention and positional feedforward network, the context vector of each token is obtained, and then the context vectors are pooled into sentence-level vectors through a pooling strategy. , The dimension is denoted as , This is typically equal to the hidden layer dimension of a large language model, such as 768, 1024, or 4096. The input is a linear mapping layer, consisting of a weight matrix. and bias vector Together they work. The shape is K×d, where K represents the total number of emotion categories, and each row corresponds to the projection vector of an emotion. The dimension is K, providing a learnable offset for each emotion category. K-dimensional logits are obtained through matrix multiplication and addition, then fed into the Softmax function. Softmax transforms the logits into a probability distribution, where all components are non-negative and sum to 1. Each component corresponds to the confidence level of an emotion category. The final output probability vector represents the model's emotional characteristics of the overall emotional tendency of the speech.

[0036] Furthermore, a multimodal fusion decision network (such as a cross-modal Transformer or a fusion contrast model) is used to jointly infer the output features of each modality to determine whether physical conflict exists. A multimodal attention fusion network automatically learns the weights of different features in judging physical conflict through an attention mechanism; for example, the intensity speed feature and the anger emotion feature have higher weights. This allows for deep fusion and inference of multimodal features. By comprehensively analyzing the combination of multiple features such as body movement speed features, body posture features, and emotional features, the system accurately outputs the judgment result of whether physical conflict exists, achieving efficient recognition in complex scenarios.

[0037] The embodiment of the application breaks through the limitation of traditional single visual mode, fuses dynamic vision, three-dimensional posture and multi-dimensional information of voice emotion, reduces the interference of environmental factors such as strong light and shielding on single mode, and reduces the false negative and false positive rates. The dynamic vision sensor captures the speed characteristics of rapid action, the depth camera provides three-dimensional data, and the normal behavior such as hug and the limb conflict can be accurately distinguished; the emotional characteristics supplement the semantic and emotional dimensions, and the multi-feature fusion makes the judgment more comprehensive. The multi-modal data acquisition and fusion mechanism significantly improves the accuracy, timeliness and explainability of the limb conflict behavior recognition, and is especially suitable for scenes such as campus and public transportation that require high security level, and provides intelligent and automatic behavior recognition capability for security systems.

[0038] In another embodiment of the present disclosure, in the step S102, the event stream collected by the dynamic vision sensor is used to extract the limb motion speed feature by using event space-time pyramid convolution, including:

[0039] Step 1, the event frames in the event stream collected by the dynamic vision sensor are divided into multiple scale regions in the spatial dimension to obtain multi-scale spatial tiles;

[0040] Step 2, for each spatial tile, the event stream is segmented in the time dimension, and the activation of the event in multiple time periods is extracted to obtain multiple space-time sub-regions;

[0041] Step 3, for each space-time sub-region, an event aggregation convolution operation is performed to extract a speed vector feature;

[0042] Step 4, the speed vector features are fused to obtain the limb motion speed feature.

[0043] In the embodiments of the present disclosure, by dividing the multi-scale space blocks in the spatial dimension, segmenting each space block in the time dimension, extracting the velocity vector features of each space-time sub-region, and fusing the velocity vector features, the limb motion velocity features are obtained, which provide a key basis for judging the intensity of limb motion. For the above step 1, the event frames in the event stream collected by the dynamic vision sensor are cut into multiple scale regions in the spatial dimension to obtain multi-scale space blocks. For example, the original image, 1 / 2 scaling, and 1 / 4 scaling are used for segmentation, and different scale space blocks can correspond to the overall motion and local fine motion of the limb, respectively, so that the subsequent extracted velocity features will not miss any motion information in the spatial range, and the spatial characteristics of the limb motion are laid a foundation for being fully captured. For the above step 2, for each space block, the event stream is segmented in the time dimension, and the event stream corresponding to each space block is divided into several continuous time periods according to a certain time interval. Then, the occurrence of events in each time period is observed, and the activation of events in multiple time periods is extracted, such as the number and distribution of events. By combining spatial information with time information, the event activation in different time periods is captured, which can reflect the dynamic changes of the limb motion in the time dimension, such as the different performances of a punching motion in the initial, process and end stages. Then, a plurality of space-time sub-regions are obtained. For the above step 3, for each space-time sub-region, an event aggregation convolution operation is performed to extract the velocity vector features. The event aggregation convolution integrates and processes the event information in each space-time sub-region, analyzes the density of events, that is, the number of events occurring in a unit of time and space, and the change trend of events, such as the transition from sparse to dense or from dense to sparse. Through these analyses, the speed direction and speed size of the limb motion in the region are calculated to form the velocity vector features. The velocity vector features can intuitively and accurately reflect the motion state of the limb in a specific space-time range. For the above step 4, the velocity vector features are fused to obtain the limb motion velocity features. For example, the velocity vector features extracted from each space-time sub-region are integrated together by using weighted fusion and the like. Through fusion, the limb motion velocity information in different spatial scales and different time periods can be comprehensively integrated to form a feature that fully reflects the velocity of the limb motion. The limb motion velocity feature can accurately reflect the intensity of the limb motion, and provides an important speed dimension basis for subsequent judgment of whether there is a limb conflict. The multi-scale space division considers both the overall and local limb motion, and the time segmentation captures the motion changes at different times, so that the extracted velocity features can fully reflect the motion state of the limb in the space-time range. The event aggregation convolution operates on each space-time sub-region to accurately calculate the speed direction and size of the limb motion in the region, so that the velocity vector features are more accurate. Through the combination of space and time, the problem that the traditional frame camera is prone to blur when capturing fast motion is effectively solved, and the extraction effect of the velocity features of fast limb motion is improved.

[0044] In another embodiment of this disclosure, in step 2 above, an event aggregation convolution operation is performed for each spatiotemporal sub-region to extract velocity vector features, including:

[0045] Step 1: For each spatiotemporal sub-region, perform event aggregation convolution operation to obtain the number of events, the inter-event displacement, and the inter-event time interval;

[0046] Step 2: Determine the velocity vector characteristics based on the number of events, the displacement between events, and the time interval between events. The formula is as follows:

[0047] ;

[0048] in, Represents the characteristics of the velocity vector. Indicates the number of events. Indicates displacement between events. Indicates the time interval between events.

[0049] In this embodiment, an event aggregation convolution operation is used to extract three key parameters from each spatiotemporal sub-region: the number of events, the displacement between events, and the time interval between events. Then, based on these three parameters, a velocity vector feature is calculated to accurately quantify the velocity state of limb movements within the spatiotemporal sub-region. For step one above, the data points for each event include pixels... coordinates, pixels Coordinates, timestamp, and polarity; polarity reflects the change in pixel brightness, represented as... , For each spatiotemporal sub-region, parameter extraction is performed using event aggregation convolution. Event aggregation convolution is a convolution method specifically designed for handling dynamic visual sensor event streams. It performs sliding window aggregation on discrete brightness change events within a spatiotemporal sub-region, yielding the event count, inter-event displacement, and inter-event time interval. The event count represents the total number of events contained within the statistical window, reflecting the activity level within that sub-region. The inter-event displacement represents the distance between adjacent events. direction and The coordinate difference of the direction reflects the spatial distance of limb movement. The time interval between events represents the difference in timestamps between two adjacent events, reflecting the temporal rhythm of the action. Efficient parameter extraction is achieved through convolution operations, ensuring accurate capture of action details in each spatiotemporal sub-region. Regarding step two above, Indicates the first For adjacent events The instantaneous velocity component in the direction. Indicates the first For adjacent events The instantaneous velocity component in the direction. Derived by the formula:

[0050] ;

[0051] The obtained speed vector feature integrates the motion information of events in the spatio-temporal sub-region, can reflect the speed of the action, can reflect the direction of the action (positive and negative of the component), and provides a quantitative basis for judging the intensity of the limb action. 、 The event aggregation convolution operation simultaneously obtains the number of events, the displacement between events and the time interval, covers the number, spatial change and time span of the action, provides a complete data basis for speed calculation, and avoids the deviation of the speed feature caused by the lack of parameters. The speed components of multiple events are integrated through the average formula, which weakens the accidental error of a single event, and the obtained speed vector feature can better reflect the real speed trend of the limb action. The speed vector feature is directly related to the spatial displacement and time interval of the event, has clear physical meaning, and is convenient for weight distribution and logical reasoning of the speed vector feature in subsequent multi-modal fusion. As shown in Figure 2 , Figure 2 a flowchart for determining the limb action speed feature, including the following steps:

[0052] S201, dividing the event frame in the event stream collected by the dynamic vision sensor into multiple scale regions in the spatial dimension, to obtain multiple scale space tiles;

[0053] S202, for each space tile, segmenting the event stream in the time dimension, extracting the activation of events in multiple time periods, to obtain multiple spatio-temporal sub-regions;

[0054] S203, for each spatio-temporal sub-region, performing event aggregation convolution operation to obtain the number of events, the displacement between events and the time interval between events;

[0055] S204, determining the speed vector feature according to the number of events, the displacement between events and the time interval between events;

[0056] S205, fusing the speed vector feature to obtain the limb action speed feature; the flow ends.

[0057] In another embodiment of the present disclosure, in the step S103, the limb pose feature is extracted from the three-dimensional data collected by the depth camera by using a three-dimensional pose estimation network model, including:

[0058] Step 1: extracting the position information of the joint node from the three-dimensional data collected by the depth camera by using a three-dimensional pose estimation network model;

[0059] Step 2: constructing a limb pose time sequence feature sequence according to the position information of the joint node;

[0060] Step 3, inputting the limb posture time sequence feature sequence into a time convolution network model to model and classify the limb action, and determining the limb posture feature.

[0061] In the embodiments of the present disclosure, the three-dimensional pose estimation network model is used to extract the joint position information from the three-dimensional data, the limb pose time sequence feature sequence is constructed according to the position information of the joint, the limb pose time sequence feature sequence is input into the time convolution network model, the limb pose feature is determined through modeling and classification, and the precise description of the spatial pose and dynamic change of the limb is realized. For the above step 1, based on the three-dimensional data collected by the depth camera, the three-dimensional pose estimation network model (3DPoseNet) is used for processing. The three-dimensional pose estimation network model can automatically identify the human body joints such as shoulder joint, elbow joint, wrist joint, etc. through the feature learning of the three-dimensional data, and output the coordinates (x, y, z) of each joint in the three-dimensional space. For example, when identifying the punching action, the three-dimensional position coordinates of the wrist joint and the elbow joint can be accurately positioned, reflecting the spatial stretching state of the arm. The ambiguity of the joint position in the 2D image is broken through, and an accurate spatial coordinate basis is provided for subsequent pose analysis. For the above step 2, the limb pose time sequence feature sequence is constructed. Based on the extracted joint position information, the joint coordinates of the continuous frames can be concatenated in time sequence to form the limb pose time sequence feature sequence. The limb pose time sequence feature sequence can directly reflect the dynamic change of the limb pose, for example, in the limb contact action, the displacement of the chest position from back to front, the angle change of the elbow joint from bending to straightening, etc. will form a specific numerical change rule in the time sequence. The static spatial pose is converted into dynamic time sequence, which provides data support for capturing the continuity of the action. For the above step 3, the time convolution network model models and classifies the limb action. The limb pose time sequence feature sequence is input into the time convolution network (TCN, Temporal Convolutional Network) model, and the limb pose time sequence feature sequence is modeled and classified through multi-layer convolution operation, and finally the limb pose feature is output. The time convolution network can expand the time receptive field without increasing the parameters through the dilated convolution kernel, which can effectively capture the long-distance time dependence in the time sequence, such as the complete time sequence logic of “lifting hand-accelerating-hitting” in the punching action. Through the learning of different action sequences, such as the slow and gentle pose sequence of “hugging” and the violent and rapid pose sequence of “pushing and touching”, the model can classify the motion pattern of the pose and convert it into a distinguishable limb pose feature, which provides a key basis for judging the limb conflict. Based on the three-dimensional data of the depth camera, the three-dimensional coordinates of the joints can be accurately obtained compared with the 2D image, which avoids the pose misjudgment caused by occlusion and overlap in the plane view, and lays a precise foundation for subsequent feature extraction. By constructing the limb pose time sequence feature sequence, the static joint position is converted into dynamic action trajectory, which can record the change process of the limb pose over time, and makes up for the limitations of single moment pose information.The time convolution network is good at capturing local and global dependencies in time series data, and can effectively distinguish similar poses and improve the classification accuracy of different limb behaviors through the time evolution pattern of actions.

[0062] In yet another embodiment of the present disclosure, the limb pose time sequence feature sequence comprises: an inter-joint speed and angle change sequence;

[0063] In step 2, the limb pose time sequence feature sequence is constructed according to the position information of the joint nodes, comprising:

[0064] According to the position information of the joint nodes, the inter-joint speed and angle change sequence is constructed, and the formula is represented as:

[0065] ;

[0066] ; ;

[0067] wherein, represents a set of joint node positions, represents the position of the joint node, represents the position coordinates of the joint node, represents the time difference, represents the time difference, represents the first joint node, represents the second joint node, represents the change amount of the angle between the joint node and the joint node , represents the inverse cosine function.

[0068] In an embodiment of the present disclosure, based on the position information of the joint nodes, the inter-joint speed is obtained by calculating the difference in joint position between adjacent time points, and the angle change amount between the joints is calculated by using vector dot product and inverse cosine function. Finally, these speeds and angle changes are arranged in time sequence to form an inter-joint speed and angle change sequence reflecting the dynamic relationship of joint motion, so as to depict the motion trend and spatial angle change law of the joint in the limb action. Through the formula:

[0069] ;

[0070] ​The spatial angle relationship between the joints can be accurately quantified by vector dot product and module length calculation. The distance interference to angle judgment is avoided, and the robustness of the feature is enhanced. The joint speed directly reflects the speed of the movement, and the angle change reflects the spatial posture conversion of the joint. The combination of the two can better capture the dynamic characteristics of the action, and provide a key basis for distinguishing between "hug" (slow, small angle change) and "push and touch" (fast, large angle change). The joint speed and angle change sequence is recorded in time sequence, which completely retains the evolution process of the body movement from the beginning to the end, and provides rich dynamic features for the subsequent time convolution network to capture the long-term dependence of the action.

[0071] In another embodiment of the present disclosure, the body posture time sequence feature sequence includes: an elbow and wrist movement speed sequence and a fist movement sequence;

[0072] In the above step 2, the body posture time sequence feature sequence is constructed according to the position information of the joint nodes, including:

[0073] Step 1: According to the position information of the joint nodes, the elbow and wrist movement speed sequence is constructed, and the formula is expressed as:

[0074] ;

[0075] ;

[0076] Wherein, represents the position coordinates of the elbow or wrist joint node, represents the position coordinates of the elbow or wrist joint node at the time t, represents the time difference;

[0077] Step 2: In the case that the elbow and wrist movement speed sequence is greater than the first threshold value, and the speed direction is towards the target person, the fist movement sequence is determined according to the position coordinates of the elbow and wrist joint nodes.

[0078] In the embodiment of the present disclosure, the movement speed is calculated by the position information of the joint nodes to form the elbow and wrist movement speed sequence, and then the fist movement sequence is selected from the elbow and wrist movement speed sequence based on the speed threshold value and the direction condition. For the above step 1, the instantaneous speed of the elbow and wrist joints is calculated to form a time sequence reflecting the dynamic change, and the elbow and wrist movement speed sequence is obtained, and the formula is expressed as:

[0079] ;

[0080] ;

[0081] Wherein, represents the position coordinates of the elbow or wrist joint node,​ Indicates in time, The time difference is represented by the following: Regarding step two above, based on the elbow and wrist movement speed sequence, a time sequence matching the characteristics of a punch is extracted through conditional filtering. The movement speed of the elbow and wrist is greater than a preset first threshold, which is set based on the speed difference between normal and violent movements. For example, the speed of a punch is usually much greater than the speed of a normal hand swing. For instance, if the movement speed of the elbow and wrist is 3 m / s, the first threshold is 2 m / s. The speed direction is towards the target person. For example, if the vector angle between the movement speed direction of the elbow and wrist and the target person is less than a preset angle threshold, it is judged as a punching action. For example, the preset angle threshold is 90 degrees. Then, the punching action sequence is obtained based on the position coordinates of the corresponding elbow and wrist joint nodes. Focusing on the elbow and wrist, compared to features of all joints, reduces interference from irrelevant information and improves sensitivity to conflicting movements. For example, the speed changes of the elbow and wrist in a punching action are much more significant than those of other joints; targeted features can improve recognition efficiency. Speed ​​changes are recorded in a time sequence format, preserving the dynamic process of the punching motion. Combined with directional changes, this provides a complete basis for subsequent punching motion recognition, improving the temporal accuracy of motion recognition.

[0082] In another embodiment of this disclosure, the limb posture temporal feature sequence includes: a distance change sequence between the chest positions of the target person and a limb contact action sequence;

[0083] In step 2 above, a temporal feature sequence of limb posture is constructed based on the position information of the joints, including:

[0084] Step 1: Based on the location information of the key points, construct a sequence of distance changes between the chest positions of the target person. The formula is as follows:

[0085] ;

[0086] ;

[0087] in, This indicates the change in distance between the chest positions of the target person. This indicates the coordinates of the position of the chest joint nodes. Indicates in time, and These represent the first target and the second target, respectively.

[0088] Step 2: If the instantaneous rate of change of the distance between the chest positions of the target person is less than the second threshold and the distance between the chest positions of the target person is less than the third threshold, determine the limb contact action sequence based on the position coordinates of the chest joint nodes.

[0089] In this embodiment of the disclosure, based on the chest position information of the target person, the distance between the chest positions of the first target person and the second target person is calculated and formed into a sequence that changes over time. Then, based on threshold conditions for distance magnitude and rate of change, a sequence of limb contact actions is selected from the distance change sequence. Regarding step one above, for continuous time frames ( ), calculate the distance between the chest positions of the first target and the second target respectively. Then arrange them in chronological order. This forms a sequence of distance changes between the chest positions of the target individuals. For example, when two people approach each other from a distance, It will gradually decrease when a collision occurs. It may fluctuate rapidly within a small range; when separated, The distance between the chest positions of the target individuals gradually increases. The sequence of distance changes between the chest positions of the target individuals directly reflects the dynamic relationship between their relative positions. Regarding step two above, based on the sequence of distance changes between the chest positions of the target individuals, a temporal sequence that meets the characteristics of limb contact is extracted through conditional filtering to obtain the limb contact action sequence. For example, if the instantaneous rate of change of the distance between the chest positions of the target individuals, as represented by the sequence of distance changes between the chest positions of the target individuals, is less than the second threshold, and the distance between the chest positions of the target individuals is less than the third threshold, the formula is expressed as:

[0090] ;and ;

[0091] in, The instantaneous rate of change of the distance between the chest positions of the target person. This indicates the second threshold; the minus sign "-" indicates that the velocity direction is towards the target's chest area. This indicates the distance between the chest areas of the target individuals. Indicates the third threshold, for example, , For the "pushing and colliding" behavior, the time sequence is as follows: , ; , ; , ; , ; For the "hugging" behavior, the time sequence is as follows: 、 ; 、 ; 、 ; 、 ; 、 ; . The corresponding limb contact action sequence is obtained according to the position coordinates of the chest joint node. Focusing on the chest, which is the core area of the human body, the distance change can intuitively reflect the relative position relationship between the characters (such as approaching and moving away), and compared with the limbs and other parts, it can more stably reflect the overall interaction state, and provide core space features for judging the limb conflict such as pushing and colliding. The double conditions of “distance less than the third threshold value” (close enough in space) and “instantaneous change rate less than the second threshold value” can effectively distinguish “hugging” from “pushing and colliding”, and reduce misjudgment. The limb contact action sequence constructed can accurately capture the close-range interaction state between characters, and provide key space and dynamic basis for the recognition of contact behaviors such as “pushing and colliding” in limb conflict. As shown in Figure 3 , Figure 3 is a flowchart for determining the limb posture feature, including the following steps:

[0092] S301, according to the three-dimensional data collected by the depth camera, the position information of the joint node is extracted by using a three-dimensional posture estimation network model;

[0093] S302, according to the position information of the joint node, a limb posture time sequence feature sequence is constructed; the limb posture time sequence feature sequence can include: joint speed and angle change sequence, elbow and wrist action speed sequence, fist action sequence, distance change sequence between target character chest position, and limb contact action sequence;

[0094] S303, input the limb posture time sequence feature sequence into the time convolution network model to model and classify the limb action, and determine the limb posture feature; the flow ends.

[0095] In another embodiment of the present disclosure, as shown in Figure 4 , the step S105 above, the limb action speed feature, the limb posture feature and the emotion feature are input into the multi-modal attention fusion network to determine whether there is a limb conflict behavior, including:

[0096] S401, after the limb action speed feature, the limb posture feature and the emotion feature are unified in dimension by splicing or weighted summation, input into the multi-head sub-attention structure to obtain a cross-modal fusion unified feature vector;

[0097] S402, input the cross-modal fusion unified feature vector into the full connection classifier to obtain the probability score of whether there is a limb conflict behavior;

[0098] The method further comprises:

[0099] Step 3, training the multi-modal attention fusion network using a weighted cross-entropy loss.

[0100] In the embodiments of the present disclosure, after unifying the limb motion speed feature, the limb posture feature and the emotion feature in the same dimension, a cross-modal fusion unified feature vector is generated through a multi-head sub-attention structure; then the cross-modal fusion unified feature vector is input into a fully connected classifier to obtain a probability score of the existence of a limb conflict behavior; and the network is trained using a weighted cross-entropy loss to optimize the model performance. Through deep fusion of multi-modal features and model training optimization, accurate judgment of the limb conflict behavior is realized. For the above step S401, since the dimensions of the limb motion speed feature, the limb posture feature and the emotion feature may be different, the dimensions need to be unified through splicing or weighted summation to ensure that the features can be input into the attention network. The unified dimension features are input into a multi-head sub-attention structure, which includes multiple parallel sub-attention heads, each of which focuses on learning the correlation between different modal features. The features are weighted and fused by calculating the attention weights between the features, and finally a cross-modal fusion unified feature vector is output. For example, after the limb motion speed feature, the limb posture feature and the emotion feature are unified in the same dimension through splicing or weighted summation, they are represented as 、 、 respectively, and are input into the multi-head sub-attention structure to obtain a cross-modal fusion unified feature vector represented as:

[0101] );

[0102] For the above step S402, the fully connected classifier is composed of a multi-layer neural network, the input is the cross-modal fusion unified feature vector, and the cross-modal fusion unified feature vector is mapped to a probability score in the interval [0, 1] through linear transformation and activation function (such as ReLU, Sigmoid). The closer the score is to 1, the higher the possibility of the existence of a limb conflict behavior; the closer it is to 0, the higher the possibility of a normal behavior. The cross-modal fusion unified feature vector is input into the fully connected classifier to obtain the probability score of the existence of a limb conflict behavior, which is represented by the formula:

[0103] );

[0104] wherein, and represent the weights and biases of the fully connected classifier, represents the cross-modal fusion unified feature vector, mapping to [0, 1], to obtain a probability score of the limb conflict behavior. For step 3 above, the weighted cross-entropy loss introduces a weight parameter to make the loss function pay more attention to the prediction error of the limb conflict behavior sample. The multi-modal attention fusion network is trained using the weighted cross-entropy loss, which is expressed by the formula:

[0105] ;

[0106] wherein, represents a true label, represents a multi-modal attention fusion network output probability, and represent positive and negative sample weights, respectively. In the multi-modal attention fusion network training phase, the multi-modal attention fusion network output probability and the true label are substituted into the weighted cross-entropy loss formula, and the network parameters are updated through back propagation to minimize the loss value. Finally, the multi-modal attention fusion network learns a large number of normal samples while significantly improving the recognition ability of a small number of conflict samples. The multi-head sub-attention structure can automatically learn the weight of different modal features, avoid information redundancy caused by simple splicing, enhance the relevance between features, and solve the information island problem in traditional modal fusion. The fully connected classifier outputs a probability score based on the fused unified feature vector, reducing the false alarm or false alarm caused by a single feature misjudgment. The weighted cross-entropy loss assigns higher weights to a small number of samples to solve the sample imbalance problem, so that the model pays more attention to key conflict samples in training, and improves the recognition stability in complex scenes.

[0107] Based on the same disclosure concept, the embodiments of the disclosure also provide a limb conflict behavior recognition device. Since the principles of the problems solved by these devices are similar to the aforementioned limb conflict behavior recognition method, the implementation of the device can be referred to the implementation of the aforementioned method, and the repeated parts will not be described again.

[0108] The embodiments of the disclosure provide a limb conflict behavior recognition device, as shown in Figure 5 , comprising:

[0109] The monitoring place arrangement module 501 is configured to arrange a monitoring module in a monitoring place, wherein the monitoring module at least includes a dynamic visual sensor, a depth camera, and a voice acquisition unit.

[0110] The dynamic visual sensor acquisition module 502 is configured to extract limb motion speed features by using event spatio-temporal pyramid convolution based on an event stream acquired by the dynamic visual sensor.

[0111] The depth camera acquisition module 503 is configured to extract limb posture features by using a three-dimensional posture estimation network model based on three-dimensional data acquired by the depth camera.

[0112] The voice collection unit collection module 504 is configured to extract emotion features by using a voice recognition model and a large language model according to voice data collected by the voice collection unit;

[0113] The limb conflict behavior determination module 505 is configured to input the limb motion speed feature, the limb posture feature and the emotion feature into a multi-modal attention fusion network to determine whether there is a limb conflict behavior.

[0114] In another embodiment of the present disclosure, the dynamic visual sensor collection module 502 is configured to divide event frames in an event stream collected by the dynamic visual sensor into regions of multiple scales in a spatial dimension to obtain multi-scale spatial tiles;

[0115] For each spatial tile, the event stream is segmented in a time dimension to extract activation of events in multiple time periods to obtain multiple spatio-temporal sub-regions;

[0116] For each spatio-temporal sub-region, an event aggregation convolution operation is performed to extract a speed vector feature;

[0117] The speed vector features are fused to obtain a limb motion speed feature.

[0118] In another embodiment of the present disclosure, the dynamic visual sensor collection module 502 is configured to, for each spatio-temporal sub-region, perform an event aggregation convolution operation to obtain the number of events, the inter-event displacement and the inter-event time interval;

[0119] The speed vector feature is determined according to the number of events, the inter-event displacement and the inter-event time interval, and is expressed by a formula as follows:

[0120] ;

[0121] wherein, represents the speed vector feature, represents the number of events, represents the inter-event displacement, represents the inter-event time interval.

[0122] In another embodiment of the present disclosure, the depth camera collection module 503 is configured to extract position information of a joint node by using a three-dimensional posture estimation network model according to three-dimensional data collected by the depth camera;

[0123] The limb posture time sequence feature sequence is constructed according to the position information of the joint node;

[0124] The limb posture time sequence feature sequence is input into a time convolution network model to model and classify limb motions, and a limb posture feature is determined.

[0125] In another embodiment of this disclosure, the limb posture temporal feature sequence includes: a sequence of inter-joint velocity and angle changes;

[0126] The depth camera acquisition module 503 is used to construct a sequence of velocity and angle changes between joints based on the position information of the joints, expressed by the following formula:

[0127] ;

[0128] ; ;

[0129] in, A set representing the positions of joint nodes. express The position of each joint node. Indicates the position coordinates of the joint nodes. Indicates in time, Indicates time difference, Indicates the first Each joint node Indicates the first Each joint node Represents joint nodes and joint nodes The change in the angle between them This represents the inverse cosine function.

[0130] In another embodiment of this disclosure, the limb posture temporal feature sequence includes: elbow and wrist movement speed sequence and punching movement sequence;

[0131] The depth camera acquisition module 503 is used to construct a sequence of elbow and wrist movement speeds based on the position information of the joint points, expressed by the following formula:

[0132] ;

[0133] ;

[0134] in, This indicates the coordinates of the elbow or wrist joint. Indicates in time, Indicates time difference;

[0135] If the elbow and wrist movement speeds, as represented by the elbow and wrist movement speed sequence, are greater than a first threshold and the speed direction is toward the target person, the punching action sequence is determined based on the position coordinates of the elbow and wrist joint nodes.

[0136] In yet another embodiment of the present disclosure, the limb posture time sequence feature sequence comprises a distance change sequence between chest positions of the target person and a limb contact action sequence.

[0137] The depth camera acquisition module 503 is configured to construct a distance change sequence between chest positions of the target person according to the position information of the joint node, and the formula is expressed as:

[0138]

[0139]

[0140] wherein, represents the distance change between the chest positions of the target person, represents the position coordinates of the chest joint node, represents the position coordinates of the chest joint node at the moment, represent the first target person and the second target person, respectively;

[0141] In a case where the instantaneous change rate of the distance between the chest positions of the target person represented by the distance change sequence between the chest positions of the target person is less than a second threshold value, and the distance between the chest positions of the target person is less than a third threshold value, the limb contact action sequence is determined according to the position coordinates of the chest joint node.

[0142] In yet another embodiment of the present disclosure, the limb conflict behavior determination module 505 is configured to input the limb action speed feature, the limb posture feature and the emotion feature into a multi-head sub-attention structure after uniform dimension by splicing or weighted summation to obtain a cross-modal fusion uniform feature vector.

[0143] The cross-modal fusion uniform feature vector is input into a fully connected classifier to obtain a probability score of whether there is a limb conflict behavior.

[0144] The limb conflict behavior determination module 505 is further configured to:

[0145] The multi-modal attention fusion network is trained by using a weighted cross-entropy loss.

[0146] Based on the same disclosure, the present disclosure provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is run by a processor to perform the steps of the limb conflict behavior recognition method according to any one of the above embodiments.

[0147] ​​​​Those skilled in the art can clearly understand the present disclosure from the above description of the embodiments that the present disclosure can be implemented by hardware, or by means of software and necessary universal hardware platform. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments of the present disclosure.

[0148] Those skilled in the art can understand that the drawings are only schematic of a preferred embodiment, and the modules or flows in the drawings are not necessarily required for implementing the present disclosure.

[0149] Those skilled in the art can understand that the modules in the devices in the embodiments can be distributed in the devices in the embodiments according to the description of the embodiments, or can be changed and located in one or more devices different from the embodiments. The modules in the above embodiments can be combined as one module, or can be further split into multiple sub-modules.

[0150] The above serial numbers of the embodiments of the present disclosure are only for description, not representing the advantages and disadvantages of the embodiments.

[0151] Obviously, those skilled in the art can make various modifications and changes to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and changes of the present disclosure belong to the scope of the claims of the present disclosure and their equivalent technologies, the present disclosure also intends to include these modifications and changes.

Claims

1. A method for identifying physical conflict behavior, characterized in that, include: A monitoring module is deployed in the monitoring area, and the monitoring module includes at least: a dynamic visual sensor, a depth camera, and a voice acquisition unit; Based on the event stream acquired by the dynamic visual sensor, event spatiotemporal pyramid convolution is used to extract limb movement speed features; Based on the 3D data acquired by the depth camera, a 3D pose estimation network model is used to extract limb pose features; Based on the voice data collected by the voice acquisition unit, emotional features are extracted using a voice recognition model and a large language model; The body movement speed features, body posture features, and emotional features are input into a multimodal attention fusion network to determine whether physical conflict behavior exists, including: The limb movement speed feature, the limb posture feature, and the emotion feature are concatenated or weighted and summed to unify the dimensions, and then input into a multi-head sub-attention structure to obtain a cross-modal fusion unified feature vector. The cross-modal fused unified feature vector is input into a fully connected classifier to obtain a probability score of whether physical conflict behavior exists. The method further includes: The multimodal attention fusion network is trained using weighted cross-entropy loss.

2. The method as described in claim 1, characterized in that, The step of extracting limb movement speed features using event spatiotemporal pyramid convolution based on the event stream acquired by the dynamic visual sensor includes: The event frames in the event stream acquired by the dynamic vision sensor are divided into regions of multiple scales in the spatial dimension to obtain multi-scale spatial patches. For each spatial tile, the event flow is segmented along the time dimension, and the activation status of the event in multiple time periods is extracted to obtain multiple spatiotemporal sub-regions; For each spatiotemporal sub-region, perform event aggregation convolution operation to extract velocity vector features; The velocity vector features are fused to obtain the limb movement velocity features.

3. The method as described in claim 2, characterized in that, The step of performing event aggregation convolution operation for each spatiotemporal sub-region to extract velocity vector features includes: For each spatiotemporal sub-region, perform event aggregation convolution operation to obtain the number of events, the inter-event displacement, and the inter-event time interval; Based on the number of events, the displacement between events, and the time interval between events, the velocity vector characteristics are determined, expressed by the following formula: ； in, Represents the characteristics of the velocity vector. Indicates the number of events. Indicates displacement between events. Indicates the time interval between events.

4. The method as described in claim 1, characterized in that, The step of extracting limb pose features using a 3D pose estimation network model based on the 3D data acquired by the depth camera includes: Based on the 3D data acquired by the depth camera, a 3D pose estimation network model is used to extract the position information of the joints; Based on the position information of the joints, a temporal feature sequence of limb postures is constructed; The temporal feature sequence of limb postures is input into a temporal convolutional network model to model and classify limb movements, thereby determining limb posture features.

5. The method as described in claim 4, characterized in that, The temporal feature sequence of limb posture includes: the sequence of velocity and angle changes between joints; The step of constructing a temporal feature sequence of limb posture based on the position information of the joints includes: Based on the position information of the joints, a sequence of velocity and angle changes between joints is constructed, expressed by the following formula: ； ； ； in, A set representing the positions of joint nodes. express The position of each joint node. Indicates the position coordinates of the joint nodes. Indicates in time, Indicates time difference, Indicates the first Each joint node Indicates the first Each joint node Represents joint nodes and joint nodes The change in the angle between them This represents the inverse cosine function.

6. The method as described in claim 4, characterized in that, The limb posture temporal feature sequence includes: elbow and wrist movement speed sequence and punching movement sequence; The step of constructing a temporal feature sequence of limb posture based on the position information of the joints includes: Based on the position information of the joint points, a velocity sequence of elbow and wrist movements is constructed, expressed by the formula: ； ； in, This indicates the coordinates of the elbow or wrist joint. Indicates in time, Indicates time difference; If the elbow and wrist movement speeds, as represented by the elbow and wrist movement speed sequence, are greater than a first threshold and the speed direction is toward the target person, the punching action sequence is determined based on the position coordinates of the elbow and wrist joint nodes.

7. The method as described in claim 4, characterized in that, The limb posture temporal feature sequence includes: the distance change sequence between the chest positions of the target person and the limb contact action sequence; The step of constructing a temporal feature sequence of limb posture based on the position information of the joints includes: Based on the location information of the key points, a distance change sequence between the chest positions of the target person is constructed, expressed by the formula: ； ； in, This indicates the change in distance between the chest positions of the target person. This indicates the coordinates of the position of the chest joint nodes. Indicates in time, and These represent the first target and the second target, respectively. If the instantaneous rate of change of the distance between the chest positions of the target person, as represented by the distance change sequence between the chest positions of the target person, is less than a second threshold, and the distance between the chest positions of the target person is less than a third threshold, the limb contact action sequence is determined based on the position coordinates of the chest joint nodes.

8. A device for recognizing physical conflict behavior, characterized in that, include: A monitoring site deployment module is used to deploy monitoring modules in a monitoring site. The monitoring module includes at least: a dynamic visual sensor, a depth camera, and a voice acquisition unit. The dynamic visual sensor acquisition module is used to extract limb movement speed features by using event spatiotemporal pyramid convolution based on the event stream acquired by the dynamic visual sensor. The depth camera acquisition module is used to extract limb posture features based on the 3D data acquired by the depth camera using a 3D pose estimation network model. The voice acquisition unit acquisition module is used to extract emotional features based on the voice data acquired by the voice acquisition unit using a speech recognition model and a large language model. A physical conflict behavior determination module is used to input the physical movement speed features, the physical posture features, and the emotional features into a multimodal attention fusion network to determine whether physical conflict behavior exists, including: The limb movement speed feature, the limb posture feature, and the emotion feature are concatenated or weighted and summed to unify the dimensions, and then input into a multi-head sub-attention structure to obtain a cross-modal fusion unified feature vector. The cross-modal fused unified feature vector is input into a fully connected classifier to obtain a probability score of whether physical conflict behavior exists. The physical conflict behavior determination module is also used for: The multimodal attention fusion network is trained using weighted cross-entropy loss.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method for recognizing physical conflict behavior as described in any one of claims 1 to 7.

Images