An intelligent video-based behavior feature analysis and recognition system

Abstract

The application relates to the technical field of intelligent video monitoring, in particular to a behavior feature analysis and recognition system based on intelligent video, which comprises a video data acquisition module for extracting human body skeleton key point coordinates in a monitoring video, position change amounts between adjacent frames and relative trunk distribution states; a physical feature extraction module for determining a center of mass offset trend and an offset change rate trend according to the key point coordinates and changes; and determining a joint linkage state according to an upper limb joint angle change amount and consistency; and an interaction feature analysis module for, when containing at least two target objects, determining a party that initiatively initiates an approach and an interaction potential change trend according to a shortening trend of a relative distance and acceleration and combining self physical features of each target object. The application solves the technical problem that, due to a lack of time sequence sensing capability for center of mass offset, joint coordination and interaction cause and effect and the like physical evolution trends, an existing system cannot timely and accurately recognize a behavior evolution initiation stage.

Description

Technical Field

[0001] This invention relates to the field of intelligent video surveillance technology, and in particular to a behavioral feature analysis and recognition system based on intelligent video. Background Technology

[0002] Video-based behavior recognition is a core technology in the field of intelligent video surveillance, widely used in scenarios such as public safety, campus security, and industrial inspection. Existing behavior recognition methods can be mainly divided into two categories: one is deep learning methods based on RGB images or video frames, which extract appearance features for classification through convolutional neural networks; the other is recognition methods based on human skeletal key points, which extract skeletal point coordinates and then use graph neural networks or classifiers to determine behavior.

[0003] However, the above methods face problems such as delayed identification timing and high false alarm rate in actual deployment. Furthermore, in multi-person interaction scenarios, existing methods can only detect superficial features such as "the distance between two people shortens" or "physical contact", and cannot distinguish between the active initiator and the passive party at the physical level, resulting in unclear causal responsibility attribution.

[0004] Therefore, how to construct a video behavior feature analysis and recognition system that can perceive the physical development of behavior and identify and clarify the causal responsibility of interactions in the early stages of behavioral evolution is a technical problem that urgently needs to be solved in this field.

[0005] Chinese Patent Publication No. CN120877371A discloses a method and system for identifying abnormal behavior in video surveillance based on edge AI, belonging to the field of intelligent video analysis technology. The method includes: performing frame segmentation processing on the video surveillance data stream to obtain a video frame sequence; establishing a target trajectory prediction model to predict the location region of the current frame and generate target location prediction data; extracting multi-scale features from the video frame sequence; fusing and matching the target feature vector with the target location prediction data to construct an enhanced feature matrix; employing an attention mechanism to assign weights to key behavioral features; setting a dynamic threshold adjustment mechanism to dynamically adjust the abnormal behavior judgment threshold based on personnel density; inputting the adjusted feature data into an abnormal behavior classifier for identification and judgment; outputting the abnormal behavior identification result; and generating an abnormal event report. This invention improves the accuracy of abnormal behavior detection in complex monitoring scenarios, reduces false alarm and false negative rates, and achieves millisecond-level real-time response capabilities.

[0006] However, the aforementioned method and system for identifying abnormal behavior in video surveillance based on edge AI has the following problems:

[0007] The scheme cannot perceive the physical development of behavior from normal to abnormal, nor can it perform advanced identification in the initial stage; its abnormality judgment relies on human-preset thresholds and lacks the ability to adapt to different scenarios; in multi-person interaction scenarios, it can only output group behavior category labels and cannot determine the active initiator and passive party in physical conflict from a physical perspective, resulting in the lack of causal responsibility attribution. Summary of the Invention

[0008] To address this, the present invention provides a behavior feature analysis and recognition system based on intelligent video, which overcomes the problem in the prior art that the lack of temporal perception of physical evolution trends such as centroid shift, joint coordination, and interactive causality leads to the inability to identify behavior in a timely and accurate manner during the initial stage of behavior evolution.

[0009] To achieve the above objectives, this invention provides a behavior feature analysis and recognition system based on intelligent video. It includes: The video data acquisition module is used to access the monitoring video stream in real time, extract the position coordinates of the key points of the human skeleton of the target object in the field of view in the image space frame by frame, record the position change and direction of each key point coordinate between adjacent acquisition times, and record the distribution state of each key point relative to the center of the torso at the current time and the previous time. The physical feature extraction module is used to determine the offset trend and offset rate trend of the human body's center of mass in the horizontal direction based on the position coordinates of each key point at the current acquisition time and the direction and amount of position change compared with the previous time. It also determines the joint linkage state of the current action based on the angle change and consistency between the upper limb joints and the trunk joints at the current time and the previous time. The interaction feature analysis module is used to determine the party that actively initiates the approach during the relative motion and the trend of interaction potential energy change, based on the shortening trend and the difference in shortening acceleration of the relative distance between the target objects at the current time relative to the previous time, as well as the physical characteristics of each target object, when there are at least two target objects. The behavior evolution recognition module is used to temporally correlate the horizontal offset trend of the centroid, the joint linkage state, and the motion trend of the target object corresponding to the party that actively initiates the approach at multiple consecutive acquisition times, determine the continuity and cumulative trend of the changes in each physical state over time since the start of the current behavior, and output the corresponding evolution stage recognition results.

[0010] Furthermore, the video data acquisition module includes, The skeleton key point extraction unit is used to estimate human pose for each frame of the surveillance video stream, extract the position coordinates of the skeleton key points of each target object in the field of view in the image space, and assign an independent target number to each target object. The temporal state recording unit is used to cache the skeletal key point position coordinate data of each frame in the order of acquisition time, record the position change and direction of each key point coordinate between adjacent acquisition times, and record the distribution state of each key point relative to the center of the torso at the current time and the previous time.

[0011] Furthermore, the change in position is determined based on the position coordinates of key points in adjacent frames; The direction of change is determined based on the offset of the position coordinates of key points in adjacent frames in the horizontal and vertical directions of the image plane. The distribution state is determined based on the comparison results of the orientation and distance of each key point in adjacent frames relative to the center point of the torso.

[0012] Furthermore, the physical feature extraction module includes, The centroid offset trend determination unit is used to determine the offset trend and offset change rate trend of the human centroid in the horizontal direction based on the position coordinates of each key point at the current acquisition time and the direction and amount of position change compared with the previous time. The joint linkage state determination unit is used to determine the joint linkage state of the current action based on the change in angle between the upper limb joint and the trunk joint at the current moment relative to the previous moment and the consistency of the change.

[0013] Furthermore, the angle change is determined based on the angle value formed between the upper limb joint and the adjacent trunk joint in adjacent frames; The consistency of change is determined based on the direction of angle change of each of the upper limb joints and the direction of change of the angle of each joint in each frame during multiple consecutive acquisition times.

[0014] Furthermore, the interaction feature parsing module includes, The interaction initiator determination unit is used to determine the party that actively initiates the approach during the relative motion process based on the difference in the shortening trend and shortening acceleration of the relative distance between each target object at the current time relative to the previous time, and in combination with the physical characteristics of each target object itself. The potential energy trend determination unit is used to determine the trend of interaction potential energy based on the active initiator determined by the interaction initiator determination unit, combined with the offset trend of the active party's centroid and the continuity of the joint linkage state in multiple consecutive acquisition moments.

[0015] Furthermore, the shortening trend is determined based on the comparison of the relative distance between the centroids of two target objects in adjacent frames; The shortening acceleration is determined based on the relationship between the shortening of the relative distance between two target objects in consecutive acquisition times, and is used to determine the rate at which the two target objects are relatively close to each other.

[0016] Furthermore, the behavior evolution recognition module includes, The state-time correlation unit is used to construct a temporal state chain based on the centroid offset trend determination results, joint linkage state determination results, and motion trend determination results of the active approaching party at multiple consecutive acquisition times. The evolution stage identification unit is used to identify the evolution stage of the current behavior based on the continuity of changes in each physical state in the time-series state chain and the cumulative trend of changes, and output the corresponding evolution stage identification result.

[0017] Furthermore, the continuity of change is determined based on the persistence of each physical state in the time-series state chain across multiple consecutive acquisition times; The cumulative trend of the changes is determined based on the frame-by-frame changes in the centroid offset, joint angle changes, and centroid displacement of the active party in the time-series state chain.

[0018] Furthermore, the evolutionary stages include at least a normal stage, an early evolutionary stage, a middle evolutionary stage, and a completion evolutionary stage, wherein, Based on the centroid offset trend determination result of no offset or offset direction not continuously stable, and the joint linkage state determination result of non-linkage state, it is identified as a normal stage. Based on the fact that the centroid offset trend maintains a stable change continuity across multiple consecutive acquisition times and the centroid offset shows an increasing cumulative trend, the joint linkage state has not yet been activated, and it is identified as an early stage of evolution. Based on the fact that the centroid offset trend maintains a stable and continuous change across multiple consecutive acquisition times, and the joint linkage state is activated and remains stable, while the change in the angle of each joint shows an increasing cumulative trend, it is identified as the mid-stage of evolution. Based on the fact that the center of the hip extends beyond the vertical support range boundary of the center of the ankle in the horizontal direction, and at least one key point of the supporting foot has been raised compared to the start time, it is identified as the evolution completion stage; The vertical support range is the vertical projection range of the area enclosed by the contact points between the human body's two feet and the support surface on the horizontal plane. The start time is the moment when the centroid offset trend is first determined to be a valid acquisition time.

[0019] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention achieves the extraction of features with clear physical semantics from the original skeletal coordinates by using a progressive processing link of four modules: video data acquisition, physical feature extraction, interactive behavior feature analysis, and behavior evolution recognition. This solves the problem that traditional deep features are difficult to interpret. By tracking the continuity and cumulative trend of changes in each physical feature over a continuous time sequence, behavior recognition is improved from single-frame pattern matching to dynamic perception of behavior evolution stages. It can complete situational recognition in the early stages of behavior evolution. At the same time, the interactive behavior feature analysis module achieves physical-level attribution of the active initiator and the passive party in the interactive scene by fusing displacement comparison and physical feature consistency judgment, overcoming the defect of traditional methods that cannot distinguish between active and passive approach by relying only on distance changes.

[0020] Furthermore, this invention provides a temporally correlated raw data foundation for subsequent physical feature analysis by caching bone coordinates frame by frame and comparing the positional changes, directions of change, and relative trunk distribution states between adjacent frames at each key point. Based on this, the physical feature extraction module tracks the consistency of the offset direction of the hip relative to the ankle in consecutive frames and the frame-by-frame trend of the offset amount, thereby realizing the validity of the centroid offset trend and the distinction between the increasing / decreasing state of the rate of change. By constructing angles with at least two upper limb joints as vertices and comparing the consistency of the opening / closing directions and the synchronization of the angle changes of multiple upper limb joints in consecutive frames, it realizes the quantitative discrimination of joint linkage state and collaborative physical intention, which can be stably captured in the early stage of behavioral evolution.

[0021] Furthermore, this invention combines displacement comparison, centroid offset direction consistency judgment, and joint linkage state comparison to achieve multi-physical feature fusion judgment of the party actively initiating the approach. It can isolate the party with the true intention of actively leaning forward and coordinated limb characteristics from the surface motion amplitude, effectively overcoming the shortcomings of traditional methods that rely solely on distance changes or motion amplitude and cannot distinguish between active and passive approach. It solves the problem of false alarms and missed alarms caused by unclear responsible parties in interactive scenarios. At the same time, the potential energy trend determination unit further tracks the frame-by-frame change direction of the centroid displacement of the active party and combines the continuity of its centroid offset trend with the continuous activation of the joint linkage state to comprehensively judge the enhancement or decay of interactive potential energy, significantly improving the timeliness of warnings. In addition, since all judgments are based on the comparison of physical state trends in continuous time sequence rather than single-frame transient matching, false alarms caused by non-substantive interactive behaviors such as pedestrians briefly approaching and then separating are avoided.

[0022] Furthermore, this invention continuously tracks physical states such as centroid offset trend and joint linkage state during the behavioral development period, and identifies behavior by judging the continuity and cumulative trend of changes in physical states over continuous time. It does not rely on pattern matching of single-frame pose, and can identify the target object in time before it completes dangerous behavior, thus fundamentally reducing the false alarm rate. Attached Figure Description

[0023] Figure 1 This is a block diagram of the module connection of the intelligent video-based behavioral feature analysis and recognition system according to an embodiment of the present invention; Figure 2 This is a block diagram illustrating the internal composition and data processing logic of the physical feature extraction module in an embodiment of the present invention. Figure 3 This is a block diagram illustrating the internal composition and data processing logic of the interactive feature parsing module in an embodiment of the present invention. Figure 4 This is a block diagram illustrating the internal composition and data processing logic of the behavior evolution recognition module in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0025] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0026] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0027] Please see Figure 1 The diagram shown is a block diagram of the behavioral feature analysis and recognition system based on intelligent video according to an embodiment of the present invention. The behavioral feature analysis and recognition system according to an embodiment of the present invention includes: The video data acquisition module is used to access the monitoring video stream in real time, extract the position coordinates of the key points of the human skeleton of the target object in the field of view in the image space frame by frame, record the position change and direction of each key point coordinate between adjacent acquisition times, and record the distribution state of each key point relative to the center of the torso at the current time and the previous time. The physical feature extraction module, which is connected to the video data acquisition module, is used to determine the offset trend and offset rate trend of the human body's center of mass in the horizontal direction based on the position coordinates of each key point at the current acquisition time and the direction and amount of position change compared with the previous time. It also determines the joint linkage state of the current action based on the angle change and consistency between the upper limb joints and the trunk joints at the current time relative to the previous time. The interaction feature parsing module is connected to the video data acquisition module and the physical feature extraction module respectively. It is used to determine the party that actively initiates the approach during the relative motion and the trend of interaction potential energy change, based on the shortening trend and the difference in shortening acceleration of the relative distance between each target object at the current time relative to the previous time, as well as the physical characteristics of each target object itself, under the condition of containing at least two target objects. The behavior evolution recognition module is connected to the physical feature extraction module and the interaction feature parsing module respectively. It is used to perform temporal correlation on the horizontal offset trend of the centroid, the joint linkage state, and the motion trend of the target object corresponding to the party that actively initiates the approach at three or more consecutive acquisition times. It determines the continuity and cumulative trend of the change of each physical state over time since the start of the current behavior, and outputs the corresponding evolution stage recognition result.

[0028] Specifically, this invention utilizes a progressive processing chain of four modules: video data acquisition, physical feature extraction, interactive behavior feature analysis, and behavior evolution recognition. This enables the extraction of features with clear physical semantics from the original skeletal coordinates, including centroid offset trends, offset change rate trends, joint linkage states, and interactive potential energy change directions. This solves the problem of traditional deep features being difficult to interpret. By tracking the continuity and cumulative trends of changes in each physical feature over a continuous time sequence, behavior recognition is upgraded from single-frame pattern matching to dynamic perception of behavior evolution stages. It can complete situational recognition in the early stages of behavior evolution. At the same time, the interactive behavior feature analysis module achieves physical-level attribution of the active initiator and the passive party in the interactive scene by fusing displacement comparison and physical feature consistency judgment. This overcomes the shortcomings of traditional methods that rely solely on distance changes and cannot distinguish between active and passive approach.

[0029] In this embodiment, the video data acquisition module includes, The skeleton key point extraction unit is connected to the surveillance camera to access the surveillance video stream in real time, perform human pose estimation processing on each frame of the image, extract the position coordinates of the human skeleton key points of each target object in the field of view in the image space, and assign independent target numbers to distinguish multiple target objects in the field of view. The temporal state recording unit is connected to the skeletal key point extraction unit. It is used to cache the skeletal key point position coordinate data corresponding to each frame in the order of acquisition time, record the position change and direction of each key point coordinate between adjacent acquisition times, and record the distribution state of each key point relative to the center of the torso at the current time and the previous time.

[0030] The position change is determined based on the degree of difference between the position coordinates of the key point in the current frame and the position coordinates of the same key point in the previous frame, and is used to determine the displacement amplitude of the key point between adjacent acquisition times.

[0031] The direction of change is determined by the offset of the position coordinates of the key point in the current frame relative to the position coordinates of the same key point in the previous frame in the horizontal and vertical directions of the image plane, which is used to determine the displacement direction of the key point.

[0032] The distribution status is determined based on the comparison between the orientation and distance of each key point in the current frame relative to the center point of the torso and the orientation and distance of each key point in the previous frame relative to the center point of the torso. This is used to determine whether and how the spatial distribution of each limb relative to the torso changes between adjacent time points.

[0033] In this embodiment, for the distribution of each key point relative to the center of the torso, the timing state recording unit takes the midpoint of the left and right hips as the center of the torso, determines the orientation region and distance of each key point relative to the center of the torso in the current frame and the previous frame, respectively, compares the orientation region and distance of each key point in the two frames point by point, determines whether the orientation of each key point relative to the torso has changed, or whether the distance has increased or decreased, and takes the comparison result as the distribution change.

[0034] It is worth noting that if a key point is not successfully detected in a frame due to occlusion, the timing state recording unit marks the position information of the key point as missing and does not calculate its change. The recording will be restored when the key point is detected again in a subsequent frame.

[0035] In this embodiment, human pose estimation can be implemented using deep learning-based human pose estimation algorithms, such as OpenPose, HRNet, or MediaPipe. Those skilled in the art can choose to use them according to the actual application scenario, and no specific limitation is made here.

[0036] Please see Figure 2 The diagram shown illustrates the internal structure and data processing logic of the physical feature extraction module in this embodiment of the invention. In this embodiment, the physical feature extraction module includes: The centroid offset trend determination unit is connected to the video data acquisition module and is used to determine the offset trend and offset rate trend of the human centroid in the horizontal direction based on the position coordinates of each key point at the current acquisition time and the direction and amount of position change compared with the previous time. The joint linkage state determination unit is connected to the video data acquisition module and is used to determine the joint linkage state of the current action based on the angle change and consistency between the upper limb joint and the trunk joint at the current moment and the previous moment. The angle change is determined based on the difference between the angle value formed by the upper limb joint and the adjacent trunk joint in the current frame and the corresponding angle value in the previous frame, and is used to determine the deflection amplitude of the upper limb joint between adjacent acquisition times.

[0037] The consistency of change is determined based on the comparison results of the angle change directions of at least two upper limb joints and the comparison results of whether the angle change direction of each joint is synchronized in three or more consecutive acquisition times. This is used to determine whether each upper limb joint deflects in the same direction between adjacent acquisition times and whether the angle change of each joint has a synchronous increasing or decreasing trend in time sequence.

[0038] In this embodiment, for any acquisition time, the centroid offset trend determination unit compares the offset direction of the hip center relative to the ankle center in the horizontal direction in the current frame with the offset direction in the previous frame. If the two directions are consistent, it is determined that the centroid has an offset trend in that direction. If the offset direction is consistent in three or more consecutive acquisition times, it is determined that the offset trend is valid. Here, a single acquisition time is a frame.

[0039] To determine the trend of offset change, the centroid offset trend determination unit compares the horizontal offset of the hip center relative to the ankle center in the current frame with the offset in the previous frame. If the offset in the current frame is greater than the offset in the previous frame, it is determined that the centroid offset shows an increasing relationship between adjacent time points. If the offset between adjacent time points shows the above increasing relationship in three or more consecutive acquisition time points, it is determined that the centroid offset is in a state of increasing rate of change, and the direction of the rate of change is the same as the offset direction. If the offset in the current frame is less than the offset in the previous frame, it is determined that the centroid offset shows a decreasing relationship between adjacent time points. If the offset between adjacent time points shows the above decreasing relationship in three or more consecutive acquisition time points, it is determined that the centroid offset is in a state of decreasing rate of change.

[0040] For determining the offset direction, the centroid offset trend determination unit takes the coordinate difference direction between the hip center and the ankle center in the horizontal direction in the current frame as the current offset direction, and takes the corresponding coordinate difference direction in the previous frame as the previous offset direction. The two directions are consistent because they point in the same direction in the horizontal plane. The magnitude of the offset is determined by the absolute value of the coordinate difference between the hip center and the ankle center in the horizontal direction.

[0041] In this embodiment, for any one of at least two upper limb joints, the joint linkage state determination unit uses the joint as the vertex, the line connecting the upper limb joint to the adjacent trunk joint as the first side, and the line connecting the upper limb joint to the distal joint of the upper limb joint as the second side. By calculating the angle between the first side and the second side in the image plane, the angle in the current frame and the angle in the previous frame are determined. If the angle in the current frame is greater than the angle in the previous frame, it is determined that the joint angle increases between adjacent time points, and the direction of change is the opening direction. If the angle in the current frame is less than the angle in the previous frame, it is determined that the joint angle decreases between adjacent time points, and the direction of change is the closing direction.

[0042] To determine the consistency of the angle change direction, the joint linkage state determination unit determines the angle change direction of at least two upper limb joints respectively, compares the angle change directions of each joint with each other, and if the angle change directions of at least two upper limb joints are both opening or both closing, then the angle change directions of each joint are determined to be consistent, and the current action is in a joint linkage state; if the angle change directions of the upper limb joints are both opening and closing, then the angle change directions of each joint are determined to be inconsistent, and the current action is in a non-linkage state.

[0043] In this embodiment, the upper limb joints include at least the elbow joint and the wrist joint, and may also include the shoulder joint.

[0044] Specifically, this invention provides a temporally correlated raw data foundation for subsequent physical feature analysis by caching bone coordinates frame by frame and comparing the positional changes, directions of change, and relative trunk distribution states between adjacent frames at each key point. Based on this, the physical feature extraction module tracks the consistency of the offset direction of the hip relative to the ankle in consecutive frames and the frame-by-frame trend of the offset amount, thereby realizing the validity of the centroid offset trend and the distinction between the increasing / decreasing state of the rate of change. Furthermore, by constructing angles with at least two upper limb joints as vertices and comparing the consistency of the opening / closing directions and the synchronization of the angle changes of multiple upper limb joints in consecutive frames, it realizes the quantitative discrimination of joint linkage state and collaborative physical intention, which can be stably captured in the early stages of behavioral evolution.

[0045] Please see Figure 3The diagram shown illustrates the internal structure and data processing logic of the interaction feature parsing module in this embodiment of the invention. In this embodiment, the interaction feature parsing module includes: The interaction initiator determination unit, which is connected to the video data acquisition module and the physical feature extraction module, is used to determine the party that actively initiates the approach during the relative motion process when there are at least two target objects, based on the difference in the shortening trend and shortening acceleration of the relative distance between the target objects at the current time relative to the previous time.

[0046] The potential energy trend determination unit, which is connected to the interaction initiator determination unit and the physical feature extraction module, is used to determine the trend of interaction potential energy based on the active initiator determined by the interaction initiator determination unit, combined with the centroid offset trend and joint linkage state of the active initiator in three or more consecutive acquisition times.

[0047] The shortening trend is determined by comparing the relative distance between the centroids of two target objects in the current frame with the relative distance between the centroids of two target objects in the previous frame, in order to determine whether the spatial distance between two target objects continues to decrease between adjacent acquisition times.

[0048] The shortening acceleration is determined based on the relationship between the shortening of the relative distance between two target objects in three or more consecutive acquisition times, and is used to determine the rate at which the two target objects are relatively close to each other.

[0049] In this embodiment, the interaction initiator determination unit determines the first relative distance between the centroids of the first target object and the second target object in the current frame, and the second relative distance between the centroids of the first target object and the second target object in the previous frame. If the first relative distance is less than the second relative distance, it is determined that the relative distance between the two target objects shows a shortening relationship between adjacent time points. If the relative distance shows a shortening relationship in three or more consecutive acquisition time points, it is determined that there is an effective relative approach trend between the two target objects. The centroid of the target object is determined by the position of the center point of the target object's torso, which is the midpoint of the left and right hips.

[0050] To determine the shortening acceleration, the interaction initiator unit, based on the determination of a relative approaching trend, determines the amount of shortening of the relative distance between adjacent moments. It compares the shortening amount at the current moment with the shortening amount at the previous moment. If the shortening amount at the current moment is greater than the shortening amount at the previous moment, it is determined that the shortening amount shows an increasing relationship between adjacent moments. If the shortening amount between adjacent moments in three or more consecutive acquisition moments shows an increasing relationship, it is determined that there is a shortening acceleration between the two target objects. If the shortening amount between adjacent moments in three or more consecutive acquisition moments shows a decreasing relationship, it is determined that the shortening acceleration is decreasing.

[0051] For determining the party that actively initiates the approach, the interaction initiator determination unit, based on the determination of a valid relative approach trend, determines the first displacement of the centroid of the first target object relative to the previous moment, and the second displacement of the centroid of the second target object relative to the previous moment. If the first displacement is greater than the second displacement, it is initially determined that the first target object has a larger range of motion. Simultaneously, the interaction initiator determination unit obtains the centroid offset trend determination results and joint linkage state determination results of each target object output by the physical feature extraction module. It compares the centroid offset trend direction of the first target object with the displacement direction of the first target object. If the two directions are consistent, it is determined that the first target object has a forward tilting posture matching the approach direction. It also compares the joint linkage state of the first target object with the joint linkage state of the second target object. If the first target object is in a joint linkage state while the second target object is in a non-linkage state, it is further confirmed that the first target object is the party that actively initiates the approach. If the second displacement is greater than the first displacement, and the centroid offset direction of the second target object is consistent with its displacement direction, and the second target object is in a joint linkage state while the first target object is in a non-linkage state, it is determined that the second target object is the party that actively initiates the approach.

[0052] To determine the trend of interactive potential energy changes, after identifying the party that actively initiates the approach, the potential energy trend determination unit obtains the centroid offset trend determination result and joint linkage state determination result of the active party from the physical feature extraction module. In three or more consecutive acquisition moments, the potential energy trend determination unit tracks the frame-by-frame change direction of the centroid displacement of the active party. If the centroid displacement of the active party increases frame by frame in three or more consecutive acquisition moments, and the centroid offset trend of the active party maintains a stable continuity of change within the same time period and the joint linkage state remains active, then the interactive potential energy is determined to be in an increasing trend. If the centroid displacement of the active party decreases frame by frame in three or more consecutive acquisition moments, or the centroid offset trend is interrupted and the joint linkage state is released, then the interactive potential energy is determined to be in a decreasing trend.

[0053] Regarding the method for determining the displacement, the interaction initiator determines it by the sum of the absolute values ​​of the coordinate differences between the target object's centroid in the current frame and its position coordinates in the previous frame.

[0054] Interactive potential energy, which is jointly defined by the centroid displacement of the initiating party, the continuity of the centroid offset trend, and the activation of the joint linkage state, is used to measure the physical approach pressure exerted by the initiating party on the passive party: when the centroid displacement of the initiating party increases frame by frame, the centroid offset trend remains stable and continuous, and the joint linkage state remains active, the interactive potential energy is enhanced, and the initiating party's movement trend points towards the passive party and continues to accumulate; conversely, when the centroid displacement of the initiating party decreases frame by frame, or the centroid offset trend is interrupted, or the joint linkage state is released, the interactive potential energy decays, and the approach pressure of the initiating party is waning.

[0055] Specifically, this invention combines displacement comparison, centroid offset direction consistency judgment, and joint linkage state comparison to achieve multi-physical feature fusion judgment of the party actively initiating the approach. It can isolate the party with the true intention of actively leaning forward and coordinated limb characteristics from the surface motion amplitude, effectively overcoming the shortcomings of traditional methods that rely solely on distance changes or motion amplitude and cannot distinguish between active and passive approach. It solves the problem of false alarms and missed alarms caused by unclear responsible parties in interactive scenarios. At the same time, the potential energy trend determination unit further tracks the frame-by-frame change direction of the centroid displacement of the active party and combines the continuity of its centroid offset trend with the continuous activation of the joint linkage state to comprehensively judge the enhancement or decay of interactive potential energy, significantly improving the timeliness of warnings. In addition, since all judgments are based on the comparison of physical state trends in continuous time sequence rather than single-frame transient matching, false alarms caused by non-substantive interactive behaviors such as pedestrians briefly approaching and then separating are avoided.

[0056] Please see Figure 4 The diagram shown illustrates the internal composition and data processing logic of the behavior evolution recognition module in this embodiment of the invention. In this embodiment, the behavior evolution recognition module includes... The state-time correlation unit is connected to the physical feature extraction module and the interactive feature parsing module. It is used to obtain the results of the centroid offset trend in the horizontal direction, the joint linkage state, and the motion trend of the target object corresponding to the party that actively initiates the approach at three or more consecutive acquisition times. It constructs a temporal state chain according to the order of acquisition times and temporally correlates the above three physical state judgment results at each time. The evolution stage identification unit, which is connected to the state-time correlation unit, is used to identify the evolution stage of the current behavior based on the continuity of the changes of each physical state in the time-series state chain and the cumulative trend of the changes, and output the corresponding evolution stage identification result.

[0057] Change continuity is determined based on the persistence of each physical state in the time-series state chain across three or more consecutive acquisition times. It is used to determine whether each physical state remains stable over time or is interrupted.

[0058] The cumulative trend of changes is determined by the cumulative statistical method of the changes in the centroid offset, the changes in the angles of each joint, and the centroid displacement of the active party in the temporal state chain in each frame of three or more consecutive acquisition times. This method is used to determine the temporal trend characteristics of the changes in each physical state.

[0059] In this embodiment, the state-time association unit combines the centroid offset trend determination result, joint linkage state determination result, and active party motion trend determination result received at each moment according to the order of acquisition time, generates a state vector corresponding to that moment, and arranges the state vectors of all moments in chronological order to form a complete temporal state chain.

[0060] Regarding the continuity of changes in each physical state in the time-series state chain, the state-series association unit compares the judgment result of each physical state at the current moment with that at the previous moment. If the judgment result of a certain physical state at the current moment is the same as that at the previous moment, it is determined that the physical state continues between adjacent moments; if they are different, it is determined that the physical state is interrupted or changed between adjacent moments; if the physical state continues in three or more consecutive acquisition moments, it is determined that the physical state has stable continuity of change.

[0061] For the cumulative trend of the changes corresponding to each physical state in the temporal state chain, the state temporal association unit obtains the frame-by-frame change direction of the centroid offset, the change of each joint angle, and the centroid displacement of the active party in three or more consecutive acquisition times. If a certain change increases in each frame in three or more consecutive acquisition times, it is determined that the change shows an increasing cumulative trend; if it decreases in each frame, it is determined that the change shows a decreasing cumulative trend; if the frame-by-frame direction alternates between increasing and decreasing, it is determined that the change shows a fluctuating cumulative trend.

[0062] In this embodiment, the evolution stage identification unit starts from the beginning of the time sequence state chain, traverses the state vectors of each time point in chronological order, and identifies the evolution stage of the current behavior based on the continuity of the changes in each physical state and the cumulative trend of the changes. The evolution stage includes the normal stage, the early stage of evolution, the middle stage of evolution, and the completion stage of evolution.

[0063] For the identification of the normal stage, after the evolution stage identification unit traverses the temporal state chain, if the centroid offset trend determination result is no offset or the offset direction is not continuous and stable, and the joint linkage state determination result is non-linkage state, then the current behavior is identified as being in the normal stage.

[0064] For the transition recognition from the normal stage to the early stage of evolution, if the evolution stage recognition unit detects a centroid shift trend in three or more consecutive acquisition times and the result is that there is a shift trend in a fixed direction and the shift trend has a stable change continuity, and the centroid shift amount shows an increasing cumulative trend in three or more consecutive acquisition times, and the joint linkage state determination result is still a non-linkage state, then the current behavior is recognized as a transition from the normal stage to the early stage of evolution.

[0065] For the transition identification from the early stage of evolution to the middle stage of evolution, the evolution stage identification unit, based on the already identified early stage of evolution, further detects that the joint linkage state determination result is a linkage state and that the linkage state maintains a stable change continuity in three or more consecutive acquisition times. At the same time, the change in the angle of each joint shows an increasing cumulative trend in three or more consecutive acquisition times. Then, the current behavior is identified as a transition from the early stage of evolution to the middle stage of evolution.

[0066] For the transition recognition from the intermediate stage to the completed stage of evolution, the evolution stage recognition unit, based on the already identified intermediate stage of evolution, determines the transition from the intermediate stage of evolution to the completed stage of evolution if the horizontal offset of the hip center relative to the ankle center reaches a point where the hip center exceeds the vertical support range boundary of the ankle center, and at least one key point of the supporting foot has risen in the vertical direction compared to the initiation time. The vertical support range is the vertical projection range of the polygonal area enclosed by the contact points between the human body's two feet and the support surface on the horizontal plane. When the position of the hip center in the horizontal direction exceeds the boundary of this projection range, the human body's center of mass has left the stable support area. The start time is the moment when the center of mass offset trend is first determined to be a valid acquisition moment.

[0067] In this embodiment, for interactive scenarios, the evolution stage identification unit, based on the identification of the early or middle stage of evolution, also acquires the cumulative trend of the centroid displacement of the target object corresponding to the party that actively initiates the approach in the state time sequence chain. If the centroid displacement of the active party shows an increasing cumulative trend in three or more consecutive acquisition moments and its displacement direction points towards the passive party, then the movement trend of the active party is used as an auxiliary basis for determining the evolution stage of the interactive behavior, and the active party and the passive party are simultaneously marked when the evolution stage identification result is output. If the centroid displacement of the active party shows a decreasing cumulative trend in three or more consecutive acquisition moments, then it is determined that the interactive potential energy is decaying. At this time, even if the centroid offset trend and joint linkage state are still in the middle stage of evolution, the evolution stage identification result marks the interactive potential energy decay state.

[0068] Specifically, this invention continuously tracks physical states such as centroid offset trend and joint linkage state during the behavioral development period, and identifies behavior by judging the continuity and cumulative trend of changes in physical states over continuous time. It does not rely on pattern matching of single-frame pose, and can identify the target object in time before it has completed the dangerous behavior, thus fundamentally reducing the false alarm rate.

[0069] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A behavior feature analysis and recognition system based on intelligent video, characterized in that, include, The video data acquisition module is used to access the monitoring video stream in real time, extract the position coordinates of the key points of the human skeleton of the target object in the field of view in the image space frame by frame, record the position change and direction of each key point coordinate between adjacent acquisition times, and record the distribution state of each key point relative to the center of the torso at the current time and the previous time. The physical feature extraction module is used to determine the offset trend and offset rate trend of the human body's center of mass in the horizontal direction based on the position coordinates of each key point at the current acquisition time and the direction and amount of position change compared with the previous time. It also determines the joint linkage state of the current action based on the angle change and consistency between the upper limb joints and the trunk joints at the current time and the previous time. The interaction feature analysis module is used to determine the party that actively initiates the approach during the relative motion and the trend of interaction potential energy change, based on the shortening trend and the difference in shortening acceleration of the relative distance between the target objects at the current time relative to the previous time, as well as the physical characteristics of each target object, when there are at least two target objects. The behavior evolution recognition module is used to temporally correlate the horizontal offset trend of the centroid, the joint linkage state, and the motion trend of the target object corresponding to the party that actively initiates the approach at multiple consecutive acquisition times, determine the continuity and cumulative trend of the changes in each physical state over time since the start of the current behavior, and output the corresponding evolution stage recognition results.

2. The intelligent video-based behavioral feature analysis and recognition system according to claim 1, characterized in that, The video data acquisition module includes, The skeleton key point extraction unit is used to estimate human pose for each frame of the surveillance video stream, extract the position coordinates of the skeleton key points of each target object in the field of view in the image space, and assign an independent target number to each target object. The temporal state recording unit is used to cache the skeletal key point position coordinate data of each frame in the order of acquisition time, record the position change and direction of each key point coordinate between adjacent acquisition times, and record the distribution state of each key point relative to the center of the torso at the current time and the previous time.

3. The intelligent video-based behavioral feature analysis and recognition system according to claim 2, characterized in that, The change in position is determined based on the position coordinates of key points in adjacent frames; The direction of change is determined based on the offset of the position coordinates of key points in adjacent frames in the horizontal and vertical directions of the image plane. The distribution state is determined based on the comparison results of the orientation and distance of each key point in adjacent frames relative to the center point of the torso.

4. The intelligent video-based behavioral feature analysis and recognition system according to claim 3, characterized in that, The physical feature extraction module includes, The centroid offset trend determination unit is used to determine the offset trend and offset change rate trend of the human centroid in the horizontal direction based on the position coordinates of each key point at the current acquisition time and the direction and amount of position change compared with the previous time. The joint linkage state determination unit is used to determine the joint linkage state of the current action based on the change in angle between the upper limb joint and the trunk joint at the current moment relative to the previous moment and the consistency of the change.

5. The intelligent video-based behavioral feature analysis and recognition system according to claim 4, characterized in that, The angle change is determined based on the angle value formed between the upper limb joint and the adjacent trunk joint in adjacent frames. The consistency of change is determined based on the direction of angle change of each of the upper limb joints and the direction of change of the angle of each joint in each frame during multiple consecutive acquisition times.

6. The intelligent video-based behavioral feature analysis and recognition system according to claim 5, characterized in that, The interaction feature parsing module includes, The interaction initiator determination unit is used to determine the party that actively initiates the approach during the relative motion process based on the difference in the shortening trend and shortening acceleration of the relative distance between each target object at the current time relative to the previous time, and in combination with the physical characteristics of each target object itself. The potential energy trend determination unit is used to determine the trend of interaction potential energy based on the active initiator determined by the interaction initiator determination unit, combined with the offset trend of the active party's centroid and the continuity of the joint linkage state in multiple consecutive acquisition moments.

7. The intelligent video-based behavioral feature analysis and recognition system according to claim 6, characterized in that, The shortening trend is determined based on the comparison of the relative distance between the centroids of two target objects in adjacent frames; The shortening acceleration is determined based on the relationship between the shortening of the relative distance between two target objects in consecutive acquisition times, and is used to determine the rate at which the two target objects are relatively close to each other.

8. The intelligent video-based behavioral feature analysis and recognition system according to claim 7, characterized in that, The behavior evolution recognition module includes, The state-time correlation unit is used to construct a temporal state chain based on the centroid offset trend determination results, joint linkage state determination results, and motion trend determination results of the active approaching party at multiple consecutive acquisition times. The evolution stage identification unit is used to identify the evolution stage of the current behavior based on the continuity of changes in each physical state in the time-series state chain and the cumulative trend of changes, and output the corresponding evolution stage identification result.

9. The intelligent video-based behavioral feature analysis and recognition system according to claim 8, characterized in that, The continuity of change is determined based on the persistence of each physical state in the time-series state chain across multiple consecutive acquisition times. The cumulative trend of the changes is determined based on the frame-by-frame changes in the centroid offset, joint angle changes, and centroid displacement of the active party in the time-series state chain.

10. The intelligent video-based behavioral feature analysis and recognition system according to claim 9, characterized in that, The evolutionary stages include at least a normal stage, an early evolutionary stage, a middle evolutionary stage, and a completion evolutionary stage, wherein... Based on the centroid offset trend determination result of no offset or offset direction not continuously stable, and the joint linkage state determination result of non-linkage state, it is identified as a normal stage. Based on the fact that the centroid offset trend maintains a stable change continuity across multiple consecutive acquisition times and the centroid offset shows an increasing cumulative trend, the joint linkage state has not yet been activated, and it is identified as an early stage of evolution. Based on the fact that the centroid offset trend maintains a stable and continuous change across multiple consecutive acquisition times, and the joint linkage state is activated and remains stable, while the change in the angle of each joint shows an increasing cumulative trend, it is identified as the mid-stage of evolution. Based on the fact that the center of the hip extends beyond the vertical support range boundary of the center of the ankle in the horizontal direction, and at least one key point of the supporting foot has been raised compared to the start time, it is identified as the evolution completion stage; The vertical support range is the vertical projection range of the area enclosed by the contact points between the human body's two feet and the support surface on the horizontal plane. The start time is the moment when the centroid offset trend is first determined to be a valid acquisition time.

Images