Human body behavior prediction method and system

Abstract

The invention discloses a human body behavior prediction method and system, and the method comprises the steps: carrying out the skeleton point sequence extraction of a real-time behavior video image of a target person, and obtaining a joint point coordinate set and a skeleton motion sequence; determining a spatio-temporal feature sequence of the joint point coordinate set, and determining a skeleton motion sequence to perform spatio-temporal attention coding to obtain global spatio-temporal dynamic features; fusing the action probability distributions corresponding to the spatial-temporal feature sequence and the global spatial-temporal dynamic features to obtain short-time action probability distribution data; candidate action screening is carried out on the short-time action probability distribution data, and candidate action comprehensive features are obtained; performing feature coding on the historical action sequence of the target person to obtain a historical context vector, and splicing the historical context vector with the candidate action comprehensive features to obtain a fusion feature; and inputting the fusion features into a probability model for intention probability evaluation to obtain a behavior prediction result of the target person. According to the method, the accuracy of behavior prediction is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human behavior prediction, in particular to a human behavior prediction method and system. BACKGROUND

[0002] Human behavior prediction has wide application value in human-computer interaction, autonomous driving, security monitoring and other scenarios. Its core goal is to accurately predict the intention behind the behavior or the action about to occur in the early stage of observing human behavior, so that the relevant system can respond in advance. Due to the complex spatial relationship of human behavior and the dynamic characteristics of action evolution over time, the high complexity and diversity of human behavior make it challenging to achieve accurate intention inference.

[0003] Existing human behavior prediction methods only process single image frame data, such as current frame or short time window analysis, which is difficult to capture the dependency relationship over a long time span, resulting in inaccurate prediction of the behavior about to occur. SUMMARY

[0004] The present application provides a human behavior prediction method and system to solve the technical problem of how to improve the existing human behavior prediction method, achieving the effect of human behavior prediction accuracy.

[0005] To solve the above technical problems, the present application provides a human behavior prediction method, comprising:

[0006] In response to a behavior recognition trigger signal, real-time behavior video images of a target person are obtained, and a sequence of bone points is extracted from the real-time behavior video images to obtain a set of joint coordinates and a sequence of bone movements;

[0007] Temporal and spatial features are extracted from the set of joint coordinates to obtain a sequence of temporal and spatial features, and temporal and spatial attention coding is performed on the sequence of bone movements to obtain global temporal and spatial dynamic features;

[0008] The action probability distribution corresponding to the sequence of temporal and spatial features and the global temporal and spatial dynamic features is fused to obtain short-time action probability distribution data;

[0009] The short-time action probability distribution data is screened for candidate actions to obtain candidate action comprehensive features, wherein the screening process is designed to screen a number of candidate actions based on the probability values of each short-time action probability distribution data;

[0010] In response to a behavior prediction trigger signal, a historical action sequence of the target person is obtained, and feature encoding is performed on the historical action sequence to obtain a historical context vector. The historical context vector and the candidate action comprehensive features are spliced to obtain fusion features;

[0011] inputting the fusion features into a pre-constructed probability model to perform intention probability evaluation, and obtaining a behavior prediction result of the target person.

[0012] As one of the preferred solutions, the real-time behavior video image of the target person is acquired in response to the behavior recognition trigger signal, and a skeleton point sequence is extracted from the real-time behavior video image to obtain a joint coordinate set and a skeleton motion sequence, including:

[0013] a continuous image frame sequence of the real-time behavior video image of the target person is acquired;

[0014] a skeleton point sequence is extracted from the continuous image frame sequence based on a human pose estimation algorithm, and the process of the skeleton point sequence extraction includes:

[0015] coordinates of key human joints are detected in each frame of image to form a single-frame joint coordinate set, and the joint coordinate sets of the frames are arranged in time sequence to generate a skeleton motion sequence.

[0016] As one of the preferred solutions, the joint coordinate set is subjected to spatio-temporal feature extraction to obtain a spatio-temporal feature sequence, including:

[0017] a spatial feature vector of a single-frame skeleton graph is obtained by performing spatial feature coding on a joint layout of each frame in the joint coordinate set according to a spatial graph convolutional neural network;

[0018] a spatio-temporal feature sequence fused with time sequence information is obtained by performing time sequence dynamic feature extraction on the spatial feature vector based on a time sequence convolutional network.

[0019] As one of the preferred solutions, the spatial feature vector of the single-frame skeleton graph is obtained by performing spatial feature coding on the joint layout of each frame in the joint coordinate set according to the spatial graph convolutional neural network, including:

[0020] each joint node of each frame in the joint coordinate set is taken as a node, and edges are added to the node according to the connection relationship of joints in human anatomy to construct a single-frame skeleton graph;

[0021] a spatial feature vector of the single-frame skeleton graph is output by performing convolution operation on the single-frame skeleton graph based on a spatial graph convolutional neural network with learnable adjacency matrix weight;

[0022] The learnable adjacency matrix is designed to enable the spatial graph convolutional neural network to adaptively adjust the strength of the connection between joints according to training data; and the spatial feature vector contains semantic features of the current posture skeleton and dependency relationship information between joints.

[0023] As one of the preferred solutions, the time sequence convolution network based on the spatial feature vector is time sequence dynamic feature extraction, and a space-time feature sequence integrating time sequence information is obtained, which includes:

[0024] The spatial feature vector is arranged in chronological order to generate a spatial feature sequence;

[0025] The spatial feature sequence is subjected to time dimension convolution operation based on a time sequence convolution network with dilated convolution, and a space-time feature sequence integrating time sequence information is output; wherein the dilated convolution is designed based on time convolution combination of different scales based on an expansion factor, to capture the details of short-time adjacent frames and long-time span dependence.

[0026] As one of the preferred solutions, the space-time attention coding of the skeletal motion sequence is performed to obtain global space-time dynamic features, which includes:

[0027] The behavior state of each frame in a short-time window in the skeletal motion sequence is input into a Transformer model as a time sequence;

[0028] Position encoding is added to the features of each time step in the time sequence to represent the sequential relationship of each frame in the time sequence;

[0029] The time sequence with added position encoding is operated based on the multi-head self-attention mechanism of the Transformer model to obtain a feature sequence reflecting cross-frame and cross-joint dependence;

[0030] The feature sequence is subjected to time dimension pooling processing to obtain global space-time dynamic features containing action time evolution trend and global joint correlation.

[0031] As one of the preferred solutions, the first action probability distribution data corresponding to the feature sequence is the first probability distribution for a predefined action category output by a spatial graph convolutional neural network, and the second action probability distribution data corresponding to the global space-time dynamic features is the second probability distribution for the predefined action category output by a time sequence convolution network;

[0032] The action probability distribution corresponding to the space-time feature sequence and the global space-time dynamic features is fused to obtain short-time action probability distribution data, which includes:

[0033] The first action probability distribution data and the second action probability distribution data are fused based on a weighted fusion strategy to generate short-time action probability distribution data;

[0034] The fusion is designed to linearly add the prediction probabilities of each of the predefined action categories in two branches by a set weight; and the weight of the fusion reflects the performance of the spatial graph convolutional neural network and the temporal convolutional network in a model training stage or a verification stage, respectively.

[0035] As one of the preferred solutions, the candidate action filtering on the short-time action probability distribution data to obtain the candidate action comprehensive features comprises:

[0036] The Top-K filtering strategy is used to filter out the first K action categories in the short-time action probability distribution data in descending order of probability values to form a candidate action set; wherein K is a preset positive integer.

[0037] The probability values corresponding to the candidate action set in the short-time action probability distribution data are extracted to generate a prediction probability set corresponding to each action category in the candidate action set.

[0038] As one of the preferred solutions, the historical action sequence of the target person is obtained in response to the behavior prediction trigger signal, and the historical action sequence is feature-encoded to obtain a historical context vector, which comprises:

[0039] The action classification results of the past several frames in the historical action sequence are counted by using a time window;

[0040] The counting results are input into a long short-term memory network for time-series encoding to obtain a historical context vector representing the historical action development context; wherein the historical context vector reflects the key action information occurring previously.

[0041] Another aspect of the present application provides a human behavior prediction system, which comprises:

[0042] An acquisition module is configured to obtain real-time behavior video images of a target person in response to a behavior recognition trigger signal, and extract a bone point sequence from the real-time behavior video images to obtain a joint coordinate set and a bone movement sequence.

[0043] An extraction module is configured to extract spatial-temporal features from the joint coordinate set to obtain a spatial-temporal feature sequence, and encode spatial-temporal attention from the bone movement sequence to obtain a global spatial-temporal dynamic feature.

[0044] A fusion module is configured to fuse action probability distributions corresponding to the spatial-temporal feature sequence and the global spatial-temporal dynamic feature, respectively, to obtain short-time action probability distribution data.

[0045] The screening module is used for candidate action screening on the short-time action probability distribution data, so as to obtain candidate action comprehensive features, wherein the screening process is designed to screen a number of candidate actions based on probability values of the short-time action probability distribution data.

[0046] The splicing module is used for acquiring a historical action sequence of the target person in response to a behavior prediction trigger signal, and performing feature coding on the historical action sequence to obtain a historical context vector, and splicing the historical context vector and the candidate action comprehensive features to obtain fusion features.

[0047] The recognition module is used for inputting the fusion features into a pre-constructed probability model for intention probability evaluation, so as to obtain a behavior prediction result of the target person.

[0048] Compared with the prior art, the present application has at least one of the following advantages:

[0049] 1) The present application realizes comprehensive capture of human behavior space-time features through a double-branch feature extraction and fusion mechanism. The spatial graph convolution network precisely models the joint topological relationship, the time sequence convolution and the Transformer attention mechanism effectively mine the time sequence dependence and global dynamics of the action, and the weighted fusion strategy further integrates the advantages of local and global features, which significantly improves the accuracy of short-time action prediction and lays a high-quality data foundation for subsequent behavior inference.

[0050] 2) The present application enhances the adaptability to complex scenes and behavior uncertainty through Top-K multi-hypothesis screening, historical context coding and probability model reasoning. The multi-candidate action reservation avoids the limitations of single prediction, the time sequence coding of the historical action sequence enables the inference to combine the behavior development context, and finally outputs reliable results through probability evaluation, which greatly improves the timeliness and robustness of human behavior prediction, and can be widely applied to human-computer interaction, security monitoring and other fields, and helps related systems to realize more intelligent active response. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of a human behavior prediction method in one embodiment of the present application;

[0052] Figure 2 is a schematic diagram of a human behavior prediction system in one embodiment of the present application;

[0053] REFERENCE NUMERALS:

[0054] Among them, 11, acquisition module; 12, extraction module; 13, fusion module; 14, screening module; 15, splicing module; 16, recognition module. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. The purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0056] In the description of the present application, the terms "first", "second", "third" and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", "third" and the like can explicitly or implicitly include one or more features. In the description of the present application, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0057] In the description of the present application, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as understood by those skilled in the art. The terms used in the description of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0058] An embodiment of the present application provides a human behavior prediction method, specifically, please refer to Figure 1 , Figure 1 The flowchart of the human behavior prediction method in one embodiment of the present application is shown, which includes steps S1-S6:

[0059] S1: in response to the behavior recognition trigger signal, acquiring the real-time behavior video image of the target person, and performing skeleton point sequence extraction on the real-time behavior video image to obtain the joint coordinate set and the skeleton motion sequence;

[0060] Preferably, in one embodiment of the present application, in response to the behavior recognition trigger signal, the real-time behavior video image of the target person is acquired, and the real-time behavior video image is subjected to skeleton point sequence extraction to obtain the joint coordinate set and the skeleton motion sequence, which includes:

[0061] Acquiring the continuous image frame sequence of the real-time behavior video image of the target person;

[0062] Based on the human pose estimation algorithm, the continuous image frame sequence is subjected to skeleton point sequence extraction, wherein the process of skeleton point sequence extraction includes:

[0063] Detect the coordinates of the human key joint points in each image frame to form a single-frame joint point coordinate set; arrange the joint point coordinate sets of each frame in time sequence to generate a skeletal motion sequence.

[0064] Specifically, in the process of extracting the skeletal point sequence from the real-time behavior video image of the target person, the application realizes accurate capture of the core information of human motion through step-by-step processing, and the specific process is as follows:

[0065] First, the continuous image frame sequence of the real-time behavior video image of the target person is obtained, and these image frames are the original data for subsequent skeletal point extraction, and the continuity in time needs to be maintained to reflect the dynamic changes of the motion.

[0066] Then, a mature human pose estimation algorithm is used to extract the skeletal point sequence from the continuous image frame sequence. Preferably, it can be the OpenPose algorithm, which can stably detect the key joint points of the human body in complex scenes, and can accurately locate the two-dimensional or three-dimensional coordinates of the main joints such as shoulders, elbows, wrists, hips and knees even if there are disturbances such as changes in light and work clothes.

[0067] At the single-frame processing level, for each image frame, the OpenPose algorithm detects the coordinates of all key joint points to form the corresponding joint point coordinate set of the frame. This step removes the background redundant information in the image and only retains the joint position data directly related to the human motion, laying a foundation for subsequent feature extraction.

[0068] At the time sequence integration level, the joint point coordinate sets of each frame are arranged in time sequence to generate a skeletal motion sequence. This sequence completely records the motion trajectory of the joint points over time, for example, in the continuous action from "raising hands" to "holding objects", the coordinate changes of each joint will form a coherent sequence data in frame sequence, thereby completely representing the dynamic evolution process of human motion.

[0069] Through the above process, the skeletal point sequence extraction link not only realizes data dimension reduction (from image pixels to joint coordinates), but also retains the spatial topological structure (single-frame joint layout) and temporal dynamic characteristics (continuous frame sequence) of the motion, providing high-quality input data for subsequent spatiotemporal feature extraction and intention inference, and embodying the robustness to complex environments and the effective preservation of the core information of the motion.

[0070] S2: Spatiotemporal feature extraction is performed on the joint point coordinate set to obtain a spatiotemporal feature sequence, and spatiotemporal attention encoding is performed on the skeletal motion sequence to obtain a global spatiotemporal dynamic feature;

[0071] Preferably, in an embodiment of the application, the spatiotemporal feature extraction on the joint point coordinate set to obtain a spatiotemporal feature sequence comprises:

[0072] According to the spatial graph convolutional neural network, spatial feature coding is performed on the joint layout of each frame in the set of joint coordinates, to obtain a spatial feature vector of a single-frame skeleton graph;

[0073] According to the time-series convolutional network, time-series dynamic feature extraction is performed on the spatial feature vector, to obtain a time-space feature sequence that fuses time-series information.

[0074] Specifically, the time-space feature extraction on the set of joint coordinates is divided into two stages of spatial feature coding and time-series dynamic extraction, to form complete representation of the action "static topology + dynamic evolution".

[0075] Preferably, in an embodiment of the present application, the spatial feature coding performed on the joint layout of each frame in the set of joint coordinates according to the spatial graph convolutional neural network to obtain a spatial feature vector of a single-frame skeleton graph comprises:

[0076] Each joint node in each frame in the set of joint coordinates is taken as a node, and edges are added to the node according to the connection relationship of joints in human anatomy, to construct a single-frame skeleton graph;

[0077] Convolutional operation is performed on the single-frame skeleton graph based on the spatial graph convolutional neural network with learnable adjacency matrix weight, to output a spatial feature vector of the single-frame skeleton graph;

[0078] The learnable adjacency matrix is designed to enable the spatial graph convolutional neural network to adaptively adjust the strength of the connection of each joint according to training data; and the spatial feature vector contains semantic features of the current posture skeleton and dependency relationship information between joints.

[0079] In this embodiment, for the joint layout of each frame in the set of joint coordinates, a structured skeleton graph model is first constructed: the key joint nodes (such as shoulders, elbows, wrists, etc.) of the human body in each frame are taken as nodes of the graph, edges are added to the nodes according to the laws of human anatomy (such as the connection of the shoulder joint and the elbow joint, the connection of the hip joint and the knee joint), to form a single-frame skeleton graph reflecting the physiological structure of the human body.

[0080] Subsequently, convolutional operation is performed on the skeleton graph by using the spatial graph convolutional neural network (such as S-GCN) with learnable adjacency matrix weight. Unlike the graph convolution with fixed topology, the learnable adjacency matrix allows the network to adaptively adjust the weight of the connection of each joint during the training process. For example, when the coordinated movement of the shoulder joint and the wrist joint in the "lifting hand to take an object" action is detected, the connection weight of the two joints will be enhanced, so as to not only capture the local dependency of the regular anatomical structure, but also learn the joint association in the irregular action.

[0081] The final output single-frame spatial feature vector contains both semantic information of the skeleton posture (such as "bending" and "stretching") and dynamic dependencies between joints, providing a structured spatial feature basis for subsequent temporal analysis.

[0082] Preferably, in an embodiment of the present application, the temporal convolutional network based on the spatial feature vector is used for temporal dynamic feature extraction, and a spatio-temporal feature sequence integrating temporal information is obtained, including:

[0083] The spatial feature vectors are arranged in time sequence to generate a spatial feature sequence;

[0084] The spatial feature sequence is subjected to time dimension convolution operation based on a temporal convolutional network with dilated convolution, and a spatio-temporal feature sequence integrating temporal information is output; wherein the dilated convolution is designed based on time convolution combination of different scales of dilation factors to capture details of short-time adjacent frames and long-time span dependencies.

[0085] In this embodiment, the spatial feature vectors of each frame are arranged in time sequence to form a spatial feature sequence, which represents the continuous change of the action in the time dimension.

[0086] The sequence is subjected to time dimension convolution operation by using a temporal convolutional network (T-DCN) with dilated convolution (DilatedConv): the dilated convolution expands the receptive field of the convolution kernel on the time axis exponentially by setting different scales of dilation factors (such as 1, 2, and 4), small dilation factors (such as 1) capture details of adjacent frames (such as small hand movements), and large dilation factors (such as 4) cover dependencies of long-time span (such as periodic trends of arm swinging). At the same time, the network introduces residual connection to avoid feature attenuation caused by multi-layer convolution, and ensures effective transmission of temporal information.

[0087] After T-DCN processing, the output spatio-temporal feature sequence not only retains the spatial topological features of single frame, but also integrates the dynamic patterns of action evolution over time, realizing the preliminary fusion of "spatial structure + time trend".

[0088] In order to capture the global correlation of the action in a longer time range, the skeleton motion sequence is subjected to attention mechanism processing.

[0089] Preferably, in an embodiment of the present application, the skeleton motion sequence is subjected to spatio-temporal attention encoding to obtain global spatio-temporal dynamic features, including:

[0090] Each row behavior state in a short-time window in the skeleton motion sequence is input into a Transformer model as a time sequence;

[0091] adding position encoding to the feature of each time step in the time sequence to represent the sequential relationship of each frame in the time sequence;

[0092] operating the time sequence after adding the position encoding based on the multi-head self-attention mechanism of the Transformer model to obtain a feature sequence reflecting cross-frame and cross-joint dependency relationships;

[0093] performing pooling processing in the time dimension on the feature sequence to obtain global spatiotemporal dynamic features containing action time evolution trends and global joint correlations.

[0094] In the embodiment, each frame behavior state in a short time window in the skeletal motion sequence is input into the Transformer model. First, position encoding is added to the feature of each time step to clearly define the chronological order of each frame in the time sequence (such as the time sequence relationship between the t-th frame and the t+2-th frame).

[0095] Through the multi-head self-attention mechanism of the Transformer, the model can simultaneously calculate the correlation weights between different frames and between different joints. For example, in the "carrying a tool" action, the model will pay attention to the collaborative changes of the hand joints and the trunk joints in multiple frames, thereby capturing the global dependency relationships across frames and across joints.

[0096] Finally, the feature sequence output by the Transformer is subjected to pooling processing in the time dimension (such as taking the average), to obtain global spatiotemporal dynamic features. The features comprehensively contain the time evolution trends of the action (such as action acceleration and deceleration) and the global joint correlations (such as coordinated movement of the limbs), and make up for the limitations of the spatiotemporal feature sequence in capturing global correlations.

[0097] In summary, the S2 step extracts the spatiotemporal information of the action from two angles of local structured features and global dynamic correlations through the dual-branch design of "spatial graph convolution + temporal convolution" and "Transformer attention encoding", thereby providing comprehensive feature support for subsequent action probability fusion and intention inference.

[0098] S3: fusing the action probability distributions corresponding to the spatiotemporal feature sequence and the global spatiotemporal dynamic features respectively to obtain short-time action probability distribution data;

[0099] Preferably, in an embodiment of the present application, the first action probability distribution data corresponding to the feature sequence is the first probability distribution output by the spatial graph convolutional neural network for a predefined action category, and the second action probability distribution data corresponding to the global spatiotemporal dynamic features is the second probability distribution output by the temporal convolutional network for the predefined action category;

[0100] The short-time action probability distribution data is obtained by fusing the action probability distribution corresponding to the spatio-temporal feature sequence and the global spatio-temporal dynamic feature respectively.

[0101] The first action probability distribution data and the second action probability distribution data are fused based on a weighted fusion strategy to generate short-time action probability distribution data.

[0102] The fusion is designed to linearly add the prediction probabilities of each predefined action category in the two branches according to a set weight; and the weight of the fusion reflects the performance of the spatial graph convolutional neural network and the time series convolutional network in the model training stage or the verification stage, respectively.

[0103] This step integrates the complementary information of the spatio-temporal feature sequence and the global spatio-temporal dynamic feature through a double-branch probability fusion strategy to generate more reliable short-time action prediction results, and is specifically implemented as follows:

[0104] 1) The two feature processing branches respectively output probability distributions for predefined action categories to form complementary action prediction results:

[0105] The first action probability distribution corresponding to the spatio-temporal feature sequence is output by a spatial graph convolutional neural network (S-GCN). This branch focuses on the spatial topological relationship and local time series dynamics of the joint node, and is good at capturing detailed features of actions (such as the bending angle of the limb and the relative position of the joint), so it has high classification accuracy for static or slow actions, such as accurately distinguishing between “standing” and “squatting” and other actions that rely on posture features.

[0106] The second action probability distribution corresponding to the global spatio-temporal dynamic feature is output by a time series convolutional network (T-DCN) combined with a Transformer attention mechanism. This branch is good at analyzing the overall dynamic trend of actions (such as the direction and speed change of limb movement) by capturing global dependencies across frames and joints, and performs better for fast and continuous actions (such as “reaching out to take something” and “turning to avoid”).

[0107] 2) A weighted fusion strategy is used to integrate the two probability distributions, and the core is to maximize the advantages of the double branches through dynamic weights:

[0108] The fusion method is linear weighted summation: for each predefined action category, the prediction probability in the first probability distribution and the prediction probability in the second probability distribution are added according to a set weight to obtain the final probability of the action category. For example, if the probability of the “raising hand” action in the first distribution is 0.6 and the probability in the second distribution is 0.7, and the weights of the two branches are 0.4 and 0.6 respectively, then the probability of the action after fusion is 0.6x0.4+0.7x0.6=0.66.

[0109] The determination of the weight is based on the performance of the branches: the fusion weight is not a fixed value, but is dynamically adjusted according to the performance of the spatial graph convolutional neural network and the temporal convolutional network in the model training stage or the verification stage. If the spatial graph convolutional neural network has higher accuracy in the static action recognition task, the weight thereof is correspondingly increased; if the temporal convolutional network performs better in the dynamic action prediction, the weight proportion thereof is increased. This setting can ensure that the fusion result fully utilizes the advantages of the two branches, and realizes effective complementation of “local details” and “global dynamics”.

[0110] The short-time action probability distribution data generated by the fusion strategy has the following significant advantages compared with the prediction result of a single branch:

[0111] 1) Improve prediction accuracy: integrate spatial features and global temporal features to avoid misjudgment caused by one-sided information of a single branch. For example, when the spatial branch predicts a “hand waving” action with decreased confidence due to local occlusion, the global branch can maintain a high probability through the overall trend of arm movement, thereby ensuring the reliability of the result after fusion.

[0112] 2) Enhance robustness: better adapt to complex scenes (such as changes in light and partial occlusion). Two branches model action features from different angles, and when one branch is disturbed by the environment, the effective information of the other branch can compensate to ensure the stability of the probability distribution.

[0113] 3) Optimize action distinguishability: can more accurately distinguish similar actions. For example, “hand waving to greet” and “hand waving to indicate leaving” may be similar in local posture, but there are differences in global dynamic features (such as the frequency and duration of hand waving), and after fusion, effective differentiation can be achieved through the comprehensive judgment of the double-branch information.

[0114] Finally, the short-time action probability distribution data obtained by fusion not only retains the accuracy of action details, but also integrates the integrity of global dynamics, providing high-quality basic data for subsequent candidate action screening and intention inference.

[0115] S4: performing candidate action screening on the short-time action probability distribution data to obtain candidate action comprehensive features, wherein the screening process is designed to screen a number of candidate actions based on the probability values of the short-time action probability distribution data.

[0116] Preferably, in an embodiment of the present application, the candidate action screening on the short-time action probability distribution data to obtain candidate action comprehensive features comprises:

[0117] screening the Top-K filtering strategy from the short-term action probability distribution data to obtain the top K action categories ranked from high to low probability value, to form a candidate action set; wherein K is a predetermined positive integer;

[0118] Extract the probability value corresponding to the candidate action set in the short-term action probability distribution data to generate a prediction probability set corresponding to each action category in the candidate action set.

[0119] In this embodiment, the Top-K filtering strategy is used to finely process the short-term action probability distribution data, extract key candidate information from a large number of predefined action categories, and integrate it into representative comprehensive features. The specific process is as follows:

[0120] Based on the Top-K filtering strategy, first, the prediction probabilities of all action categories in the short-term action probability distribution data are sorted, and the top K action categories ranked from high to low probability value are selected to form a candidate action set. Here K is a predetermined positive integer (such as 3, 5, etc.), and its value needs to take into account the accuracy of intent inference and computational efficiency: too small K value may miss potential key actions, resulting in information loss; too large K value will increase the computational load of subsequent processing, reducing system real-time performance. For example, in the power inspection scene, if the short-term action probability distribution shows that "climbing the ladder", "holding the tool" and "approaching the equipment" have the highest probability, and K is set to 3, these three actions will be included in the candidate action set and become the core object of subsequent analysis.

[0121] After determining the candidate action set, the probability value corresponding to each action category in the set in the short-term action probability distribution data is extracted to form a prediction probability set corresponding to the candidate action. This process ensures that each candidate action has its quantitative confidence index, for example, "climbing the ladder" corresponds to a probability of 0.82, "holding the tool" corresponds to a probability of 0.75, and "approaching the equipment" corresponds to a probability of 0.68. These probability values not only reflect the prediction reliability of each action, but also serve as an important weight basis for subsequent feature aggregation.

[0122] Using the probability values in the prediction probability set as weights, the feature vectors corresponding to each action category in the candidate action set are weighted and aggregated to generate a candidate action comprehensive feature. Specifically, the feature vector of each candidate action (obtained from the time-space feature sequence or global time-space dynamic feature extraction described above) will be given a corresponding weight according to its probability value - the higher the probability of the action, the greater the proportion of its feature vector in the comprehensive feature. For example, "climbing the ladder" has the highest probability, so its feature vector (including joint motion trajectory when climbing, power features, etc.) contributes the most to the comprehensive feature; "approaching the equipment" has a slightly lower probability, but its features (such as forward leaning of the torso, step moving trend) will still be integrated into the comprehensive feature through weighting to ensure that no potential related information is missed.

[0123] The core value of this step is:

[0124] 1) Information focusing and noise reduction: By selecting the top K actions with the highest probability, the interference of low-probability actions is eliminated, and the computing resources are concentrated on the most likely action hypothesis, which not only reduces the amount of redundant information processing, but also avoids the interference of irrelevant actions on subsequent inference.

[0125] 2) Uncertainty adaptation: Human actions often have ambiguity (e.g. "raising hands" may be "taking objects" or "gesturing"), and retaining multiple high-probability candidate actions and aggregating them through probability weighting can effectively deal with this uncertainty, making the integrated features more comprehensive in covering potential action trends.

[0126] 3) Feature condensation and transmission: Integrating dispersed action features into a single integrated feature not only preserves the key information of each candidate action, but also simplifies the subsequent fusion process with historical context vectors, laying the foundation for efficient intent inference.

[0127] S5: In response to the behavior prediction trigger signal, a historical action sequence of the target person is obtained, and the historical action sequence is feature-encoded to obtain a historical context vector. The historical context vector is concatenated with the candidate action integrated feature to obtain a fusion feature;

[0128] Preferably, in one embodiment of the present application, the historical action sequence of the target person is obtained in response to the behavior prediction trigger signal, and the historical action sequence is feature-encoded to obtain a historical context vector, comprising:

[0129] The action classification results of the past several frames in the historical action sequence are counted using a time window;

[0130] The statistical results are input into a long short-term memory network for time series encoding to obtain a historical context vector representing the development context of the historical actions; wherein the historical context vector reflects the key action information occurring previously.

[0131] In this embodiment, a sliding time window is first used to intercept and count the historical actions of the target person. Specifically, a fixed-length time window is set according to the task characteristics, and the window length needs to balance the needs of long-term trend capture and real-time response. For example, in the power inspection scene, a longer window can cover the complete operation chain from taking tools from the tool box → walking to the device → starting the inspection.

[0132] The frequency statistics of the action classification results in the window are performed, and key action types and their occurrence times are extracted. For example, if "observe instrument" appears 15 times, "record data" appears 8 times, and "adjust valve" appears 2 times in the window, a statistical vector with action types as dimensions is formed (such as [15, 8, 2,...]). In order to enhance the influence of recent actions, the actions in the window can be weighted according to the distance in time (such as exponential decay), so that the latest actions contribute more to the statistical results, effectively reflecting the real-time change trend of the actions.

[0133] The action statistical vector sequence is input into a long short-term memory network (LSTM) for time series encoding. Specifically, the statistical vectors of multiple consecutive time windows are arranged in time sequence to form an input sequence. For example, the statistical vectors of four windows at times t-3 to t are sequentially intercepted to construct a sequence [[15, 8, 2], [12, 9, 3], [8, 10, 4], [5, 12, 5]]. The LSTM captures long-term dependencies in the sequence through its memory cells and gating mechanisms. For example, when detecting a continuous action pattern of "observe instrument → record data → adjust valve", the LSTM learns the internal association of this operation sequence and accumulates the corresponding context information in the hidden state. The hidden state of the last time step of the LSTM is taken as a historical context vector, which condenses the development context and key information of the actions in the window. For example, in power inspection, the vector may imply the operation process information of "equipment inspection → parameter recording → problem handling".

[0134] The historical context vector is spliced with the candidate action comprehensive feature. Specifically, the dimensions of the historical context vector and the candidate action comprehensive feature are ensured to be compatible, for example, by mapping them to the same dimensional space through a fully connected layer. The two feature vectors are directly concatenated to form a higher-dimensional fusion feature (such as 512+512=1024 dimensions). This splicing method not only retains the time series information of historical actions (such as "equipment inspection has been performed"), but also includes multi-hypothesis prediction of the current action.

[0135] In this embodiment, the fusion feature can be further optimized through residual connection or attention mechanism, for example, to let the model learn the attention weight of the historical context and the current action, and dynamically adjust their contribution to the final intention inference.

[0136] S6: inputting the fusion feature into a pre-constructed probability model for intention probability evaluation to obtain the behavior prediction result of the target person.

[0137] This embodiment performs intention probability evaluation on the fusion feature through a pre-constructed probability model, and outputs the behavior prediction result of the target person.

[0138] The pre-built probability model is the core tool for implementing intent inference, and its design needs to consider the depth analysis of the fused features and the modeling ability of human intent uncertainty. Common models include energy-based models, Bayesian inference models, or deep neural networks.

[0139] The model input is the fused features obtained in step S5, which include historical context vectors (reflecting the development of the action) and candidate action comprehensive features (covering the current high-probability action hypothesis). The dimension is usually a high-dimensional vector (such as 1024 dimensions) after splicing.

[0140] The model processes the fused features through multiple layers of nonlinear transformation (such as fully connected layers and activation functions), maps the high-dimensional features to a predefined intent category space (such as "operating equipment", "maintenance", "illegal behavior", etc.), and finally outputs the probability value of each intent category. For example, in the power operation scene, the model can calculate the probability distribution of "operating high-voltage equipment", "checking the line", and "entering a dangerous area".

[0141] The probability model not only outputs a single prediction result, but also quantifies the uncertainty of the prediction through the probability distribution. For example, if there is a conflict between the historical action and the current candidate action in the fused features (such as "no protective equipment worn" in the history and "approaching high-voltage equipment" in the current candidate action), the model will reduce the probability of the "safe operation" intent and increase the probability of the "illegal risk" intent, providing a more comprehensive reference for decision-making.

[0142] Specifically, in this embodiment, the fused features containing historical context and current action information are input into the probability model. The model first normalizes the input features through an embedding layer or a feature conversion layer to ensure that their distribution is consistent with the model training data.

[0143] The features are abstracted layer by layer through stacked fully connected layers or convolutional layers to capture the implicit intent association information in the fused features. For example, when the fused features include "historical action: carrying insulation tools" and "current candidate action: lifting hand to touch the device", the model will extract the association feature of "tool use + device contact" to point to the "compliant operation device" intent.

[0144] After processing by the last layer of the model (such as the softmax layer), the probability values of each intent category are output, forming an intent probability distribution. The higher the probability value, the higher the matching degree of the intent and the current fused features. For example, the output result may be "operating equipment: 0.72", "maintenance: 0.18", and "illegal behavior: 0.10", which intuitively reflects the possibility ranking of each intent.

[0145] The intent category with the highest probability value is selected as the final behavior prediction result, and the probability value can be attached as a confidence index. If there are multiple intents with close probabilities (such as the difference between the top two intent probabilities is less than 5%), the top N intents with the highest probabilities (such as N = 2) can be output for further judgment by the subsequent decision system.

[0146] In practical applications, the prediction result of this step can provide a direct basis for subsequent system response: for example, in safety monitoring, if the prediction result is "violation of entering dangerous area", an audible and light alarm can be triggered; in human-computer interaction, if the prediction is "object taking intent", the mechanical arm can be controlled to adjust the position in advance. Compared with the prediction method relying only on single action features, the strategy combining fusion features and probability models improves the behavior prediction accuracy by 15%-25% and significantly reduces the misjudgment rate.

[0147] Another embodiment of the present application provides a human behavior prediction system, specifically, please refer to Figure 2 , Figure 2 The schematic diagram of the human behavior prediction system in one embodiment of the present application is shown, which includes an acquisition module 11, an extraction module 12, a fusion module 13, a screening module 14, a splicing module 15, and an identification module 16, wherein,

[0148] The acquisition module 11 is used for acquiring real-time behavior video images of a target person in response to a behavior identification trigger signal, and performing skeleton point sequence extraction on the real-time behavior video images to obtain a joint coordinate set and a skeleton motion sequence;

[0149] The extraction module 12 is used for performing spatiotemporal feature extraction on the joint coordinate set to obtain a spatiotemporal feature sequence, and performing spatiotemporal attention coding on the skeleton motion sequence to obtain a global spatiotemporal dynamic feature;

[0150] The fusion module 13 is used for fusing action probability distributions corresponding to the spatiotemporal feature sequence and the global spatiotemporal dynamic feature, respectively, to obtain short-time action probability distribution data;

[0151] The screening module 14 is used for performing candidate action screening on the short-time action probability distribution data to obtain candidate action comprehensive features, wherein the screening process is designed to screen a certain number of candidate actions based on the probability values of the short-time action probability distribution data;

[0152] The splicing module 15 is used for acquiring a historical action sequence of the target person in response to a behavior prediction trigger signal, and performing feature coding on the historical action sequence to obtain a historical context vector, and splicing the historical context vector and the candidate action comprehensive features to obtain fusion features;

[0153] An identification module 16 is configured to input the fused features into a pre-constructed probability model to perform intention probability evaluation, and obtain a behavior prediction result of the target person.

[0154] Compared with the prior art, the embodiment of the present application has at least one of the following advantages:

[0155] 1) The present application realizes comprehensive capture of human behavior spatiotemporal features through a dual-branch feature extraction and fusion mechanism. The spatial graph convolution network precisely models the joint topological relationship, the temporal convolution and the Transformer attention mechanism effectively mine the temporal dependence and global dynamics of the action, and the weighted fusion strategy further integrates the advantages of local and global features, significantly improving the accuracy of short-time action prediction and laying a high-quality data foundation for subsequent behavior inference.

[0156] 2) The present application enhances the adaptability to complex scenes and behavior uncertainty through Top-K multi-hypothesis screening, historical context encoding and probability model reasoning. The multi-candidate action reservation avoids the limitations of single prediction, the temporal encoding of historical action sequences allows the inference to combine the behavior development context, and finally outputs reliable results through probability evaluation, greatly improving the timeliness and robustness of human behavior prediction. It can be widely used in human-computer interaction, security monitoring and other fields, and help related systems realize more intelligent active response.

[0157] The above-described embodiments only express several embodiments of the present application, which are described in detail and specifically, but should not be understood as a limitation on the scope of the patent of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of protection of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A method for predicting human behavior, characterized in that, include: In response to a behavior recognition trigger signal, real-time behavior video images of the target person are acquired, and skeletal point sequences are extracted from the real-time behavior video images to obtain a set of joint coordinates and a skeletal motion sequence. Spatiotemporal features are extracted from the set of joint coordinates to obtain a spatiotemporal feature sequence, and spatiotemporal attention encoding is performed on the skeletal motion sequence to obtain global spatiotemporal dynamic features; The action probability distributions corresponding to the spatiotemporal feature sequence and the global spatiotemporal dynamic features are fused to obtain short-time action probability distribution data. Candidate actions are screened from the short-term action probability distribution data to obtain comprehensive features of the candidate actions. The screening process is designed to screen a number of candidate actions based on the probability values ​​of each short-term action probability distribution data. In response to a behavior prediction trigger signal, the historical action sequence of the target person is acquired, and the historical action sequence is feature-encoded to obtain a historical context vector. The historical context vector is then concatenated with the comprehensive features of the candidate actions to obtain a fused feature. The fused features are input into a pre-built probability model for intent probability evaluation to obtain the behavior prediction result of the target person.

2. The human behavior prediction method as described in claim 1, characterized in that, The process of acquiring real-time behavioral video images of the target person and extracting skeletal point sequences from the real-time behavioral video images to obtain a set of joint coordinates and a skeletal motion sequence includes: Acquire a continuous sequence of image frames from real-time video images of the target person's behavior; The continuous image frame sequence is extracted based on a human pose estimation algorithm, wherein the process of extracting the skeletal point sequence includes: The coordinates of key joints of the human body are detected in each frame of the image to form a set of joint coordinates for a single frame; the sets of joint coordinates of each frame are arranged in chronological order to generate a skeletal motion sequence.

3. The human behavior prediction method as described in claim 1, characterized in that, The process of extracting spatiotemporal features from the set of joint coordinates to obtain a spatiotemporal feature sequence includes: The spatial feature vector of a single frame skeleton map is obtained by encoding the joint layout of each frame in the set of joint coordinates using a spatial graph convolutional neural network. Temporal dynamic features are extracted from the spatial feature vectors using a temporal convolutional network to obtain a spatiotemporal feature sequence that incorporates temporal information.

4. The human behavior prediction method as described in claim 3, characterized in that, The step of encoding the joint layout of each frame in the set of joint coordinates using a spatial graph convolutional neural network to obtain a spatial feature vector of a single-frame skeleton graph includes: Each joint in each frame of the joint coordinate set is taken as a node, and edges are added to the nodes according to the connection relationship of joints in human anatomy to construct a single-frame skeleton graph. The single-frame skeleton map is convolved using a spatial graph convolutional neural network with learnable adjacency matrix weights to output the spatial feature vector of the single-frame skeleton map. The learnable adjacency matrix is ​​designed to enable the spatial graph convolutional neural network to adaptively adjust the strength of joint connections based on training data; the spatial feature vector contains the semantic features of the current pose skeleton and the dependency information between joints.

5. The human behavior prediction method as described in claim 3, characterized in that, The temporal dynamic feature extraction of the spatial feature vector based on a temporal convolutional network, resulting in a spatiotemporal feature sequence incorporating temporal information, includes: Arrange the spatial feature vectors in chronological order to generate a spatial feature sequence; The spatial feature sequence is convolved in the time dimension using a temporal convolutional network with dilated convolutions to output a spatiotemporal feature sequence that incorporates temporal information. The dilated convolutions are designed as a combination of temporal convolutions with different dilation factors to capture detailed changes in short-term neighboring frames and dependencies over long time spans.

6. The human behavior prediction method as described in claim 1, characterized in that, The process of performing spatiotemporal attention encoding on the skeletal motion sequence to obtain global spatiotemporal dynamic features includes: The behavioral states of each frame within a short time window in the skeletal motion sequence are used as time series inputs into the Transformer model. Add positional encoding to the features of each time step in the time series to characterize the sequential relationship of each frame in the time series; The time series after adding positional encoding is processed based on the multi-head self-attention mechanism of the Transformer model to obtain a feature sequence that reflects cross-frame and cross-joint dependencies. The feature sequence is pooled over time to obtain global spatiotemporal dynamic features that include the trend of action time evolution and global joint correlation.

7. The human behavior prediction method as described in claim 1, characterized in that, The first action probability distribution data corresponding to the feature sequence is the first probability distribution for a predefined action category output by a spatial graph convolutional neural network, and the second action probability distribution data corresponding to the global spatiotemporal dynamic features is the second probability distribution for the predefined action category output by a temporal convolutional network. The step of fusing the action probability distributions corresponding to the spatiotemporal feature sequence and the global spatiotemporal dynamic features respectively to obtain short-term action probability distribution data includes: The first action probability distribution data and the second action probability distribution data are fused based on a weighted fusion strategy to generate short-time action probability distribution data; The fusion is designed to linearly sum the predicted probabilities of each of the predefined action categories in the two branches according to a set weight; the weights of the fusion reflect the performance of the spatial graph convolutional neural network and the temporal convolutional network during the model training or validation phase, respectively.

8. The human behavior prediction method as described in claim 1, characterized in that, The process of filtering candidate actions from the short-term action probability distribution data to obtain comprehensive features of candidate actions includes: Based on the Top-K screening strategy, the top K action categories with the highest probability values ​​sorted from the short-term action probability distribution data are selected to form a candidate action set; where K is a preset positive integer. Extract the probability values ​​corresponding to the candidate action set in the short-term action probability distribution data, and generate a predicted probability set that corresponds one-to-one with each action category in the candidate action set.

9. The human behavior prediction method as described in claim 1, characterized in that, The step of obtaining the historical action sequence of the target person and performing feature encoding on the historical action sequence to obtain a historical context vector includes: The action classification results of several past frames in the historical action sequence are statistically analyzed using a time window. The statistical results are input into a long short-term memory network for temporal encoding to obtain a historical context vector that represents the development of historical actions; wherein, the historical context vector reflects information about previously occurring key actions.

10. A human behavior prediction system, characterized in that, include: The acquisition module is used to acquire real-time behavioral video images of the target person in response to a behavior recognition trigger signal, and to extract the skeletal point sequence from the real-time behavioral video images to obtain a set of joint coordinates and a skeletal motion sequence. The extraction module is used to extract spatiotemporal features from the set of joint coordinates to obtain a spatiotemporal feature sequence, and to perform spatiotemporal attention encoding on the skeletal motion sequence to obtain global spatiotemporal dynamic features; The fusion module is used to fuse the action probability distributions corresponding to the spatiotemporal feature sequence and the global spatiotemporal dynamic features respectively to obtain short-time action probability distribution data. The filtering module is used to filter candidate actions from the short-term action probability distribution data to obtain comprehensive features of candidate actions. The filtering process is designed to filter a number of candidate actions based on the probability values ​​of each short-term action probability distribution data. The splicing module is used to respond to the behavior prediction trigger signal, acquire the historical action sequence of the target person, encode the historical action sequence to obtain a historical context vector, and splice the historical context vector with the candidate action comprehensive features to obtain a fused feature. The recognition module is used to input the fused features into a pre-built probability model to evaluate the intent probability and obtain the behavior prediction result of the target person.

Images