Behavior rule and deep learning fused behavior prediction method
By integrating behavioral patterns with deep learning, and optimizing deep neural networks using feature encoding and a hybrid error loss function, the problem of high training data requirements was solved, and highly accurate behavior prediction was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2026-03-10
AI Technical Summary
Existing technologies for behavior prediction suffer from high training data requirements and poor performance in tasks with high complexity and uncertainty.
This paper proposes a method that integrates behavioral patterns with deep learning. It uses long short-term memory neural networks and convolutional neural networks for feature encoding, combines behavioral pattern models with deep learning models, designs a loss function based on experience-pattern hybrid error, and optimizes deep neural networks to reduce the amount of training data and improve prediction accuracy.
While reducing the training data requirements, it improves the accuracy and consistency of behavior prediction, overcomes the dependence of deep neural networks on large-scale training data, and enhances the physical consistency of behavior prediction.
Smart Images
Figure CN121640558A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to artificial intelligence technology, and more particularly to a behavior prediction method that integrates behavioral patterns with deep learning. Background Technology
[0002] One of the ultimate goals of artificial intelligence research is to build a machine that can accurately understand and predict human behavior and intentions so that it can better serve humanity. With the rapid development of machine learning and computer vision, the focus of research on human behavior in the field of artificial intelligence has shifted from behavior recognition to behavior prediction.
[0003] Behavior recognition infers a person's behavior (current state) based on the complete execution of an action, while behavior prediction predicts a person's behavior (future state) based on the incomplete execution of an action.
[0004] Behavior recognition is typically applicable to non-emergency situations, such as entertainment, video retrieval, and visual monitoring; while behavior prediction infers subsequent behaviors before they are fully executed, which is particularly important in emergency scenarios. For example, in-vehicle intelligent systems can predict and warn drivers of unsafe behaviors that could lead to traffic accidents before such behaviors occur, rather than successfully recognizing them only after they have occurred.
[0005] However, behavior prediction enables AI agents to better interact with users, understand their behavior, predict their choices, and ultimately coordinate actions directly performed by humans and autonomous actions delegated to AI agents, thus truly achieving human-centered AI.
[0006] Current mainstream behavior prediction methods have certain advantages and limitations, as detailed below:
[0007] Prediction based on deep learning benefits from its excellent function fitting ability, enabling it to better adapt to observations of human behavior and extract complex patterns, but it requires massive amounts of expensive training data.
[0008] Predictions based on behavioral patterns stem from assumptions and expectations about human behavior and gain extrapolation capabilities beyond observation through experimentally verified mathematical laws, but they perform poorly in tasks with greater complexity and uncertainty. Summary of the Invention
[0009] The technical problem to be solved by this invention is: addressing the technical problems existing in the prior art, this invention provides a behavior prediction method that integrates behavioral patterns and deep learning, realizing the fusion of behavioral pattern models from behavioral science and deep learning models from artificial intelligence at the structural level, which can achieve high behavior prediction accuracy while reducing the data overhead required for model training.
[0010] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:
[0011] A behavior prediction method that integrates behavioral patterns and deep learning includes the following steps:
[0012] Step 1: Obtain the behavioral video with embedded human planar coordinate position information and preprocess it so that the processed video frames have the same temporal and spatial feature resolution;
[0013] Step 2: Combine long short-term memory neural networks and convolutional neural networks to encode human appearance and key point features in video frames;
[0014] Step 3: Use a long short-term memory neural network to encode the surrounding environment and human-environment interaction features of the behavioral video frames;
[0015] Step 4: Concatenate the human appearance feature encoding, key point feature encoding, surrounding environment feature encoding, and human-environment interaction feature encoding to obtain feature fusion encoding;
[0016] Step 5: Design an auxiliary task based on human movement trajectory learning, and use a long short-term memory neural network to generate human movement prediction trajectories to obtain a human movement trajectory generation network;
[0017] Step 6: Transform the human behavior pattern model that remains unchanged under time translation transformation into a computable behavior pattern model;
[0018] Step 7: Design a loss function based on a hybrid error of experience and regularity, and integrate the computable behavioral regularity model into the human movement trajectory generation network;
[0019] Step 8: Using a fully connected neural network, combining feature fusion encoding with the hidden state of the person's movement trajectory in the specified video frame as input, a linear function is used for classification to predict the behavior category.
[0020] Furthermore, the preprocessing in step 1 includes:
[0021] Step 1.1: Extract video frames from the behavioral video using an equal-interval downsampling method to ensure that the video frames have the same temporal feature resolution;
[0022] Step 1.2: Crop the video frames to a fixed size so that the video frames have the same spatial feature resolution.
[0023] Furthermore, step 2 specifically includes the following steps:
[0024] Step 2.1: Use a masked recurrent convolutional neural network along the time axis T1 to T2 of the video frame sequence.obs The human visual feature vector F at each time step is extracted using a two-dimensional convolutional kernel of size w*h. hv , among which, T obs Let w be the total number of video frames, and h be the width and height of the two-dimensional convolution kernel, respectively.
[0025] Step 2.2: Convert the human visual feature vector F hv The RolAlign method is used to reduce the features and input them into a long short-term memory neural network to encode human appearance features;
[0026] Step 2.3: Extract the original features of human keypoints using a publicly available human keypoint detection model trained on the MSCOCO dataset;
[0027] Step 2.4: Use linear transformation to embed the original features of human body key points in a low dimension to obtain human body key point feature encoding.
[0028] Furthermore, step 3 specifically includes the following steps:
[0029] Step 3.1: Use a pre-trained scene semantic segmentation network model to extract scene semantics from each video frame and encode the scenes in the video;
[0030] Step 3.2: Based on the geometric relationship between people and other objects in the video frame, obtain the human-ring interaction feature encoding using a human-ring interaction quantization rule designed by the user.
[0031] Furthermore, the formula for the human-environment interaction quantification rule is as follows:
[0032] HEI k (b,k)=[dis(x b ,x k ),dis(y b ,y k )]
[0033] Among them, HEI k The value represents the interaction between a person and object k, where dis(·) is a metric function based on spatial distance, and x b ,y b Let x and y represent the person's horizontal and vertical coordinates in the video frame, respectively. k ,y k Let w represent the x and y coordinates of the k-th object in the video frame, respectively. b ,h b These represent the width and height of a person, respectively.
[0034] Furthermore, step 5 specifically includes the following steps:
[0035] Step 5.1: Extract the human movement trajectory embedding e in the (t-1)th frame using the hyperbolic tangent function. t-1 The extraction formula is shown below:
[0036]
[0037] Where act(·) is a nonlinear function, It is the position vector of the person in the (t-1)th frame;
[0038] Step 5.2: Generate the predicted human movement trajectory in frame t using a Long Short-Term Memory (LSTM) neural network. t As shown below:
[0039]
[0040] in, For the input of the t-th video frame, h t-1 Let t be the hidden state of the person's movement trajectory in the (t-1)th video frame.
[0041] Furthermore, in step 6, when transforming the human behavior pattern model that remains unchanged under time translation into a computable behavior pattern model, specifically, this refers to using time variable embedding to transform the human behavior pattern model that remains unchanged under time translation into a computable behavior pattern model. The transformation process is as follows:
[0042]
[0043] Among them, M rule (·) represents a behavioral pattern model, M cpt (·,t) represents the computable behavioral pattern model, where t is the sequence number of the video frame.
[0044] Furthermore, step 7 specifically includes the following steps:
[0045] Step 7.1: Calculate the predicted trajectory of a person's movement based on the computable behavioral pattern model in the (t+1)th video frame. The calculation process is as follows:
[0046]
[0047] Where, loc self Let loc be the position vector of the target person in the scene. other A family of position vectors for other people in the scene;
[0048] Step 7.2: Using the L2 norm of the difference between the computable behavior pattern model and the prediction results of the human movement trajectory generation network as the pattern inconsistency value of the neural network, a loss function based on pattern inconsistency is designed as follows:
[0049]
[0050] in, To find the difference function, Z is the predicted human movement trajectory generated by the human movement trajectory generation network, where Z is the predicted human movement trajectory from the computable behavioral pattern model.
[0051] Step 7.3: Using the difference between the prediction result of the human movement trajectory generation network and the correct trajectory, design a loss function based on empirical error and structural error penalty terms, as shown below:
[0052]
[0053] Where Loss is the empirical error function, and R is the structural error function. The network generates a human movement trajectory, which predicts the human's movement trajectory. Y represents the correct trajectory, and λ represents the structural error weighting parameter.
[0054] Step 7.4: Combining the two loss functions based on regular inconsistency and those based on experience and structural error, design a loss function based on a hybrid empirical-regular error, as shown below:
[0055]
[0056] Where, λ PHY The weight parameters are inconsistent according to the pattern.
[0057] Furthermore, the expression for the behavior category in step 8 is as follows:
[0058]
[0059] Where flat(·) is a linear function, This represents the feature fusion encoding of the specified M video frames.
[0060] The present invention also proposes a behavior prediction system that integrates behavioral patterns and deep learning, comprising an interconnected microprocessor and a computer-readable storage medium, wherein the microprocessor is programmed or configured to execute any of the behavior prediction methods that integrate behavioral patterns and deep learning.
[0061] Compared with the prior art, the advantages of the present invention are as follows:
[0062] This invention improves the physical consistency of deep neural network extrapolation results by transforming mathematically model-based behavioral patterns into computable behavioral pattern models, and designs a loss function based on experience-pattern hybrid error to guide deep neural network optimization, while reducing the training data overhead of deep neural networks. By cascading and fusing features from three aspects—human body, environment, and human-environment interaction—and using the auxiliary task of predicting human movement trajectory for multi-task learning, the accuracy of deep neural networks in predicting human behavior is improved.
[0063] The method of this invention overcomes the problem that deep neural networks require large-scale training data when learning and predicting human behavior, while improving the accuracy of the network model in predicting human behavior categories. Attached Figure Description
[0064] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.
[0065] Figure 2 This is a schematic diagram illustrating the process of designing a loss function based on a hybrid error of experience and regularity in an embodiment of the present invention.
[0066] Figure 3 This is a structural diagram of the behavior prediction model according to an embodiment of the present invention. Detailed Implementation
[0067] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.
[0068] Example 1
[0069] This embodiment proposes a behavior prediction method that integrates behavioral patterns and deep learning, such as... Figure 1 As shown, it includes the following steps:
[0070] Step 1: Obtain behavioral videos with embedded human planar coordinate position information. In this embodiment, behavioral videos from the ActEV-18 dataset are collected and preprocessed to ensure that the processed video frames have the same temporal and spatial feature resolution. In this embodiment, the ActEV-18 dataset contains 29 behavioral categories and consists of 119 segments containing human movement and environmental visual information. This dataset was later annotated to enhance it to include human two-dimensional planar coordinate information. Preprocessing in Step 1 includes:
[0071] Step 1.1: Extract video frames from the behavioral video using an equal-interval downsampling method to ensure that the video frames have the same temporal feature resolution. In this embodiment, the temporal resolution of the cropped video frames is fixed at 2.5Hz.
[0072] Step 1.2: Crop the video frames to a fixed size so that the video frames have the same spatial resolution. In this embodiment, the spatial resolution of the cropped video frames is fixed at 1920*1080.
[0073] Step 2: Combine Long Short-Term Memory Neural Network and Convolutional Neural Network to encode human appearance and key point features in video frames, specifically including the following steps:
[0074] Step 2.1: Use a masked recurrent convolutional neural network along the time axis T1 to T2 of the video frame sequence. obs The human visual feature vector F at each time step is extracted using a two-dimensional convolutional kernel of size w*h. hv , among which, T obs Let w be the total number of video frames, and h be the width and height of the two-dimensional convolution kernel, respectively.
[0075] Step 2.2: Convert the human visual feature vector F hv The RolAlign method is used to reduce the features and input them into a long short-term memory neural network to encode human appearance features;
[0076] Step 2.3: Extract the original features of human keypoints using a publicly available human keypoint detection model trained on the MSCOCO dataset;
[0077] Step 2.4: Use linear transformation to embed the original features of human body key points in a low dimension to obtain human body key point feature encoding.
[0078] Step 3: Use a Long Short-Term Memory (LSTM) neural network to encode the surrounding environment and human-environment interaction features of the behavioral video frames. This includes the following steps:
[0079] Step 3.1: Use a pre-trained scene semantic segmentation network model to extract scene semantics from each video frame and encode the scenes in the video;
[0080] Step 3.2: Based on the geometric relationships between people and other objects in the video frames, obtain the human-ring interaction feature encoding using a human-ring interaction quantization rule designed by the user. The formula for the human-ring interaction quantization rule is shown below:
[0081] HEI k (b,k)=[dis(x b ,x k ),dis(y b ,y k (1)
[0082] Among them, HEI k The value represents the interaction between a person and object k, where dis(·) is a metric function based on spatial distance, and x b,y b Let x and y represent the person's horizontal and vertical coordinates in the video frame, respectively. k ,y k Let w represent the x and y coordinates of the k-th object in the video frame, respectively. b ,h b These represent the width and height of a person, respectively.
[0083] Step 4: Concatenate the human appearance feature encoding, key point feature encoding, surrounding environment feature encoding, and human-environment interaction feature encoding to obtain feature fusion encoding;
[0084] Step 5: Introduce a multi-task learning paradigm, design an auxiliary task based on human movement trajectory learning, utilize a long short-term memory neural network to generate human movement prediction trajectories, realize multi-task learning of behavior, deepen the neural network model's "understanding" of human behavior, and obtain the human movement trajectory generation network, specifically including the following steps:
[0085] Step 5.1: Extract the human movement trajectory embedding e in the (t-1)th frame using the hyperbolic tangent function. t-1 The extraction formula is shown below:
[0086]
[0087] Where act(·) is a nonlinear function, It is the position vector of the person in the (t-1)th frame;
[0088] Step 5.2: Generate the predicted human movement trajectory in frame t using a Long Short-Term Memory (LSTM) neural network. t As shown below:
[0089]
[0090] in, For the input of the t-th video frame, h t-1 Let t be the hidden state of the person's movement trajectory in the (t-1)th video frame.
[0091] Step 6: Using time variable embedding, the human behavior pattern model that remains unchanged under time translation transformation is transformed into a computable behavior pattern model. The transformation process is shown below:
[0092]
[0093] Among them, M rule (·) represents a behavioral pattern model, M cpt (·,t) represents the computable behavioral pattern model, where t is the sequence number of the video frame.
[0094] It should be noted that the behavioral pattern model used in this step is a publicly available behavioral pattern model, and its specific manifestations and related formulas will not be elaborated here.
[0095] Step 7: Design a loss function based on a hybrid error of experience and regularity, integrating a computable behavioral pattern model of human movement behavior into the human movement trajectory generation network, such as... Figure 2 As shown, the specific steps include:
[0096] Step 7.1: Calculate the predicted trajectory of a person's movement based on the computable behavioral pattern model in the (t+1)th video frame. The calculation process is as follows:
[0097]
[0098] Where, loc self Let loc be the position vector of the target person in the scene. other A family of position vectors for other people in the scene;
[0099] Step 7.2: Using the L2 norm of the difference between the computable behavior pattern model and the prediction results of the human movement trajectory generation network as the pattern inconsistency value of the neural network, a loss function based on pattern inconsistency is designed as follows:
[0100]
[0101] in, To find the difference function, Z is the predicted human movement trajectory generated by the human movement trajectory generation network, where Z is the predicted human movement trajectory from the computable behavioral pattern model.
[0102] Step 7.3: Using the difference between the prediction result of the human movement trajectory generation network and the correct trajectory, design a loss function based on empirical error and structural error penalty terms, as shown below:
[0103]
[0104] Where Loss is the empirical error function, and R is the structural error function. The network generates a human movement trajectory, which predicts the human's movement trajectory. Y represents the correct trajectory, and λ represents the structural error weighting parameter.
[0105] Step 7.4: Combining the two loss functions based on regular inconsistency and those based on experience and structural error, design a loss function based on a hybrid empirical-regular error, as shown below:
[0106]
[0107] Where, θPHY The weight parameters are inconsistent according to the pattern.
[0108] Step 8: Use a fully connected neural network, combining feature fusion encoding with the hidden state of the person's movement trajectory in the specified video frame as input (denoted as Q). T The system uses a linear function for classification to predict behavior categories. The expression for each behavior category is as follows:
[0109]
[0110] Where flat(·) is a linear function, This represents the feature fusion encoding of the specified M video frames.
[0111] Through the above steps, the final structure of the behavior prediction model that integrates behavioral patterns and deep learning is as follows: Figure 3 As shown, this model guides the optimization of deep neural networks based on a loss function of empirical-regular mixed error to improve the physical consistency of deep neural network extrapolation results, while reducing the training data overhead of deep neural networks. By cascading and fusing features from three aspects—human body, environment, and human-environment interaction—and using the auxiliary task of predicting human movement trajectory for multi-task learning, the accuracy of deep neural networks in predicting human behavior overcomes the problem of requiring large-scale training data when learning and predicting human behavior, and improves the accuracy of the network model in predicting human behavior categories.
[0112] Example 2
[0113] This embodiment proposes a behavior prediction system that integrates behavioral patterns and deep learning, including a microprocessor and a computer-readable storage medium interconnected with each other. The microprocessor is programmed or configured to execute the behavior prediction method that integrates behavioral patterns and deep learning as described in Embodiment 1.
[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.
Claims
1. A method of behavior prediction fusing behavior rules and deep learning, characterized in that, The method comprises the following steps: Step 1: Obtain a behavior video embedded with human plane coordinate position information, and perform preprocessing, so that the processed video frames have the same time feature resolution and spatial feature resolution; Step 2: Combine a long short-term memory neural network and a convolutional neural network to encode human appearance and key point features of the video frames; Step 3: Use a long short-term memory neural network to encode the surrounding environment and human interaction features of the behavior video frames; Step 4: Cascade the human appearance feature encoding, key point feature encoding, surrounding environment feature encoding and human interaction feature encoding to obtain feature fusion encoding; Step 5: Use a long short-term memory neural network to generate a human movement prediction trajectory, and obtain a human movement trajectory generation network; Step 6: Convert the human behavior rule model that remains unchanged under time translation transformation into a calculable behavior rule model; Step 7: Design a loss function based on the experience-rule hybrid error, and integrate the calculable behavior rule model into the human movement trajectory generation network; Step 8: Use a fully connected neural network, combine the feature fusion encoding and the hidden state of the human movement trajectory of the specified video frame as input, and use a linear function for classification to predict the behavior category. 2.The method of claim 1, wherein, In step 1, the preprocessing includes: Step 1.1: Extract video frames from the behavior video using an equal-interval downsampling method, so that the video frames have the same time feature resolution; Step 1.2: Crop the video frames to a fixed size, so that the video frames have the same spatial feature resolution. 3.The method of claim 1, wherein, Step 2 specifically includes the following steps: Step 2.1: using a cyclic convolutional neural network with a mask along the time axis T1 to T obs extract the human visual feature vector F in each time step using a two-dimensional convolution kernel with size w*h hv , where T obs is the total number of video frames, and w, h are the width and height of the two-dimensional convolution kernel, respectively; Step 2.2: The human visual feature vector F hv Feature reduction using RolAlign method is used to input into a long short-term memory neural network for encoding of human appearance features; Step 2.3: Use a public human key point detection model trained on the MSCOCO dataset to extract human key point original features; Step 2.4: Use linear transformation to embed the human key point original features into a low-dimensional space to obtain human key point feature encoding. 4.The method of claim 1, wherein, Step 3 specifically includes the following steps: Step 3.1: Use a pre-trained scene semantic segmentation network model to extract scene semantics from each video frame, and encode the scene in the video; Step 3.2: According to the geometric relationship between the human and other objects in the video frame, use a human-designed human interaction quantization rule to obtain human interaction feature encoding.
5. The method of claim 4, wherein the fusion of behavioral rules and deep learning for behavior prediction is characterized by, The human interaction quantization rule formula is as follows: HEI k (b, k) = [dis(x b , x k ), dis(y b , y k )] where HEI k is the interaction value between the representative person and the object k, dis(·) is a distance-based metric function, x b , y b are the horizontal and vertical coordinates of the person in the video frame, x k , y k are the horizontal and vertical coordinates of the kth object in the video frame, w b , h b are the width and height of the person, respectively. 6.The method of claim 1, wherein, Step 5 specifically includes the following steps: Step 5.1: Extracting the human movement trajectory embedding e in the t-1 frame using the hyperbolic tangent function t-1 The extraction formula is as follows: where act(-) is a non-linear function, is the position vector of the person in the t-1 frame. Step 5.2: Generating the predicted human movement trajectory o of the t-th frame using a long short-term memory neural network t As follows: wherein, is the input for the tth video frame, h t-1 is the human movement trajectory hidden state for the t-1th video frame. 7.The method of claim 1, wherein, In step 6, when converting the human behavior rule model that remains unchanged under time translation transformation into a calculable behavior rule model, it specifically refers to using time variable embedding to convert the human behavior rule model that remains unchanged under time translation transformation into a calculable behavior rule model, and the conversion process is as follows: wherein M rule (·) is a behavior rule model, M cpt (·, t) is a computable behavior rule model, and t is a sequence number of a video frame. 8.The method of claim 1, wherein, Step 7 specifically includes the following steps: Step 7.1: Calculate the human movement prediction trajectory based on the calculable behavior rule model in the t+1 video frame, and the calculation process is as follows: wherein loc self is the position vector of the target person in the scene, loc other is the position vector family of other persons in the scene; Step 7.2: Use the 2-norm of the difference between the prediction results of the calculable behavior rule model and the human movement trajectory generation network as the rule inconsistency value of the neural network, and design a loss function based on the rule inconsistency, as follows: wherein, is a difference function, is a human movement trajectory generation network to predict a human movement trajectory, and Z is a predicted human movement trajectory that can be calculated by the behavior rule model. Step 7.3: The loss function is designed based on the experience error and structure error penalty term, as shown below: wherein Loss is an empirical error function, R is a structural error function, The network generates a predicted human movement trajectory for the human movement trajectory, Y is a correct trajectory, and λ is a structural error weight parameter. Step 7.4: The loss function is designed based on the experience-law hybrid error, combining the two loss functions based on the rule inconsistency and the experience and structure error, as shown below: where λ PHY is a regularized inconsistency weight parameter. 9.The method of claim 1, wherein, The expression of the behavior category in step 8 is as follows: wherein flat(-) is a linear function, representing a feature fusion encoding in the designated M video frames. 10.A behavior prediction system fusing behavior rules and deep learning, characterized in that, The computer readable storage medium comprises interconnected microprocessors and computer readable storage media, and the microprocessors are programmed or configured to perform the behavior prediction method of fusing behavior rules and deep learning according to any one of claims 1-9.