First-person video behavior prediction method and system based on cycle consistency

Abstract

This paper proposes a first-person video behavior prediction method and system based on cycle consistency. By constructing past information distillation loss functions, future information distillation loss functions, backward-forward cycle consistency loss functions, and forward-backward cycle consistency loss functions in a high-dimensional feature space, this method fully learns the temporal contextual relationships between past content and the currently observed content, and between the currently observed content and future content. During testing, future behavior is predicted based solely on the observed content. By implicitly expanding the observable temporal range, this method can, to a certain extent, overcome the objective limitation of obtaining long-range temporal information, thereby utilizing more information to assist in behavior prediction.

Description

Technical Field

[0001] The present invention relates to the field of first-person perspective videos, and in particular to first-person perspective video behavior prediction. Background Art

[0002] First-person video behavior prediction is widely used in fields such as human-computer interaction and autonomous driving. It predicts the next likely behavior based on a first-person video clip. For example, in human-computer interaction, if the machine predicts a person wants to drink water, it can offer the cup in advance, providing more considerate service. Another example is in autonomous driving, where the machine predicts a pedestrian is about to cross the road and can slow down in advance to avoid a traffic accident.

[0003] Some work aims to directly map observed video clips to future behavior categories. For example, a convolutional neural network (CNN) is used to extract visual features from each frame of the input video clip. A long short-term memory (LSTM) network then aggregates the features of each frame along the temporal dimension. Finally, a linear layer is used to map the high-dimensional features aggregated along the temporal dimension into the behavior category space, resulting in predictions about future behaviors. This approach uses only future behavior labels as supervisory signals to construct a classification loss function to guide the training of the behavior prediction model. Beyond future behavior labels, the contribution of future visual content to training the behavior prediction model is negligible, and the available information is relatively limited. Other work, inspired by the idea of ​​knowledge distillation, attempts to transfer external information to a limited set of observations during training, while still making predictions based solely on the observed content during testing. For example, label smoothing techniques are used during training to transform the traditional cross-entropy classification loss function to encourage the model to learn other behavior labels that are semantically similar to the true labels. Another example is that during the training process, it is proposed to maximize the similarity between the true value of future features and the predicted value of future features at the current moment, so that the behavior prediction model can fully learn the ability to reconstruct future features. This technology is currently the most widely used and can learn the contextual relationship between the current observation content and the future content, thereby improving the reliability of behavior prediction.

[0004] When distilling external information into a limited number of observed video clips, existing techniques typically focus solely on maximizing the similarity between the true value of future features and the current predicted value of future features during training. This results in the model learning only the contextual relationships between the current observation and future content. However, first-person videos also contain contextual relationships between the observed content and past content. The behaviors in first-person videos are diverse, short-lived, and frequently change. Therefore, comprehensive learning of these contextual relationships is crucial and helps improve the model's temporal reasoning capabilities. For example, when observing a video clip of "opening a cupboard," the model may struggle to correctly predict the next action due to the uncertainty surrounding future actions. However, if the model can understand that the past action was "taking a cup," it can reduce future uncertainty and tend to predict that the next action is likely to be "placing the cup in the cupboard." This demonstrates that simply learning the contextual relationships between the current observation and future content is insufficient; comprehensive learning requires integrating the contextual relationships between the observed content and past content. Summary of the Invention

[0005] The present invention proposes a first-person perspective video behavior prediction method based on cycle consistency, which enables the behavior prediction model to fully and comprehensively learn the temporal contextual relationship in the first-person perspective video, thereby implicitly expanding the observable temporal range when the length of the input observable video is limited, and to a certain extent overcomes the objective condition limitation of difficulty in obtaining long-distance temporal information.

[0006] This paper proposes a first-person video behavior prediction method based on cycle consistency. By constructing past information distillation loss functions, future information distillation loss functions, backward-forward cycle consistency loss functions, and forward-backward cycle consistency loss functions in a high-dimensional feature space, it fully learns the temporal contextual relationships between past content and current observations, and between current observations and future content. During testing, future behavior is predicted based solely on the observed content. By implicitly expanding the observable temporal range, this method can, to a certain extent, overcome the objective limitation of obtaining long-range temporal information, thereby utilizing more information to assist in behavior prediction.

[0007] Specifically, the present invention proposes a first-person video behavior prediction method based on cycle consistency, which includes:

[0008] Step 1: Divide the first-perspective training video into three segments according to time, representing past content, observed content, and future content, respectively. Extract features from the past content, the observed content, and the future content using a feature extraction module to obtain past features, observed features, and future features. Input the observed features into a backward mapping module based on a linear layer to obtain a first past prediction value of the observed content. Based on the similarity between the first past prediction value and the past features, a first loss function is constructed.

[0009] Step 2: Input the past prediction value into a forward mapping module based on a linear layer to obtain a first future prediction value of the first past prediction value, and construct a second loss function based on the similarity between the first future prediction value and the observed feature;

[0010] Step 3: Input the observed feature into the forward mapping module to obtain a second future prediction value of the observed content, and construct a third loss function based on the similarity between the second future prediction value and the future feature;

[0011] Step 4: input the second future prediction value into the backward mapping module to obtain a second past prediction value of the second future prediction value, and construct a fourth loss function based on the similarity between the second past prediction value and the observed feature;

[0012] Step 5: Input the second future prediction value into the classification layer to obtain a predicted category, and construct a fifth loss function based on the predicted category and the target category marked in the training video;

[0013] Step 6: Add the first loss function to the fifth loss function to form a total loss function, and iteratively train and update the feature extraction module, the forward mapping module, the backward mapping module, and the classification layer;

[0014] Step 7: Save the trained feature extraction module, forward mapping module, backward mapping module, and classification layer as a prediction model to perform the first-person video behavior prediction task.

[0015] The first perspective video behavior prediction method based on cycle consistency, wherein the first loss function

[0016]

[0017] The second loss function

[0018]

[0019] The third loss function

[0020]

[0021] The fourth loss function

[0022]

[0023] The total loss function L is:

[0024]

[0025] Where z p represents the past feature, z o represents the observed feature, z f Representing this future feature, represents the first past prediction value, represents the first future prediction value, represents the second future prediction value, represents the second past prediction value, Represents the fifth loss function.

[0026] The first-person perspective video behavior prediction method based on cycle consistency, wherein step 7 includes:

[0027] The feature extraction module extracts the original features of the first-perspective video for behavior prediction, and inputs the original features into the backward mapping module to obtain the first prediction features for the past content. The original features and the prediction features are added and input into the forward mapping module. The forward mapping module inputs the extracted second prediction features into the classification layer to obtain the behavior prediction results of the first-perspective video.

[0028] The present invention also proposes a first-person video behavior prediction system based on cycle consistency, which includes:

[0029] Module 1: Divide the first-person perspective training video into three segments according to time, representing past content, observed content, and future content, respectively. A feature extraction module extracts features from the past content, the observed content, and the future content to obtain past features, observed features, and future features. The observed features are input into a backward mapping module based on a linear layer to obtain a first past prediction value of the observed content. A first loss function is constructed based on the similarity between the first past prediction value and the past features.

[0030] Module 2: input the past prediction value into a forward mapping module based on a linear layer to obtain a first future prediction value of the first past prediction value, and construct a second loss function based on the similarity between the first future prediction value and the observed feature;

[0031] Module 3: input the observed feature into the forward mapping module to obtain a second future prediction value of the observed content, and construct a third loss function based on the similarity between the second future prediction value and the future feature;

[0032] Module 4: input the second future prediction value into the backward mapping module to obtain a second past prediction value of the second future prediction value, and construct a fourth loss function based on the similarity between the second past prediction value and the observed feature;

[0033] Module 5: Input the second future prediction value into the classification layer to obtain a predicted category, and construct a fifth loss function based on the predicted category and the target category marked in the training video;

[0034] Module 6: Add the first loss function and the fifth loss function to form a total loss function, and iteratively train and update the feature extraction module, the forward mapping module, the backward mapping module, and the classification layer;

[0035] Module 7: Save the trained feature extraction module, forward mapping module, backward mapping module, and classification layer as a prediction model to perform the first-person video behavior prediction task.

[0036] The first-perspective video behavior prediction system based on cycle consistency, wherein the first loss function

[0037]

[0038] The second loss function

[0039]

[0040] The third loss function

[0041]

[0042] The fourth loss function

[0043]

[0044] The total loss function L is:

[0045]

[0046] Where z p represents the past feature, z o represents the observed feature, z f Representing this future feature, represents the first past prediction value, represents the first future prediction value, represents the second future prediction value, represents the second past prediction value, Represents the fifth loss function.

[0047] The first-person video behavior prediction system based on cycle consistency, wherein the module 7 includes:

[0048] The feature extraction module extracts the original features of the first-perspective video for behavior prediction, and inputs the original features into the backward mapping module to obtain the first prediction features for the past content. The original features and the prediction features are added and input into the forward mapping module. The forward mapping module inputs the extracted second prediction features into the classification layer to obtain the behavior prediction results of the first-perspective video.

[0049] The present invention also proposes a storage medium for storing a program for executing any one of the first-perspective video behavior prediction methods based on cyclic consistency.

[0050] The present invention also proposes a client for use in any first-person video behavior prediction system based on cycle consistency.

[0051] The present invention also proposes a vehicle automatic driving method, wherein the road condition video collected by the vehicle is used as the first-person perspective video, and the first-person perspective video behavior prediction method based on cyclic consistency is adopted to obtain the behavior prediction result of the specified target in the road condition video to control the vehicle to perform the automatic driving task.

[0052] It can be seen from the above scheme that the advantages of the present invention are:

[0053] Compared to existing technologies, this paper proposes a first-person video behavior prediction method based on cycle consistency, which fully learns the temporal contextual relationships between past and current observations, and between current and future observations. This method implicitly expands the observable temporal range, even when the length of the observable input video is limited, reducing future uncertainty and making behavior prediction more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is the overall technical framework diagram of the present invention. DETAILED DESCRIPTION

[0055] The present invention proposes a first-person video behavior prediction method based on cycle consistency. The purpose of cycle consistency is to map the original content from one domain to another and then convert it back. The converted content should be as close to the original content as possible. We found that there are two natural temporal relationships in first-person videos that can be modeled using cycle consistency: (1) the cycle of current observation → past → current observation: given the current observation content, the corresponding past content is obtained through backward mapping on the time dimension, and then the corresponding current content is obtained through forward mapping on the time dimension. It should be as close as possible to the original current observation content; (2) the cycle of current observation → future → current observation: given the current observation content, the corresponding future content is obtained through forward mapping on the time dimension, and then the corresponding current content is obtained through backward mapping on the time dimension. It should be as close as possible to the original current observation content. During training, we divide the video into three segments of equal length, representing the past content, observation content, and future content respectively. Two cycle consistency losses are used to constrain the model's learning of temporal contextual relationships. During testing, we still only predict behavior based on the observation content, without inputting redundant external information or introducing additional computation. Therefore, our method can, to a certain extent, overcome the objective condition limitation of difficulty in obtaining long-distance temporal information when the length of the input observable video is limited, and then use more information to assist in complex behavior prediction.

[0056] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0057] like Figure 1 As shown, during training, we divide the training video into three segments of equal length, representing past content, observed content, and future content. The videos used in the test phase are consistent with the training scenario. In the high-dimensional feature space, we construct a past information distillation loss function, a backward-forward cycle consistency loss function, a future information distillation loss function, and a forward-backward cycle consistency loss function to fully learn the temporal contextual relationships between past content and current observation content, and between current observation content and future content. Next, we will introduce the technical solution in detail:

[0058] (1) Past Information Distillation

[0059] Given the current observation content and the past content, we use CNN to extract features from them respectively, and then use LSTM to perform temporal aggregation on the extracted features to obtain high-dimensional feature vectors z o and z p For z o , we use a linear layer to map it backward in the time dimension to the corresponding past content, and obtain a high-dimensional feature vector Indicates the predicted value of past content based on the current observation content. We maximize and z p The cosine similarity of derives the distillation loss function (first loss) of past information:

[0060]

[0061] (2) Backward-forward cycle consistency

[0062] We use a linear layer Forward mapping in the time dimension back to the content corresponding to the current observation to obtain a high-dimensional feature vector According to the cycle consistency theory, Should be as close to z as possible o , so we maximize and z o The cosine similarity of derives the backward-forward cycle consistency function (second loss):

[0063]

[0064] (3) Future Information Distillation

[0065] Given the current observation content and future content, we use CNN to extract features from them respectively, and then use LSTM to perform temporal aggregation on the extracted features to obtain high-dimensional feature vectors z and z, respectively. o and z f For z o , we use a linear layer to map it forward in the time dimension to the corresponding future content, and obtain a high-dimensional feature vector Indicates the predicted value of future content based on the current observation content. We maximize and z f The cosine similarity of derives the distillation loss function (third loss) of future information:

[0066]

[0067] (4) Forward-backward cycle consistency

[0068] We use a linear layer Backward mapping in the time dimension back to the content corresponding to the current observation to obtain a high-dimensional feature vector According to the cycle consistency theory, Should be as close to z as possible o , so we maximize and z o The cosine similarity of derives the forward-backward cycle consistency function (the fourth loss):

[0069]

[0070] Will Input the classification layer (linear layer) to obtain the predicted category. Based on the predicted category and the target category marked in the training video, the fifth loss function is constructed. As a cross entropy loss function, it also measures the similarity between the predicted value and the target value. Minimizing the loss function is equivalent to maximizing the similarity between the predicted value and the target value. The specific form is:

[0071]

[0072] (5) Training objectives

[0073] The overall training objectives include past information distillation loss function, backward-forward cycle consistency loss function, future information distillation loss function, forward-backward cycle consistency loss function and cross entropy classification loss function:

[0074]

[0075] in Represents the cross entropy loss function, y represents the predicted future behavior category, and It is obtained by mapping from high-dimensional feature space to behavior category space through linear layer. represents the manually labeled true future behavior category.

[0076] (6) Testing

[0077] The input video first obtains the predicted value of the past content through backward mapping, and then adds the predicted value vector of the current observation content and the past content to obtain It is then input into the forward mapping module. Because it does not require the actual past content to be input, but only uses the predicted value of the past content, it can be understood as implicitly expanding the observable time series range.

[0078] Compared to training where both past and future content are input alongside the current observation, during testing, we only make behavior predictions based on the limited current observation, without requiring past or future content. Therefore, our approach implicitly expands the observable temporal range while limiting the length of the observable video input, overcoming the objective limitation of obtaining long-distance temporal information to a certain extent.

[0079] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0080] The present invention also proposes a first-person video behavior prediction system based on cycle consistency, which includes:

[0081] Module 1: Divide the first-person perspective training video into three segments according to time, representing past content, observed content, and future content, respectively. A feature extraction module extracts features from the past content, the observed content, and the future content to obtain past features, observed features, and future features. The observed features are input into a backward mapping module based on a linear layer to obtain a first past prediction value of the observed content. A first loss function is constructed based on the similarity between the first past prediction value and the past features.

[0082] Module 2: input the past prediction value into a forward mapping module based on a linear layer to obtain a first future prediction value of the first past prediction value, and construct a second loss function based on the similarity between the first future prediction value and the observed feature;

[0083] Module 3: input the observed feature into the forward mapping module to obtain a second future prediction value of the observed content, and construct a third loss function based on the similarity between the second future prediction value and the future feature;

[0084] Module 4: input the second future prediction value into the backward mapping module to obtain a second past prediction value of the second future prediction value, and construct a fourth loss function based on the similarity between the second past prediction value and the observed feature;

[0085] Module 5: Input the second future prediction value into the classification layer to obtain a predicted category, and construct a fifth loss function based on the predicted category and the target category marked in the training video;

[0086] Module 6: Add the first loss function and the fifth loss function to form a total loss function, and iteratively train and update the feature extraction module, the forward mapping module, the backward mapping module, and the classification layer;

[0087] Module 7: Save the trained feature extraction module, forward mapping module, backward mapping module, and classification layer as a prediction model to perform the first-person video behavior prediction task.

[0088] The first-perspective video behavior prediction system based on cycle consistency, wherein the first loss function

[0089]

[0090] The second loss function

[0091]

[0092] The third loss function

[0093]

[0094] The fourth loss function

[0095]

[0096] The total loss function L is:

[0097]

[0098] Where z p represents the past feature, z o represents the observed feature, z f Representing this future feature, represents the first past prediction value, represents the first future prediction value, represents the second future prediction value, represents the second past prediction value, Represents the fifth loss function.

[0099] The first-person video behavior prediction system based on cycle consistency, wherein the module 7 includes:

[0100] The feature extraction module extracts the original features of the first-perspective video for behavior prediction, and inputs the original features into the backward mapping module to obtain the first prediction features for the past content. The original features and the prediction features are added and input into the forward mapping module. The forward mapping module inputs the extracted second prediction features into the classification layer to obtain the behavior prediction results of the first-perspective video.

[0101] The present invention also proposes a storage medium for storing a program for executing any one of the first-perspective video behavior prediction methods based on cyclic consistency.

[0102] The present invention also proposes a client for use in any first-person video behavior prediction system based on cycle consistency.

[0103] The present invention also proposes a vehicle automatic driving method, wherein the road condition video collected by the vehicle is used as the first-person perspective video, and the first-person perspective video behavior prediction method based on cyclic consistency is adopted to obtain the behavior prediction result of the specified target in the road condition video to control the vehicle to perform the automatic driving task.

Claims

1. A first-person video behavior prediction method based on cycle consistency, characterized in that: include: Step 1: Divide the first-perspective training video into three segments according to time, representing past content, observed content, and future content, respectively. Extract features from the past content, the observed content, and the future content using a feature extraction module to obtain past features, observed features, and future features. Input the observed features into a backward mapping module based on a linear layer to obtain a first past prediction value of the observed content. Based on the similarity between the first past prediction value and the past features, a first loss function is constructed. Step 2: Input the past prediction value into a forward mapping module based on a linear layer to obtain a first future prediction value of the first past prediction value, and construct a second loss function based on the similarity between the first future prediction value and the observed feature; Step 3: Input the observed feature into the forward mapping module to obtain a second future prediction value of the observed content, and construct a third loss function based on the similarity between the second future prediction value and the future feature; Step 4: input the second future prediction value into the backward mapping module to obtain a second past prediction value of the second future prediction value, and construct a fourth loss function based on the similarity between the second past prediction value and the observed feature; Step 5: Input the second future prediction value into the classification layer to obtain a predicted category, and construct a fifth loss function based on the predicted category and the target category marked in the training video; Step 6: Add the first loss function to the fifth loss function to form a total loss function, and iteratively train and update the feature extraction module, the forward mapping module, the backward mapping module, and the classification layer; Step 7: Save the trained feature extraction module, forward mapping module, backward mapping module, and classification layer as a prediction model to perform the first-person video behavior prediction task.

2. The first-person video behavior prediction method based on cycle consistency according to claim 1, characterized in that: The first loss function The second loss function The third loss function The fourth loss function The total loss function L is: Where z p represents the past feature, z o represents the observed feature, z f Representing this future feature, represents the first past prediction value, represents the first future prediction value, represents the second future prediction value, represents the second past prediction value, Represents the fifth loss function.

3. The first-person video behavior prediction method based on cycle consistency according to claim 1 or 2, characterized in that: This step 7 includes: The feature extraction module extracts the original features of the first-perspective video for behavior prediction, and inputs the original features into the backward mapping module to obtain the first prediction features for the past content. The original features and the prediction features are added and input into the forward mapping module. The forward mapping module inputs the extracted second prediction features into the classification layer to obtain the behavior prediction results of the first-perspective video.

4. A first-person video behavior prediction system based on cycle consistency, characterized by: include: Module 1: Divide the first-person perspective training video into three segments according to time, representing past content, observed content, and future content, respectively. A feature extraction module extracts features from the past content, the observed content, and the future content to obtain past features, observed features, and future features. The observed features are input into a backward mapping module based on a linear layer to obtain a first past prediction value of the observed content. A first loss function is constructed based on the similarity between the first past prediction value and the past features. Module 2: input the past prediction value into a forward mapping module based on a linear layer to obtain a first future prediction value of the first past prediction value, and construct a second loss function based on the similarity between the first future prediction value and the observed feature; Module 3: input the observed feature into the forward mapping module to obtain a second future prediction value of the observed content, and construct a third loss function based on the similarity between the second future prediction value and the future feature; Module 4: input the second future prediction value into the backward mapping module to obtain a second past prediction value of the second future prediction value, and construct a fourth loss function based on the similarity between the second past prediction value and the observed feature; Module 5: Input the second future prediction value into the classification layer to obtain a predicted category, and construct a fifth loss function based on the predicted category and the target category marked in the training video; Module 6: Add the first loss function and the fifth loss function to form a total loss function, and iteratively train and update the feature extraction module, the forward mapping module, the backward mapping module, and the classification layer; Module 7: Save the trained feature extraction module, forward mapping module, backward mapping module, and classification layer as a prediction model to perform the first-person video behavior prediction task.

5. The first-person video behavior prediction system based on cycle consistency according to claim 4, characterized in that: The first loss function The second loss function The third loss function The fourth loss function The total loss function L is: Where z p represents the past feature, z o represents the observed feature, z f Representing this future feature, represents the first past prediction value, represents the first future prediction value, represents the second future prediction value, represents the second past prediction value, Represents the fifth loss function.

6. The first-person video behavior prediction system based on cycle consistency according to claim 4 or 5, characterized in that: This module 7 includes: The feature extraction module extracts the original features of the first-perspective video for behavior prediction, and inputs the original features into the backward mapping module to obtain the first prediction features for the past content. The original features and the prediction features are added and input into the forward mapping module. The forward mapping module inputs the extracted second prediction features into the classification layer to obtain the behavior prediction results of the first-perspective video.

7. A storage medium for storing a program for executing the first-view video behavior prediction method based on cycle consistency as claimed in any one of claims 1 to 3.

8. A client, used in the first-person perspective video behavior prediction system based on cycle consistency as claimed in any one of claims 4 to 6.

9. A method for automatic vehicle driving, characterized in that: The road condition video collected by the vehicle is used as the first-perspective video, and the first-perspective video behavior prediction method based on cycle consistency of claim 3 is adopted to obtain the behavior prediction result of the specified target in the road condition video to control the vehicle to perform the automatic driving task.

Images