First-person video behavior prediction model training method based on counterfactual analysis

Abstract

The present invention provides a first-person perspective video behavior prediction model training method based on counterfactual analysis, which uses a first-person perspective video training set to perform multiple rounds of iterative training on the prediction model until convergence, wherein each round of training includes: extracting visual features and semantic features of each sample respectively; encoding and decoding the visual features and semantic features respectively to obtain a high-dimensional feature vector of the next moment corresponding to the visual features and a high-dimensional feature vector of the next moment corresponding to the semantic features; obtaining a counterfactual prediction vector and a fact prediction vector of the next moment based on the high-dimensional feature vectors; subtracting the counterfactual prediction vector from the fact prediction vector corresponding to each sample according to similarity to obtain a final behavior prediction vector of the next moment; and updating the parameters of the prediction model using the cross-entropy loss between the final prediction vector corresponding to the behavior of each sample at the next moment and the vector corresponding to the behavior label of the next moment.

Description

Technical Field

[0001] The present invention relates to the field of video image processing, specifically, to the field of first-person perspective video behavior prediction, and more specifically, to a first-person perspective video behavior prediction model training method based on counterfactual analysis, a behavior prediction method and a system. Background Art

[0002] First-person behavior prediction refers to predicting the next possible behavior based on a video clip that has already occurred. The solution represented by reference [1] is committed to extracting visual features from the input video clip and feeding this visual information into a behavior prediction model composed of two cascaded long and short-term memory networks to form a summary of the observed content and a hypothesis about future behavior, thereby making full use of this visual information to predict future behavior; the solution represented by reference [2] introduces the semantic information contained in the behavior label on the basis of visual features, and feeds the visual features and the semantic labels of past behaviors into the behavior prediction model to obtain a prediction result based on multimodal information, thereby improving the performance of behavior prediction by combining visual and semantic information; the solution represented by reference [3] considers that the semantic association between behaviors may cause the prediction model to produce semantic bias, that is, the uneven distribution of behavior labels in the data set makes the generated prediction results more biased towards high-frequency labels, and proposes a counterfactual analysis solution, which aims to weaken the side effects brought by the semantic association between behavior labels while retaining multimodal information, so that the model pays more attention to the visual content that can reflect the specific information of each case, thereby further improving the reliability of behavior prediction.

[0003] In comparison, the behavior prediction based on counterfactual analysis is more reliable than the other two schemes. However, when using the counterfactual analysis scheme to remove the bias caused by semantic information, the existing technology usually first obtains the prediction result Y of the factual stage based on visual information and semantic information. f , and then only use the semantic labels of past behaviors to predict future behaviors to obtain the prediction result Y of the counterfactual stage c Finally, the prediction results of the counterfactual stage are deducted from the prediction results of the factual stage to obtain the final prediction results (Y f -Y c ), this operation is to f and Y c Directly subtracting these two vectors without considering the differences between samples and categories will lead to overcorrection, causing the prediction accuracy of some categories to decrease after the introduction of counterfactual analysis, and failing to adaptively alleviate the bias caused by semantic information.

[0004] References:

[0005] [1]Antonino Furnari and Giovanni Maria Farinella.What Would YouExpect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs andModality Attention.In Proceedings of the IEEE International Conference onComputer Vision 2019: 6251-6260.

[0006] [2] Antoine Miech, lvan Laptev, Josef Sivic, Heng Wang, Lorenzo Torresani, and Du Tran.Leveraging the Present to Anticipate the Futurein Videos.lnProceedings of the I EEE Conference on Computer Vision and Pattern RecognitionWorkshops 2019.2915-2922.

[0007] [3] Tianyu Zhang, Weiqing Min, Jiahao Yang, Tao Liu, Shuqiang Jiang, and Yong Rui. What if we coUld not see? coUnterfactual analysis for egocentric action anticipation.ln Proceedings of the Thirtieth International Jo intConference on Artificial ln telligence 2021.1316-1322. Summary of the Invention

[0008] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a first-person video behavior prediction model training method based on counterfactual analysis, which can fully consider the differences between samples and between categories and avoid overcorrection, and a prediction model trained based on this method.

[0009] According to a first aspect of the present invention, a method for training a first-perspective video behavior prediction model based on counterfactual analysis is provided, wherein the prediction model includes a visual feature extraction module, a semantic feature extraction module, a visual feature encoding and decoding network, a semantic feature encoding and decoding network, a first linear layer, a second linear layer, and a behavior prediction module, and the method includes: S1, obtaining a first-perspective video training set, wherein the training set includes multiple video samples and each sample corresponds to a behavior label at the next moment corresponding to the video; S2, using the training set obtained in step S1 to perform multiple rounds of iterative training on the prediction model until convergence, wherein each round of training includes: S21, using a visual feature extraction module and a semantic feature extraction module to extract visual features and semantic features of each sample respectively; S22, using a visual feature encoding and decoding network to encode and decode the visual features of each sample to obtain a high-dimensional feature vector corresponding to the visual features of each sample at the next moment, and using a semantic feature encoding and decoding network to encode and decode the semantic features of each sample to obtain a high-dimensional feature vector corresponding to the semantic features of each sample at the next moment; S23, obtaining a first-perspective video training set, wherein the first-perspective video training set includes ... Use the first linear layer to process the high-dimensional feature vector of the next moment corresponding to the semantic feature of each sample to map it to the behavior category of the next moment to obtain the counterfactual prediction vector of the next moment corresponding to each sample, and use the second linear layer to process the concatenation result of the high-dimensional feature vector of the next moment corresponding to the visual feature of each sample and the high-dimensional feature vector of the next moment corresponding to the semantic feature to map it to the behavior category of the next moment to obtain the fact prediction vector of the next moment corresponding to each sample; S24, use the behavior prediction module to calculate the similarity between the counterfactual prediction vector of the next moment corresponding to each sample and the fact prediction vector, and use the similarity as the weight of the counterfactual prediction vector of the next moment corresponding to each sample, and subtract the counterfactual prediction vector of the next moment from the fact prediction vector of the next moment corresponding to each sample according to the corresponding weight to obtain the final behavior prediction vector corresponding to the behavior at the next moment; S25, update the parameters of the prediction model based on the cross entropy loss between the final prediction vector corresponding to the behavior of each sample at the next moment and the vector corresponding to the behavior label of the next moment.

[0010] Preferably, in step S24, the similarity between the counterfactual prediction vector and the factual prediction vector at the next moment corresponding to each sample is calculated in the following manner:

[0011]

[0012] Among them, Y f Represents the fact prediction vector of the next moment corresponding to the sample, Y c represents the counterfactual prediction vector of the next moment corresponding to the sample, ⊙ represents the element-by-element multiplication of the vector, ||Y f ||2 and ||Y c||2 represents vector Y f and vector Y c The modulus value of .

[0013] In some embodiments of the present invention, the similarity between the counterfactual prediction vector and the factual prediction vector at the next moment corresponding to each sample is normalized in the following manner:

[0014]

[0015] Among them, b1 and b2 are preset hyperparameters, corresponding to the lower and upper bounds of the interval after normalization, respectively, and b1<b2, and sigmoid is the activation function.

[0016] Preferably, it is characterized in that b1=-5 and b2=5.

[0017] Preferably, the final behavior prediction vector is calculated as follows:

[0018] g(Y f , Y c )=Y f -σ(Y f , Y c )·Y c

[0019] Among them, g(Y f , Y c ) represents the final prediction vector.

[0020] Preferably, the cross entropy loss is calculated as follows:

[0021]

[0022] Where N is the number of samples in the training set, Represents the vector corresponding to the behavior label of the i-th sample, (Y) i Represents the final behavior prediction vector of the i-th sample.

[0023] According to a second aspect of the present invention, a first-perspective video behavior prediction method based on counterfactual analysis is provided, the method comprising: T1, obtaining a video to be predicted; T2, using a prediction model trained by the method described in the first aspect of the present invention to predict the behavior of the video to be predicted at the next moment.

[0024] According to a third aspect of the present invention, a first-perspective video behavior prediction system based on counterfactual analysis is provided, which is used to predict the behavior at the next moment based on the video. The system includes: a visual feature extraction module for extracting visual features of the video; a semantic feature extraction module for extracting semantic features of the video; a visual feature encoding and decoding network for encoding and decoding the visual features of the video to obtain a high-dimensional feature vector at the next moment corresponding to the visual features of the video; a semantic feature encoding and decoding network for encoding and decoding the semantic features of the video to obtain a high-dimensional feature vector at the next moment corresponding to the semantic features of the video; a first linear layer for processing the high-dimensional feature vector at the next moment corresponding to the semantic features of the video The first linear layer is used to process the high-dimensional feature vector of the next moment corresponding to the semantic features of the video and the high-dimensional feature vector of the next moment corresponding to the visual features of the video to map them to the behavior category of the next moment to obtain the fact prediction vector of the next moment corresponding to the video; the behavior prediction module is used to calculate the similarity between the counterfactual prediction vector and the fact prediction vector corresponding to the next moment of the video, and use the similarity as the weight of the counterfactual prediction vector, and subtract the counterfactual prediction vector of the next moment of the video from the fact prediction vector of the next moment of the video according to the weight to obtain the final behavior prediction vector of the next moment of the video.

[0025] Preferably, the visual feature extraction module is a convolutional neural network model; the semantic feature extraction module is a Word2Vec word vector model;

[0026] The visual feature encoding and decoding network and the semantic feature encoding and decoding network are both LSTM encoding and decoding networks.

[0027] Compared with the existing technology, the advantages of the present invention are: the present invention fully considers the differences between samples and samples, labels and labels, adaptively analyzes the impact of counterfactual results on prediction behavior, and removes counterfactual results in the implementation prediction results based on the impact of counterfactual results on prediction behavior, thereby solving the problem of overcorrection and improving the accuracy of prediction. It also fully considers the differences between different samples and different categories, thereby improving the accuracy of model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0029] Figure 1 Schematic diagram comparing sample processing schemes in counterfactual analysis according to the prior art and in the counterfactual analysis of the present invention;

[0030] Figure 2 Schematic diagram of the prediction model training principle according to an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of the prediction model structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0033] In order to better understand the present invention, the concept of counterfactual analysis is first briefly introduced.

[0034] In first-person video behavior prediction, when the behavior prediction model is used to predict the behavior label of the next moment, the information that can be used includes the visual features and behavior labels of the previous moment to obtain the factual prediction result. The problem with this approach is that the model tends to remember the semantic association between the behavior labels of the previous moment and the next moment, and ignores the mining of specific visual content. To this end, in the existing technology, researchers construct counterfactual scenarios: assuming that nothing is seen (that is, there are no visual features as input to the model), the model only receives the behavior label of the previous moment as input, and obtains counterfactual prediction results. This counterfactual prediction result captures the side effects brought about by the semantic association between behavior labels. What counterfactual analysis does is to remove the counterfactual prediction result from the factual prediction result (technically, it is the subtraction of two vectors) to obtain a behavior prediction result that is closer to the visual content.

[0035] As described in the background art, the existing technology of counterfactual analysis of first-person video behavior prediction has the problem of overcorrection, which limits the accuracy of the behavior prediction model.

[0036] The inventors found in the process of studying the prediction of first-person video behavior based on counterfactual analysis that the traditional counterfactual analysis scheme only uses a simple subtraction operation Y f -Y c , without considering the differences between samples and samples, and between labels and labels, resulting in a considerable number of samples having prediction results after counterfactual analysis that are worse than the results before counterfactual analysis. To this end, the present invention proposes an adaptive counterfactual analysis scheme that fully considers the differences between samples and samples, and between labels and labels, and adaptively analyzes the impact of counterfactual results on prediction behavior to obtain weight coefficients, and based on the weight coefficients, removes counterfactuals from different dimensions of the counterfactual results. For example, assuming there are N samples in the data set, and the feature vector corresponding to each sample has m categories (m dimensions), such as Figure 1As shown in (a), the traditional counterfactual analysis scheme does not consider the differences between categories for the same sample, and directly subtracts each dimension of the counterfactual results (vertically, each sample vector has m categories, and the coefficient subtracted at each element position is 1). This processing method does not consider the differences between samples of the same category (horizontally, each category corresponds to N sample vectors, and the coefficient subtracted at each element position is 1). However, in reality, the impact of semantic deviations between labels is not always negative, and specific analysis is required to make full use of semantic deviations, that is, to adaptively choose to amplify or reduce the impact of semantic deviations. In the present invention, the inventor determines the degree of semantic deviation to be deducted by calculating the similarity between the factual prediction vector and the counterfactual prediction vector, as shown in FIG. Figure 1 As shown in (b), it is ensured that the coefficients deducted for different categories of a given sample and different samples of a given category are different (i.e., the deduction coefficients for different categories in each sample are different, and the deduction coefficients for the same category in different samples are also different). Personalized output vectors are generated for different input samples. This fully considers the differences between samples and the differences between labels, implements an adaptive counterfactual analysis scheme, and can effectively avoid the overcorrection phenomenon that may be caused by traditional counterfactual analysis schemes. In addition, the adaptive counterfactual analysis method of the present invention fully considers the differences between different samples and the differences between different categories, can alleviate the deviation caused by language information, and improves the accuracy of the model.

[0037] As is known to all, the working principle of behavior prediction based on first-person video is to use the video of a historical moment as the observation content to predict future behavior (action), and to train the prediction model by using a video training set (each sample in the training set is a video, and each sample contains a behavior label), so that the prediction model can extract the information in the video of the previous moment by processing the observation content, and predict the action of the next moment based on the extracted information. In order to solve the problem of overcorrection based on traditional counterfactual analysis, the present invention adds adaptive counterfactual analysis to each sample in the training process to obtain better behavior prediction results. According to one embodiment of the present invention, Figure 2 As shown, the prediction process of the first-person video behavior based on adaptive counterfactual analysis of the present invention can be summarized as follows:

[0038] First, for each sample's observation content (the video at the previous moment), the video's semantic tag embedding (also called semantic features) and visual feature embedding (also called visual features) are obtained. According to one embodiment of the present invention, a convolutional neural network is used to obtain the visual features in the video, and a Word2Vec word vector model is used to obtain the semantic features in the video. Convolutional neural networks and word vector models are well known to those skilled in the art and will not be described in detail in this invention.

[0039] Then, the semantic features of the sample are encoded and decoded to map the semantic information in the video at the previous moment to the next moment to obtain a high-dimensional feature vector corresponding to the semantic features at the next moment; similarly, the visual features of the sample are encoded and decoded to map the visual features in the video at the previous moment to the next moment to obtain a high-dimensional feature vector corresponding to the visual features at the next moment. According to one embodiment of the present invention, a long short-term memory codec network is used to encode and decode the semantic features, and another long short-term memory codec network is used to encode the visual features. Long short-term memory codec networks are well known to those skilled in the art and will not be described in detail here.

[0040] Secondly, based on the high-dimensional feature vector of the next moment corresponding to the semantic features of the sample and the high-dimensional feature vector of the next moment corresponding to the visual features, the factual stage prediction and the counterfactual stage prediction are respectively obtained to obtain the factual prediction vector Y of the sample. f and the counterfactual prediction vector Y c , for the fact prediction vector Y f and the counterfactual prediction vector Y c Adaptive counterfactual analysis is performed to obtain the final behavior prediction vector of the sample. The prediction of the fact stage is to concatenate the high-dimensional feature vector of the next moment corresponding to the semantic feature and the high-dimensional feature vector of the next moment corresponding to the visual feature and then send it to the first linear layer to map it to the behavior category of the next moment to obtain the m-dimensional fact prediction vector Y f , this fact predicts the vector Y f Represents the prediction result based on visual information and semantic information, which is mixed with the prediction bias caused by semantic information. The prediction of the counterfactual stage refers to sending only the high-dimensional feature vector of the next moment corresponding to the semantic feature to the second linear layer to map it to the behavior category of the next moment to obtain the m-dimensional counterfactual prediction vector Y c , the predicted result Y c The adaptive counterfactual analysis phase uses the similarity between the actual and counterfactual predictions to determine the degree of semantic bias removed, ensuring that the coefficients removed are different for different samples. The actual prediction vector is then updated based on the counterfactual prediction vector to obtain the final prediction result.

[0041] According to one embodiment of the present invention, in the adaptive counterfactual analysis stage, the fact prediction vector Y is first calculated f and the counterfactual prediction vector Y c The similarity vector between:

[0042]

[0043] Where ⊙ represents the element-by-element multiplication of vectors, ||Y f ||2 and ||Y c ||2 represents Y f and Y c The modulus value of , “·” represents the dot product, It is an m-dimensional similarity vector that can fully reflect the impact of semantic bias on fact prediction results. Since the values ​​of each position in the similarity vector are different, the larger the value of a category, the greater the impact of semantic bias on the fact prediction results of that category. The modulus of the vector is the square root of the sum of the squares of each element in the vector. For example, for an m-dimensional vector X = [x1, x2, ... x m ], its modulus is According to an example of the present invention, it is assumed that:

[0044]

[0045] Then the similarity vector between them is:

[0046]

[0047] From this, it can be seen that by calculating the similarity, the semantic bias influence of each category in the counterfactual prediction vector can be obtained, and the semantic bias influence of each category is different, thereby achieving the difference distinction between different samples of different categories.

[0048] According to one embodiment of the present invention, in order to better train the model, the present invention uses a monotonic function σ(·) to normalize the similarity vector to fix it within a preset fixed interval:

[0049]

[0050] Where b1 and b2 are hyperparameters, corresponding to the lower and upper bounds of the normalized interval, respectively. They can be any integer. According to one embodiment of the present invention, b1 = -5 and b2 = 5. Based on the previous example, the result after normalization is:

[0051]

[0052] Finally, the fact vector is updated according to the counterfactual prediction vector, and the final behavior prediction result is obtained as follows:

[0053] g(Y f , Y c )=Y f -σ(Y f , Y c )·Y c

[0054] Where g(Y f , Y c ) represents the final behavior prediction vector after being updated based on the counterfactual prediction vector, and the updated vector is used as the final prediction vector Y. This adaptive counterfactual analysis ensures that the coefficients deducted for different categories of a given sample, as well as for different samples within a given category, are different. It generates personalized output vectors for different input samples, fully accounting for the differences between samples and between labels. This allows the model to adaptively amplify or minimize the impact of semantic bias, achieving adaptive counterfactual analysis.

[0055] By performing the above adaptive counterfactual analysis on each sample in the training set, the parameters of the prediction model are updated based on the cross entropy loss between the final behavior prediction vector and the vector of the sample's actual behavior label:

[0056]

[0057] Where N is the number of samples in the training set, Represents the true behavior label vector of the i-th sample, (Y) i represents the final behavior prediction vector of the i-th sample, and (Y) i are all m-dimensional vectors, (Y) i =[Y1...Y j ,...Y m ] i , where the value of the j-th dimension (1≤j≤m) represents the probability that the sample belongs to the j-th category.

[0058] The prediction model obtained after multiple iterative training of the prediction model based on this training method can repeatedly consider different categories of semantic deviations in the counterfactual results, differentially eliminate the counterfactual results, and obtain better prediction results.

[0059] Based on the description of the previous embodiment, it can be seen that after training, the following Figure 3The prediction model includes a visual feature extraction module, a semantic feature extraction module, a visual feature encoding and decoding network, a semantic feature encoding and decoding network, a first linear layer, a second linear layer, and a behavior prediction module. For any video, when the prediction model trained by the present invention is used to predict the behavior of the video at the next moment, the visual feature extraction module extracts the visual features of the video, and the visual feature encoding and decoding network encodes and decodes the visual features to obtain a high-dimensional feature vector of the next moment corresponding to the visual features of the video; at the same time, the semantic feature extraction module extracts the semantic features of the video, and the semantic feature encoding and decoding network encodes and decodes the semantic features to obtain a high-dimensional feature vector of the next moment corresponding to the semantic features of the video; then the first linear layer encodes and decodes the semantic features of the video. The high-dimensional feature vector of the next moment corresponding to the feature is processed to map it to the behavior category of the next moment to obtain the counterfactual prediction vector of the next moment corresponding to the video, and the second linear layer processes the high-dimensional feature vector of the next moment corresponding to the semantic feature of the video and the high-dimensional feature vector of the next moment corresponding to the visual feature to map them to the behavior category of the next moment to obtain the fact prediction vector of the next moment corresponding to the video; finally, the behavior prediction module calculates the similarity between the counterfactual prediction vector of the next moment corresponding to the video and the fact prediction vector, and uses the similarity as the weight of the counterfactual prediction result, and subtracts the counterfactual prediction result of the next moment of the video from the fact prediction result of the next moment of the video according to the weight to obtain the final behavior prediction result of the next moment of the video.

[0060] The prediction model trained based on the method of the present invention no longer directly removes the counterfactual results in the counterfactual analysis stage, but instead removes the counterfactual results from the factual prediction results in a differentiated manner based on the semantic deviations of different categories in the counterfactual results, thereby obtaining a better behavior prediction effect and avoiding overcorrection.

[0061] In order to verify the effectiveness of the present invention, the inventors verified the prediction accuracy of the method of the present invention and the traditional counterfactual scheme on different data sets. The verification results on the EGTEA Gaze+ data set (the EGTEA Gaze+ data set contains 106 types of actions) are shown in Table 1, and the verification results on the EPIC-Kitchens-100 data set (the EPIC-Kitchens-100 data set contains 3806 types of actions) are shown in Table 2.

[0062] Table 1

[0063] Prediction accuracy Traditional counterfactual analysis scheme 16.72％ Adaptive Counterfactual Analysis Scheme 17.46％

[0064] Table 2

[0065] Prediction accuracy Traditional counterfactual analysis scheme 2.17％ Adaptive Counterfactual Analysis Scheme 2.44％

[0066] It can be seen from the results in Table 1 and Table 2 that the adaptive counterfactual analysis scheme of the present invention can significantly improve the prediction accuracy.

[0067] The present invention fully considers the differences between samples and samples, and between labels, adaptively analyzes the impact of counterfactual results on predicted behavior, and removes counterfactual results from the implemented prediction results based on the impact of counterfactual results on predicted behavior, thereby solving the problem of overcorrection and improving the accuracy of prediction. It also fully considers the differences between different samples and different categories, thereby improving the accuracy of the model.

[0068] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0069] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0070] Computer-readable storage media can be a tangible device that holds and stores the instructions used by an instruction execution device. Computer-readable storage media can, for example, include, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, for example, and any suitable combination thereof.

[0071] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a first-person video behavior prediction model based on counterfactual analysis, wherein the prediction model comprises a visual feature extraction module, a semantic feature extraction module, a visual feature encoding and decoding network, a semantic feature encoding and decoding network, a first linear layer, a second linear layer, and a behavior prediction module, characterized in that: The method includes: S1. Obtain a first - perspective video training set, where the training set includes multiple video samples and each sample corresponds to a behavior label at the next moment of the corresponding video; S2. Use the training set obtained in step S1 to perform multiple rounds of iterative training on the prediction model until convergence. Among them, each round of training includes: S21. Use the visual feature extraction module and the semantic feature extraction module to extract the visual feature and semantic feature of each sample respectively; S22. Use the visual feature encoding - decoding network to perform encoding - decoding processing on the visual feature of each sample to obtain a high - dimensional feature vector at the next moment corresponding to the visual feature of each sample, and use the semantic feature encoding - decoding network to perform encoding - decoding processing on the semantic feature of each sample to obtain a high - dimensional feature vector at the next moment corresponding to the semantic feature of each sample; S23. Use the first linear layer to process the high - dimensional feature vector at the next moment corresponding to the semantic feature of each sample to map it to the behavior category at the next moment to obtain a counterfactual prediction vector at the next moment corresponding to each sample, and use the second linear layer to process the concatenation result of the high - dimensional feature vector at the next moment corresponding to the visual feature of each sample and the high - dimensional feature vector at the next moment corresponding to the semantic feature of each sample to map it to the behavior category at the next moment to obtain a factual prediction vector at the next moment corresponding to each sample; S24. Use the behavior prediction module to calculate the similarity between the counterfactual prediction vector and the factual prediction vector at the next moment corresponding to each sample, and use the similarity as the weight of the counterfactual prediction vector at the next moment corresponding to each sample, and subtract the counterfactual prediction vector at the next moment from the factual prediction vector at the next moment corresponding to each sample according to the corresponding weight to obtain the final behavior prediction vector corresponding to the behavior at the next moment; S25. Update the parameters of the prediction model based on the cross - entropy loss between the final prediction vector corresponding to the behavior at the next moment of each sample and the vector corresponding to the behavior label at the next moment.

2. The method according to claim 1, characterized in that In step S24, calculate the similarity between the counterfactual prediction vector and the factual prediction vector at the next moment corresponding to each sample in the following way: Among them, Y f Represents the fact prediction vector of the next moment corresponding to the sample, Y c represents the counterfactual prediction vector of the next moment corresponding to the sample, ⊙ represents the element-by-element multiplication of the vector, ||Y f ||2 and ‖Y c ‖2 represents vector Y f and vector Y c The modulus value of .

3. The method according to claim 2, characterized in that In step S24, normalize the similarity between the counterfactual prediction vector and the factual prediction vector at the next moment corresponding to each sample in the following way: Among them, b1 and b2 are preset hyperparameters, corresponding to the lower bound and upper bound of the interval after normalization respectively, and b1 < b2, and sigmoid is the activation function.

4. The method according to claim 3, characterized in that b1 = - 5, b2 = 5.

5. The method according to claim 4, characterized in that The final behavior prediction vector is calculated in the following way: g(Y f ,AND c )=And f -σ(Y f ,AND c )·AND c Among them, g(Y f ,Y c ) represents the final prediction vector.

6. The method according to claim 5, characterized in that The cross - entropy loss is calculated in the following way: Where N is the number of samples in the training set, Represents the vector corresponding to the behavior label of the i-th sample, (Y) i Represents the final behavior prediction vector of the i-th sample.

7. A method for predicting behavior in first-person video based on counterfactual analysis, characterized in that: The method includes: T1. Obtain the video to be predicted; T2. Use the prediction model trained by the method described in any one of claims 1 - 6 to predict the behavior of the video to be predicted at the next moment.

8. A first-person video behavior prediction system based on counterfactual analysis, used to predict the behavior at the next moment based on the video, characterized in that: The system includes: A visual feature extraction module for extracting the visual feature of the video; A semantic feature extraction module for extracting the semantic feature of the video; A visual feature encoding - decoding network for performing encoding - decoding processing on the visual feature of the video to obtain a high - dimensional feature vector at the next moment corresponding to the visual feature of the video; The semantic feature encoding and decoding network is used to encode and decode the semantic features of the video to obtain the high-dimensional feature vector corresponding to the semantic features of the video at the next moment; The first linear layer is used to process the high-dimensional feature vector of the next moment corresponding to the semantic feature of the video to map it to the behavior category of the next moment to obtain the counterfactual prediction vector of the next moment corresponding to the video; The second linear layer is used to process the high-dimensional feature vector of the next moment corresponding to the semantic features of the video and the high-dimensional feature vector of the next moment corresponding to the visual features of the video to map them to the behavior category of the next moment to obtain the fact prediction vector of the next moment corresponding to the video; The behavior prediction module is used to calculate the similarity between the counterfactual prediction vector and the fact prediction vector corresponding to the next moment of the video, and use the similarity as the weight of the counterfactual prediction vector. The counterfactual prediction vector of the next moment of the video is subtracted from the fact prediction vector of the next moment of the video according to the weight to obtain the final behavior prediction vector of the next moment of the video.

9. The system according to claim 8, characterized in that The visual feature extraction module is a convolutional neural network model; The semantic feature extraction module is a Word2Vec word vector model; The visual feature encoding and decoding network and the semantic feature encoding and decoding network are both LSTM encoding and decoding networks.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any one of the methods of claims 1-6 and 7.

11. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the steps of the method according to any one of claims 1-6 and 7.

Images