Early behavior prediction method and system based on forward guidance and bidirectional optimization loss

Abstract

The application discloses an early behavior prediction method and system based on prospective guidance and bidirectional optimization loss, wherein the method comprises the following steps: acquiring an activity behavior video of a user to be predicted; extracting a three-dimensional skeleton from each frame image of the video to obtain an overall three-dimensional skeleton sequence corresponding to the video; dividing the overall three-dimensional skeleton sequence corresponding to the video into a plurality of parts to obtain a plurality of local three-dimensional skeleton sequences; and inputting a random local three-dimensional skeleton sequence into a trained early behavior prediction model to output an early behavior prediction result label; wherein the total loss function of the trained early behavior prediction model is the sum of an isomorphic prospective learning loss function and a bidirectional joint robustness loss function.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of early behavior prediction, and particularly to an early behavior prediction method and system based on forward guidance and bidirectional optimization loss. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] The early behavior prediction task aims to predict the corresponding behavior label after observing part of the sequence, which is conducive to improving the experience of video security and human-computer interaction. Since the mainstream behavior recognition algorithm cannot complete the task well, it is very meaningful to study the solution specifically for the early behavior prediction task.

[0004] With the development of depth sensors, existing behavior recognition research has explored various data modalities such as RGB video, optical flow, and three-dimensional skeleton. Among them, when expressing the same behavior, the three-dimensional skeleton has the characteristics of small data volume and strong robustness. These characteristics make the behavior recognition algorithm based on skeleton data have considerable research prospects.

[0005] The early behavior prediction task mainly has the following three challenges due to the limited observed behavior feature information:

[0006] (1) Lack of effective behavior feature information. As shown in Figure 1 , the observed sequence with an observation ratio of 0.5 has very limited human posture change context information compared to the complete sequence (observation ratio of 1.0), so the spatiotemporal change features extracted by the network model are often insufficient to distinguish human behaviors.

[0007] (2) Different categories of early sequence features are similar. As shown in Figure 1 , for the two action sequences of "brushing teeth" and "drinking water", when the observation ratio is 0.5, the sequence lacks obvious distinguishing features, and these partial sequences are easily mispredicted, belonging to difficult samples.

[0008] (3) There are noise problems in some early skeleton sequences. As shown in Figure 1 , for 3D skeleton data, due to the lack of object and other detailed information, when the observation ratio is 0.2, the features of the two actions of brushing teeth and drinking water are almost identical and do not contain key distinguishing features. Such samples are called twins sample pairs.

[0009] Recently, many studies have proposed solutions to address the problems existing in the early behavior prediction task. Some prior art introduces a teacher-student network architecture to extract complete sequence information from the teacher network to the student network, uses partial sequence information to improve recognition performance, and alleviates the problem of lack of effective features. Some prior art establishes a difficult sample library and uses the idea of an adversarial network to specially train a high-performance classifier to improve the problem of similar features. The above studies have achieved certain results, but they only consider a single problem, and more importantly, they do not consider the influence of noise data, which has certain limitations.

[0010] In the process of implementing the present application, the inventors have found that the prior art has the following technical problems:

[0011] Since the existing solutions do not comprehensively consider the problems of lack of effective features, increase of difficult samples, and interference of noise data, the early behavior prediction task has certain limitations. SUMMARY

[0012] In order to solve the problems of the prior art, the present application provides an early behavior prediction method and system based on forward guidance and bidirectional optimization loss. In order to further improve the prediction effect of the early behavior prediction task, the present application solves the problems of lack of effective features, increase of difficult samples, and interference of noise data.

[0013] In one aspect, an early behavior prediction method based on forward guidance and bidirectional optimization loss is provided.

[0014] The early behavior prediction method based on forward guidance and bidirectional optimization loss comprises:

[0015] An activity behavior video of a user to be predicted is obtained, a three-dimensional skeleton is extracted from each frame of the video to obtain an overall three-dimensional skeleton sequence corresponding to the video, and the overall three-dimensional skeleton sequence corresponding to the video is divided into a plurality of parts to obtain a plurality of local three-dimensional skeleton sequences.

[0016] A random local three-dimensional skeleton sub-sequence is input into a trained early behavior prediction model, and an early behavior prediction result label is output.

[0017] The trained early behavior prediction model has a total loss function that is the sum of an isomorphic forward learning loss function and a bidirectional joint robustness loss function.

[0018] In another aspect, an early behavior prediction system based on forward guidance and bidirectional optimization loss is provided.

[0019] The early behavior prediction system based on forward guidance and bidirectional optimization loss comprises:

[0020] An acquisition module configured to: acquire a video of an activity behavior of a user to be predicted; extract a three-dimensional skeleton from each frame image of the video to obtain an overall three-dimensional skeleton sequence corresponding to the video; divide the overall three-dimensional skeleton sequence corresponding to the video into a plurality of parts to obtain a plurality of local three-dimensional skeleton sequences;

[0021] A prediction module configured to: input each of the plurality of local three-dimensional skeleton sequences into the trained early behavior prediction model to output an early behavior prediction result label.

[0022] The total loss function of the trained early behavior prediction model is a sum of an isomorphic forward-looking learning loss function and a bidirectional joint robustness loss function.

[0023] In another aspect, an electronic device is also provided, comprising:

[0024] a memory for non-transiently storing computer readable instructions; and

[0025] a processor for running the computer readable instructions,

[0026] When the computer readable instructions are run by the processor, the method of the first aspect is executed.

[0027] In another aspect, a storage medium is also provided, which non-transiently stores computer readable instructions, wherein when the non-transient computer readable instructions are executed by a computer, the instructions of the method of the first aspect are executed.

[0028] In another aspect, a computer program product is also provided, comprising a computer program for implementing the method of the first aspect when run on one or more processors.

[0029] One of the above technical solutions has the following advantages or beneficial effects:

[0030] First, to address the problem of lack of effective feature information, an isomorphic forward-looking learning module (IFLM) is proposed, which dynamically adjusts the learning difficulty of global information through sharpness softening, thereby facilitating the model to fully exploit potential global behavior information and inter-class correlation prior knowledge.

[0031] Secondly, in order to solve the problems of noise data interference and insufficient learning of difficult samples, the application proposes a bidirectional joint robustness loss (BJRL); the module contains two noise-robust loss functions, which increase the inter-class difference of samples by optimizing the predicted values of target classes and non-target classes, respectively, and promote the model to learn the feature difference between difficult samples.

[0032] Finally, the multi-aspect fusion framework is evaluated on two public behavior recognition datasets, and has higher accuracy and more comprehensive coverage ability compared with existing schemes.

[0033] The application summarizes some problems existing in the use of human 3D skeleton information for early behavior prediction tasks. The application proposes an early behavior prediction method based on an isomorphic lookahead learning module and a bidirectional joint robustness loss. First, the method tries to alleviate the learning difficulty of potential global information in the lookahead body by balancing the sharpness, and promotes the output distribution of the early part sequence to approximate the complete sequence, thereby improving the prediction effect.

[0034] The application first improves the underfitting problem existing in the learning of difficult samples by the network from the noise angle in the early behavior prediction task. The multi-aspect fusion framework uses a bidirectional joint robustness loss to improve the anti-noise ability of the model through two mutually complementary robustness loss functions, and alleviates the problem of insufficient learning of difficult samples.

[0035] The application completes the test on two public mainstream behavior recognition dataset benchmarks. The results show that the method proposed in the application can effectively improve the prediction performance, and the test results exceed the current best results. BRIEF DESCRIPTION OF DRAWINGS

[0036] The drawings accompanying the specification of the application form part of the application and serve to provide further understanding of the application, the illustrative embodiments of the application and their description serve to explain the application without constituting an improper limitation thereof.

[0037] Figure 1 It is an inter-class different observation progress skeleton feature comparison chart for example one;

[0038] Figure 2 It is a whole block diagram of the early behavior prediction method based on the isomorphic lookahead learning module and the bidirectional joint robustness loss for example one;

[0039] Figure 3 It is a channel-level topology refinement graph convolutional network structure diagram for example one;

[0040] Figure 4 It is a feature extraction block structure diagram for example one;

[0041] Figure 5 CTR-GC structure diagram for example one;

[0042] Figure 6 Behavior prediction result comparison chart for example one. DETAILED DESCRIPTION

[0043] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0044] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0045] All data acquisition of the present embodiment is based on compliance with laws and regulations and user consent, and legal application of data.

[0046] As Figure 2 The present application proposes an early behavior prediction method based on an isomorphic forward-looking learning module and a bidirectional joint robustness loss, which comprehensively considers multiple challenges existing in the early behavior prediction task. First, in order to alleviate the problem of insufficient feature information, existing technical research has proved that using complete sequence pre-training information to guide partial sequence learning is helpful to improve the effect of early behavior prediction. However, these studies ignore the influence of pre-training model output distribution sharpness on model training. When the sharpness is too large, the model is easy to ignore the correlation prior knowledge between different categories in the complete sequence, which leads to the lack of generalization of the model and hinders the progress of the model to learn the potential global information. Therefore, the present application proposes an isomorphic forward-looking learning module (IFLM), which uses the global features obtained by pre-training the complete sequence as forward-looking information, and uses a sharpness balancing factor to soften the output of the forward-looking information. According to the training stage of the network and the observation ratio of the partial sequence, the sharpness gap between the model and the forward-looking information is dynamically balanced, and the label information is used to guide the network training together, so as to maximize the expectation to reduce the distribution difference between the model and the forward-looking information, and promote the model to fully learn the potential global feature change information, thereby improving the effect of early behavior prediction.

[0047] Secondly, the presence of noise samples exacerbates the degree of underfitting in difficult sample learning. To this end, the present invention proposes a bidirectional optimization loss module (BJLM), which constructs a noise-robust loss function through normalization operations and penalizes twin sample pairs as a special type of data perturbation. At the same time, in order to improve the problem of insufficient learning of difficult samples, the module combines two types of noise-robust loss functions, which respectively maximize the probability of the target class and minimize the probability of the non-target class, thereby promoting the model to fully explore the subtle gaps between categories, thereby increasing the inter-class distance and improving the prediction effect. Experimental results show that the proposed multi-faceted fusion framework is beneficial to early behavior prediction, especially when the behavior is in a very early stage.

[0048] For early behavior prediction tasks, the observable sequence always describes the behavior that occurs before it is fully executed.

[0049] The focus of this invention is to comprehensively improve multiple problems existing in early behavior prediction tasks. Figure 2 As shown, the method adopts an end-to-end training method in the field of deep learning, and uses CTR-GCN as the feature extraction network in the framework to generate Figure 2 The global feature and partial feature in S are transformed by using the isomorphic forward learning module. F The output logits obtained through pre-training are sharpness-balanced and then input into the learning loss function to alleviate the difficulty of learning forward-looking body information, promote the network to fully explore potential global behavioral information and the semantic information of correlations between different categories, and improve prediction errors caused by the lack of information about human posture changes. In addition, the present invention uses a noise-robust loss module to reduce the model's attention to noisy data, improving the learning confusion caused by twin sample pairs. At the same time, the module simultaneously optimizes the target class and non-target classes, expanding the distance between classes while reducing the distance within classes, and improving the learning deficiency problem of difficult samples.

[0050] Example 1

[0051] This embodiment provides an early behavior prediction method based on forward guidance and bidirectional optimization loss;

[0052] Early behavior prediction methods based on forward guidance and bidirectional optimization loss, including:

[0053] S101: Obtain a video of the user's activity behavior to be predicted; extract a 3D skeleton from each frame of the video to obtain an overall 3D skeleton sequence corresponding to the video; evenly divide the overall 3D skeleton sequence corresponding to the video into several parts according to time periods to obtain several local 3D skeleton sequences;

[0054] S102: input a random one local three-dimensional skeleton sub-sequence into the trained early behavior prediction model, and output an early behavior prediction result label;

[0055] The total loss function of the trained early behavior prediction model is the sum of the isomorphic forward learning loss function and the bidirectional joint robustness loss function.

[0056] Further, S101: obtaining an activity behavior video of a user to be predicted; extracting a three-dimensional skeleton from each frame image of the video to obtain an overall three-dimensional skeleton sequence corresponding to the video; dividing the overall three-dimensional skeleton sequence corresponding to the video into several parts according to time periods to obtain several local three-dimensional skeleton sequences; and using a Kinect visual sensor to track and collect a 3D human skeleton sequence.

[0057] Further, the trained early behavior prediction model comprises:

[0058] a first feature extraction model and a second feature extraction model;

[0059] The first feature extraction model is configured to input the overall three-dimensional skeleton sequence, perform feature extraction on the overall three-dimensional skeleton sequence, and obtain global features.

[0060] The output end of the first feature extraction model is connected to the input end of the isomorphic forward learning loss function module.

[0061] The second feature extraction model is configured to input the local three-dimensional skeleton sequence, perform feature extraction on the local three-dimensional skeleton sequence, and obtain local features.

[0062] The output end of the second feature extraction model is connected to the input end of a classifier, the output end of the classifier is connected to the input end of a bidirectional joint robustness loss function module, and the classifier is configured to output an early behavior prediction label.

[0063] The output end of the second feature extraction model is also connected to the input end of the isomorphic forward learning loss function module.

[0064] The isomorphic forward learning loss function module is configured to deploy an isomorphic forward learning loss function.

[0065] The bidirectional joint robustness loss function module is configured to deploy a bidirectional joint robustness loss function.

[0066] Further, the training process of the trained early behavior prediction model comprises:

[0067] constructing a training set, wherein the training set comprises overall three-dimensional skeleton sequences and local three-dimensional skeleton sequences with known early behavior labels;

[0068] inputting the training set into the early behavior prediction model to train the model;

[0069] stopping the training when the total loss function value of the model no longer decreases, to obtain the trained early behavior prediction model.

[0070] The training process is as follows Figure 2 Partial sequences and complete sequences are simultaneously input into the feature extraction network of the same architecture.

[0071] During the training process, the global feature parameters are frozen and will not be updated in reverse. The goal of the global feature model is to provide a more accurate probability distribution as a global feature, serving as another goal for network learning.

[0072] Loss calculation: input the output of the second feature extraction model and the output of the first feature extraction model into the learning loss, and input the partial feature network output and the label into the bidirectional joint loss to calculate the loss value.

[0073] Backpropagation: according to the loss function, calculate the gradient of each parameter to the loss. Through the chain rule, the gradient is transmitted from the output layer to the input layer, and each parameter is updated.

[0074] Parameter update: use the optimization algorithm SGD to update the network parameters according to the gradient and the learning rate.

[0075] Perform multiple iterations of training until the network model converges or reaches the preset number of training rounds.

[0076] Further, as shown in Figure 3 , the internal structure of the first feature extraction model and the second feature extraction model is the same, and the first feature extraction model comprises:

[0077] a normalization layer, a first feature extraction block, a second feature extraction block, a third feature extraction block, a fourth feature extraction block, a fifth feature extraction block, a sixth feature extraction block, a seventh feature extraction block, an eighth feature extraction block, a ninth feature extraction block, and a tenth feature extraction block connected in sequence.

[0078] Further, the working process of the first feature extraction model comprises:

[0079] After the data is normalized, it is input into ten feature extraction blocks of the same structure connected in series. Different convolution channel numbers and step lengths of the feature extraction blocks are used to introduce multi-scale receptive fields at different levels of the network, and through the acquisition and processing of different scale information, the recognition and expression ability of the network for multi-scale objects is improved.

[0080] Further, as shown in Figure 4As shown, the internal structures of the first feature extraction block, the second feature extraction block, the third feature extraction block, the fourth feature extraction block, the fifth feature extraction block, the sixth feature extraction block, the seventh feature extraction block, the eighth feature extraction block, the ninth feature extraction block and the tenth feature extraction block are the same, and the first feature extraction block comprises:

[0081] The first input layer is connected with the input end of the first CTR-GC module, the input end of the second CTR-GC module and the input end of the third CTR-GC module respectively;

[0082] The output end of the first CTR-GC module, the output end of the second CTR-GC module and the output end of the third CTR-GC module are connected with the input end of the first adder;

[0083] The output end of the first adder is connected with the input end of the batch normalization layer, and the output end of the batch normalization layer is connected with the input end of the first activation function layer;

[0084] The output end of the first activation function layer is connected with the input end of the second adder;

[0085] The output end of the second adder is connected with the input end of the first convolutional layer, the input end of the second convolutional layer, the input end of the third convolutional layer and the input end of the fourth convolutional layer respectively;

[0086] The input end of the first convolutional layer is connected with the input end of the fifth convolutional layer, and the output end of the fifth convolutional layer is connected with the input end of the first series connection splicing unit;

[0087] The input end of the second convolutional layer is connected with the input end of the sixth convolutional layer, and the output end of the sixth convolutional layer is connected with the input end of the first series connection splicing unit;

[0088] The input end of the third convolutional layer is connected with the input end of the maximum pooling layer, and the output end of the maximum pooling layer is connected with the input end of the first series connection splicing unit;

[0089] The output end of the fourth convolutional layer is connected with the input end of the first series connection splicing unit;

[0090] The output end of the first series connection splicing unit is connected with the input end of the second adder, and the output end of the second adder is connected with the input end of the second activation function layer;

[0091] The first input layer is also connected with the input end of the second adder.

[0092] Further, the working process of the first feature extraction block comprises:

[0093] The data is first subjected to a spatial feature extraction block, a spatial feature between skeleton points is extracted in combination with an adjacency matrix A in a spatial dimension, and finally subjected to a temporal feature extraction block to obtain temporal features by using 1D convolution in a time dimension.

[0094] Further, the internal structures of the first CTR-GC module, the second CTR-GC module and the third CTR-GC module are the same, and the first CTR-GC module comprises:

[0095] The second input layer is connected with the input end of the seventh convolutional layer, the input end of the eighth convolutional layer and the input end of the ninth convolutional layer respectively.

[0096] The output end of the seventh convolutional layer is connected with the input end of the multiplier;

[0097] The output end of the eighth convolutional layer is connected with the input end of the first pooling layer;

[0098] The output end of the ninth convolutional layer is connected with the input end of the second pooling layer;

[0099] The output end of the first pooling layer and the output end of the second pooling layer are connected with the input end of the second series connection splicing unit;

[0100] The output end of the second series connection splicing unit is connected with the input end of the linear layer;

[0101] The output end of the linear layer is connected with the input end of the tenth convolutional layer;

[0102] The output end of the tenth convolutional layer is connected with the input end of the third adder;

[0103] The output end of the third adder is connected with the input end of the multiplier;

[0104] The output end of the multiplier is the output end of the first CTR-GC module.

[0105] Further, the working process of the first CTR-GC module comprises:

[0106] The data is subjected to two linear change modules to reduce the feature dimension, then subjected to a stacking operation, a linear layer, 1x1 convolution, and finally subjected to channel topology refinement modeling with the adjacency matrix A, and the calculation overhead is reduced.

[0107] Channel-wise Topology Refinement Graph Convolution (CTR-GCN), in order to improve the problem of insufficient learning of difficult samples, the method of the present application introduces a channel-level topology refinement graph convolution network with stronger feature fitting capability to learn high-level semantic information features. It improves the upper limit of the model in mining subtle differences between difficult samples by simultaneously learning shared topology and channel-specific correlation strength, thereby improving the problem of insufficient learning of difficult samples.

[0108] Figure 3 Description: Figure 3 BN normalization layer, 10 feature extraction blocks (B1-B10). The three numbers in the parentheses represent the number of input channels, output channels and strides respectively. Among them, the feature extraction block is shown in Figure 4 . The structure diagram of CTR-GCN is shown in Figure 5 .

[0109] Figure 4 Description: The feature extraction block is composed of a channel topology refinement space module and a multi-scale time sequence module.

[0110] Further, the trained early behavior prediction model, the total loss function is the sum of the isomorphic forward learning loss function and the bidirectional joint robustness loss function, specifically including:

[0111]

[0112] Among them, L represents the total loss function, L1 represents the isomorphic forward learning loss function, L2 represents the bidirectional joint robustness loss function, i represents the i-th training sample, N represents the total number of training samples in the current batch, r i represents the observation ratio corresponding to the i-th training sample; represents the learning loss value of the i-th training sample; represents the bidirectional joint optimization loss value of the i-th training sample.

[0113] Further, the isomorphic forward learning loss function specifically includes:

[0114] Set a complete sequence sample containing T frames as S F ={t1,t2,…,t T}, according to the observation ratio r i , S F is divided into M different level sub-sequences, denoted as represents the result of rounding down, r i ∈(0,1], S r contains the number of observable sequence frames

[0115] Get S F The feature encoding is represented as E(Θ F ,S F ), S r The feature encoding is represented as E(Θ,S r );

[0116] The sharpness balance factor α is calculated using the sharpness factor generator (SFG):

[0117]

[0118] Where C represents the number of behavior categories, Represents the output value of the fully connected layer of the second feature extraction network;

[0119] Finally, the probability distribution p(S F ) is expressed as:

[0120]

[0121] Among them, Z i The output value of the first feature extraction network.

[0122] In measuring E(Θ,S r ) and E(Θ F ,S F ), JS divergence is used as the loss function:

[0123]

[0124] Where L1 represents the isomorphic forward learning loss function; p(S r ) is the probability distribution of the feature extraction network output, D KL Denotes KL divergence:

[0125]

[0126] Where C represents the number of categories;

[0127] It should be understood that the isomorphic prospective learning module is expected to be P The network model used as input can fully learn the potential global behavior change information in the forward-looking body and improve S P The problem of insufficient effective feature information. Some studies have solved this problem by directly converting S F With S P The logits input loss function is learned, while ignoring the difficulty of learning the forward-looking information. In the early stages of training or when the observations are small, SF With S P There is a large information gap, which makes it difficult to learn the correlation information between the target class and the non-target class in the lookahead body, making the model prediction more arbitrary, losing the generalization ability of feature recognition, and causing a severe degradation problem.

[0128] The information gap mainly reflects the sharpness of the global feature output distribution, which is the source of learning difficulty and the main reason for insufficient learning. When the sharpness of the lookahead body output distribution is too large, the probability between the target class and the non-target class in the prior knowledge approaches the non-0, i.e. 1 relationship, and the model cannot fully grasp the inter-class correlation information of the lookahead body information. Therefore, by introducing the concept of sharpness balance to soften the lookahead body output distribution, the learning difficulty of the lookahead body information is dynamically relieved by the sharpness balance factor.

[0129] Further, the bidirectional joint robustness loss function specifically includes:

[0130]

[0131] Where P represents the loss contribution of the target class, Q represents the loss contribution of the non-target class; C represents the total number of classes, And Indicates the label distribution of the corresponding c class, i.e. one-hot encoding; γ represents the balance parameter; Indicates the partial sequence of class c. Indicates the probability distribution output by the feature extraction network.

[0132]

[0133] L2=μ*TOL+ν*NTOL (9)

[0134] Where μ,ν>0 is used to balance the parameters of the two terms.

[0135] It should be understood that in order to further improve the prediction effect, the present application proposes a bidirectional joint robustness loss module. The module is composed of a loss function with noise robustness, and the module treats twin samples as a kind of label noise for processing, which is used to reduce the interference received by the model when learning difficult samples. As Figure 2 The bidirectional robustness loss module optimizes the output probability of the target class and the non-target class respectively, helps the model fully exploit the difference information between samples, and alleviates the problem of insufficient learning of difficult samples.

[0136] For early behavior prediction tasks, twin samples are invalid training samples, which will cause certain sample imbalance problems for the model. In view of the problem, the application proposes to use Focal loss instead of the conventional cross-entropy loss function.

[0137]

[0138] Wherein gamma is an adjustment factor, q(c|S P ) represents the label distribution of sample S P , and p(S P ) represents the prediction probability distribution of the model.

[0139] According to the definition, any loss function L can be robust to noise after being standardized by formula (6).

[0140]

[0141] Wherein y is the ground truth, and f is the probability distribution output by the model.

[0142] The Focal loss is standardized to construct a target class optimization loss function (TOL) which is robust to noise samples, alleviates the influence of noise and guides the model to learn the information of the target class, and accelerates the network convergence. TOL only maximizes the output probability p(c=y|S P ) of the network at the corresponding class position. Wherein y is the behavior label.

[0143] The general form of TOL is approximately represented as cross-entropy loss function For early behavior prediction tasks, sequence samples similar to target class samples may cause the maximization process of p(c=y|S P ) to stop, that is, the P term is fixed. At this time, in order to minimize the empirical risk, the Q value may increase, which means that even if the loss is reduced, the model may not learn the difference information between samples sufficiently. Therefore, the application designs N-TOL as a penalty for non-target class prediction, which is used to minimize p(c=y|S P ) at non-target class position, and suppresses the increase of Q term. N-TOL uses the mean absolute error (MAE) based loss function as the non-target class optimization loss function (N-TOL), and the mathematical theory is fully proved that MAE has good robustness to noise data.

[0144] In the training stage, the application adopts the observation ratio r i Coupling strategy combines formula (3) and (9) to update the feature encoder E(Theta, S P) and the parameters of the classifier. Finally, the complete objective function for training the multi-aspect fusion framework is shown as follows:

[0145]

[0146] During the test, the sequences with different observation ratios r i and the full-length sequences are sent into the network, and the commonly used Top-1 evaluation standard is used to evaluate the classification accuracy according to different observation ratios.

[0147] The scheme proposed in the present application is evaluated on the mainstream NTU RGB+D dataset and SYSU 3D Human-Object Interaction (3DHOI) dataset.

[0148] For the NTU RGB+D dataset, there are single-person and double-person cases in the dataset, and the present application uses 0 padding for samples with only one person. The present application uses the first 150 frames of each sample for experiments, and discards the part exceeding 150 frames in some samples. The same process is performed on the SYSU 3DHOI dataset, and the frame number is set to 400. In addition, the present application rotates the skeletons of the two datasets (shoulder parallel to X axis, spine parallel to Z axis).

[0149] Each sample in the dataset is re-divided according to the observation ratio r i , r i increases from 0.1 to 1.0 with a step of 0.1. Finally, the re-divided samples are mixed and shuffled to input the network for training. It should be noted that due to the small number of samples in the SYSU dataset, the fitting ability of CTR-GCN is strong, and the model is prone to overfitting during the training process. Therefore, the present application uses the first twenty skeleton points of the NTU-60 dataset for pre-training, which is used as the initial weight parameter for training the SYSU dataset.

[0150] All experiments are performed on NVIDIA RTX 3090 with the PyTorch deep learning framework. The feature extraction network does not use pre-trained weight parameters, and all network layers use kaiming initialization. The model of the present application is trained using the SGD optimizer with a momentum of 0.9 and a weight decay of 0.0004. The training epoch is set to 50, and the preheating strategy is used in the first 5 epochs to make the training process more stable. The learning rate is set to 0.1, and is decayed by a factor of 0.1 at epoch 25 and 35. The batch size is 128.

[0151] Results on NTU RGB+D dataset: The present application follows the cross-subject test protocol and cross-view test protocol to evaluate the trained model on the NTU RGB+D dataset. The results of the CS evaluation protocol are shown in Figure 3 . The results following the CV protocol are shown in Figure 3 . As shown in Figure 5 , the method of the present application is superior to all the most advanced methods.

[0152] CV protocol results. For the NTU-60 dataset, the present application also evaluates the model of the present application on the CV protocol, and the accuracy of the model proposed by the present application is 54.60%, 86.55%, 93.53%, 94.95% and 94.90% on the observation ratio of 20%, 40%, 60%, 80% and 100%, respectively. From Figure 6 , it can be seen that the method proposed by the present application is much better than other methods, which proves the effectiveness of the method of the present application.

[0153] Results on SYSU 3DHOI dataset: The present application further evaluates the method of the present application on the SYSU dataset. The SYSU dataset has 30 training / test dataset division methods, and in each split, the samples of 20 people are used for training, and the samples of the other 20 people are used for testing. The present application trains and tests each of the 30 different division schemes respectively, and takes the average of all results as the final experimental result. Comparison of the method of the present application with other most advanced methods can see that on the SYSU dataset, the method of the present application is still superior to all the recent methods. The method of the present application reaches an accuracy of 67.35%, 83.70%, 89.65%, 91.93%, 92.13% on the observation ratio of 20%, 40%, 60%, 80% and 100%, respectively.

[0154] Ablation experiment: The present application conducts an ablation experiment on the NTU RGB+D dataset using the CS protocol. Next, the present application will analyze the influence of the isomorphic forward learning module (IFLM) and the bidirectional joint robustness loss (BJRL) according to the experimental results.

[0155] Influence of feature extraction network: The present application uses ST-GCN, AAGCN, AAGCN, MS-G3D as the feature extraction network to conduct experiments on the NTU RGB+D dataset, respectively.

[0156] Due to the lightweight advantage of the skeleton data, the network of the present application runs very fast. The results show that the model of the present application is fast enough and can be used for real-time behavior prediction.

[0157] Figure 6The prediction results of different behaviors at different observation ratios are shown by a multi-aspect fusion framework (MPFF) model of the application.

[0158] The application proposes an early behavior prediction method based on an isomorphic prospective learning module and a bidirectional joint robustness loss. In the framework of the method, the application proposes an IFLM to eliminate learning difficulty through a dynamic sharpness softening factor, minimize the global distribution knowledge difference between the prospective body model and the training model to extract class prior knowledge. In addition, in order to alleviate the noise problem existing in the data, the application proposes a BJRL to adopt two loss functions robust to noise to promote the model to fully mine the subtle gap between difficult samples, so as to improve the early action prediction model. Experimental results show that the model trained by the method of the application realizes a highly competitive performance compared with the most advanced method.

[0159] Embodiment two

[0160] The embodiment provides an early behavior prediction system based on prospective guidance and bidirectional optimization loss;

[0161] The early behavior prediction system based on prospective guidance and bidirectional optimization loss comprises:

[0162] An acquisition module configured to: acquire an activity behavior video of a user to be predicted; extract a three-dimensional skeleton from each frame image of the video to obtain an overall three-dimensional skeleton sequence corresponding to the video; divide the overall three-dimensional skeleton sequence corresponding to the video into a plurality of parts to obtain a plurality of local three-dimensional skeleton sequences;

[0163] A prediction module configured to: input a random one of the local three-dimensional skeleton sub-sequences into the trained early behavior prediction model, and output an early behavior prediction result label.

[0164] The total loss function of the trained early behavior prediction model is the sum of the isomorphic prospective learning loss function and the bidirectional joint robustness loss function.

[0165] It should be noted that the acquisition module and the prediction module correspond to steps S101 to S102 in Embodiment One, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment One. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0166] The description of each embodiment in the above embodiments has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0167] The proposed system can be implemented in other ways. For example, the system embodiments described above are only illustrative, and the division of the above modules is only a logical functional division. In actual implementation, there can be another division manner, for example, a plurality of modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0168] Embodiment three

[0169] The embodiment also provides an electronic device, including one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method in the above embodiment one.

[0170] It should be understood that, in the embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, ready programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0171] The memory can include read-only memory and random access memory, and provide instructions and data for the processor, and a part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0172] In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software.

[0173] The method in the embodiment one can be directly embodied as hardware processor execution completion, or combined execution completion by hardware and software modules in the processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, and other mature storage media in the field. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0174] Those skilled in the art can understand that the units and algorithm steps of the examples described in combination with the embodiments can be realized in electronic hardware or in combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner depends on the specific application and design constraints of the technical solutions. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0175] Embodiment Four

[0176] The embodiment also provides a computer readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment One.

[0177] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An early behavior prediction method based on prospective guidance and bidirectional optimization loss, characterized in that, The method comprises the following steps: obtaining an activity behavior video of a user to be predicted; extracting a three-dimensional skeleton from each frame image of the video to obtain an overall three-dimensional skeleton sequence corresponding to the video; and dividing the overall three-dimensional skeleton sequence corresponding to the video into a plurality of parts to obtain a plurality of local three-dimensional skeleton sequences; inputting a random local three-dimensional skeleton sub-sequence into a trained early behavior prediction model to output an early behavior prediction result label; wherein the total loss function of the trained early behavior prediction model is a sum of an isomorphic forward learning loss function and a bidirectional joint robustness loss function; The isomorphic forward learning loss function specifically includes: setting a complete sequence sample containing T frames as , according to the observation ratio Will Divided into M subsequences with different progress, expressed as , , Indicates that the result is rounded down. , The number of observable sequence frames included ; Get The feature encoding is represented as , The feature encoding is represented as ; Use the sharpness factor generator to calculate the sharpness balance factor α: ; wherein C represents the number of behavior categories, represents the output value of the full connection layer of the second feature extraction model; finally, the probability distribution output by the global feature after the sharpness balance represents: ; wherein, Z i is the output value of the first feature extraction model; in measuring the distribution difference between and JS divergence is used as the loss function: ； wherein, denotes the homogenous forward-looking learning loss function; is a probability distribution output by the second feature extraction model, denotes the KL divergence: wherein represents the number of classes; the bidirectional joint robustness loss function specifically comprises: ； wherein C denotes the total number of classes, and denotes the label distribution of the corresponding c-th class; γ denotes a balancing parameter; denotes the partial sequence of the c-th class; denotes the probability distribution output by the second feature extraction model for the c-th class; ； ； wherein , is a parameter for balancing and .

2. The early behavior prediction method based on look-ahead guidance and bidirectional optimization loss of claim 1, wherein, the trained early behavior prediction model comprises a first feature extraction model and a second feature extraction model; the first feature extraction model is configured to input the overall three-dimensional skeleton sequence, perform feature extraction on the overall three-dimensional skeleton sequence, and obtain global features; an output end of the first feature extraction model is connected to an input end of the isomorphic forward learning loss function module; the second feature extraction model is configured to input the local three-dimensional skeleton sequence, perform feature extraction on the local three-dimensional skeleton sequence, and obtain local features; an output end of the second feature extraction model is connected to an input end of a classifier, and an output end of the classifier is connected to an input end of the bidirectional joint robustness loss function module; the classifier is configured to output an early behavior prediction label; wherein the output end of the second feature extraction model is also connected to the input end of the isomorphic forward learning loss function module.

3. The early behavior prediction method based on prospective guidance and bidirectional optimization loss of claim 1, wherein, The training process of the trained early behavior prediction model comprises: constructing a training set, wherein the training set comprises overall three-dimensional skeleton sequences and local three-dimensional skeleton sequences with known early behavior labels; inputting the training set into the early behavior prediction model to train the model; stopping the training when the total loss function value of the model no longer decreases, and obtaining the trained early behavior prediction model.

4. The early behavior prediction method based on prospective guidance and bidirectional optimization loss of claim 1, wherein, The total loss function of the trained early behavior prediction model is a sum of the isomorphic forward learning loss function and the bidirectional joint robustness loss function, and specifically comprises: ； wherein, represents a total loss function, represents an isomorphic forward learning loss function, represents a bidirectional joint robustness loss function, represents the i-th training sample, represents the i-th training sample, represents the total number of training samples in the current batch, represents the observation ratio corresponding to the i-th training sample; represents the learning loss value of the i-th training sample; represents the bidirectional joint optimization loss value of the i-th training sample.

5. The early behavior prediction method based on prospective guidance and bidirectional optimization loss of claim 2, wherein, the internal structures of the first feature extraction model and the second feature extraction model are the same, and the first feature extraction model comprises, in sequence, a normalization layer, a first feature extraction block, a second feature extraction block, a third feature extraction block, a fourth feature extraction block, a fifth feature extraction block, a sixth feature extraction block, a seventh feature extraction block, an eighth feature extraction block, a ninth feature extraction block, and a tenth feature extraction block; the working process of the first feature extraction model comprises: after data is subjected to a normalization operation, the data is input into ten feature extraction blocks with the same structure and repeated connection, different convolution channel numbers and step lengths of the feature extraction blocks are used to introduce multi-scale receptive fields at different levels of the network, and different scale information is acquired and processed.

6. An early behavior prediction system based on prospective guidance and bidirectional optimization loss, characterized in that, The method comprises the following steps: a obtaining module configured to obtain an activity behavior video of a user to be predicted; extracting a three-dimensional skeleton from each frame image of the video to obtain an overall three-dimensional skeleton sequence corresponding to the video; and dividing the overall three-dimensional skeleton sequence corresponding to the video into a plurality of parts to obtain a plurality of local three-dimensional skeleton sequences; The prediction module is configured to input each of the random one local three-dimensional skeleton sub-sequence into the trained early behavior prediction model, and output an early behavior prediction result label. The total loss function of the trained early behavior prediction model is a sum of an isomorphic forward-looking learning loss function and a bidirectional joint robustness loss function. The isomorphic forward learning loss function specifically includes: setting a complete sequence sample containing T frames as , according to the observation ratio Will Divided into M subsequences with different progress, expressed as , , Indicates that the result is rounded down. , The number of observable sequence frames included ; Get The feature encoding is represented as , The feature encoding is represented as ; Use the sharpness factor generator to calculate the sharpness balance factor α: ; wherein C represents the number of behavior categories, represents the output value of the full connection layer of the second feature extraction model; finally, the probability distribution output by the sharpness balance of the global feature represents: ; wherein, Z i is the output value of the first feature extraction model; in measuring the distribution difference between and the JS divergence is used as the loss function: ； wherein, represents the homomorphic forward-looking learning loss function; is a probability distribution output by the second feature extraction model, represents the KL divergence: wherein represents the number of classes; The bidirectional joint robustness loss function specifically includes: ； wherein C denotes the total number of classes, and denotes the label distribution of the corresponding c-th class; γ denotes a balancing parameter; denotes the partial sequence of the c-th class; denotes the probability distribution output by the second feature extraction model for the c-th class. ； ； wherein , is a parameter for balancing and .

7. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; and a processor for executing the computer readable instructions, wherein the computer readable instructions, when executed by the processor, perform the method of any one of claims 1-5. computer readable instructions, wherein the non-transitory computer readable instructions, when executed by a computer, perform the method of any one of claims 1-5.

8. A storage medium, characterized by a non-transitory ​

Images