Generative visual large model-based behavior prediction method and system

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a behavior prediction method and system based on a generative visual large model. The method comprises the steps of collecting human body action video data, and extracting human body behavior action data; preprocessing the data; constructing a generative visual large model, performing feature extraction, and performing time sequence analysis by using a long-short-term memory network; performing end-to-end training, calculating a loss function, and obtaining a personnel behavior prediction model; and outputting prediction results of the human body behavior actions in a plurality of future time frames. According to the method, high-level features can be extracted by adopting the generative visual model, so that the efficiency and accuracy of feature extraction are improved; a long-short-term memory network is adopted to perform time sequence analysis on the feature vectors, so that the accuracy and stability of action prediction are improved, the dynamic change of the action can be captured, and the future action trend can be predicted; the performance of the whole model can be optimized by adopting an end-to-end training method, the efficiency and robustness are improved, and the method adapts to variable practical application environments.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular, relates to a behavior prediction method and system based on a generative visual large model. Background Art

[0002] Human behavior prediction uses computer vision and machine learning to predict future movements and behaviors by analyzing the dynamic information of the human body. This technology has broad application prospects, including intelligent monitoring, health monitoring, human-computer interaction, sports training and other fields.

[0003] Human behavior prediction plays an important role in many aspects. First, human behavior prediction can be used for security monitoring. By predicting the behavior of people in places such as airports, train stations, shopping malls and water stations, potential dangers or abnormal behaviors can be identified in advance, thus improving the level of public safety.

[0004] The existing personnel behavior prediction methods have the following defects: (1) Complex data acquisition and processing: The acquisition of human motion data requires high-precision sensor equipment, large data volume, and complex processing. Especially in a multi-camera and multi-sensor environment, data synchronization and fusion are difficult. (2) The accuracy and real-time performance of existing algorithms are poor, especially in complex practical environments. The robustness and adaptability of algorithms still need to be enhanced; (3) Difficulty in fusing multimodal data: Human motion behavior prediction often requires the fusion of multimodal data such as visual and infrared sensors. How to effectively fuse these data to improve prediction accuracy and stability is an important challenge; (4) Personalized prediction is difficult: There are significant differences in the actions and behaviors of different individuals. How to conduct personalized behavior prediction to adapt to the characteristics of different individuals is a problem that needs to be solved. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a behavior prediction method and system based on a generative visual large model.

[0006] In a first aspect, the present invention provides a behavior prediction method based on a generative visual large model, comprising: Collect human action video data and extract human action data; human action includes the position and posture of key joints of the human body, action data and interactive objects; action data includes motion data and interactive action data; Preprocess the human behavior data and construct a sample data set; Build a generative visual big model, use the generative visual big model to extract features from the video frames of the preprocessed video data in the sample data set, use the long short-term memory network to perform time series analysis on the feature vectors, and generate time series features; Use the sample data set to perform end-to-end training on the generative vision model, calculate the loss function of the generative vision model, adjust the parameters of the generative vision model according to the loss function, obtain the target training model, and fix the feature extraction network parameters in the target training model to obtain the personnel behavior prediction model; The human behavior prediction model is loaded into the online prediction system, and the new video data is preprocessed and input into the human behavior prediction model for prediction, and the prediction results of human behavior actions in several future time frames are output.

[0007] In a second aspect, the present invention provides a behavior prediction system based on a generative visual large model, including an acquisition unit, a preprocessing unit, a model building unit, a model training unit and a loading unit; The acquisition unit is used to collect human action video data and extract human action data; human action includes the posture of key joints of the human body, action data and interactive objects; the action data includes motion data and interactive action data; A preprocessing unit, used for preprocessing human behavior action data and constructing a sample data set; A model building unit, used to build a generative visual big model, use the generative visual big model to extract features from video frames of preprocessed video data in a sample data set, use a long short-term memory network to perform time series analysis on feature vectors, and generate time series features; A model training unit is used to perform end-to-end training on the generative vision model using a sample data set, calculate the loss function of the generative vision model, adjust the parameters of the generative vision model according to the loss function, obtain a target training model, and fix the feature extraction network parameters in the target training model to obtain a personnel behavior prediction model; The loading unit is used to load the human behavior prediction model into the online prediction system, pre-process the new video data and input it into the human behavior prediction model for prediction, and output the prediction results of human behavior actions in several future time frames.

[0008] Based on the above technical solution, the present invention can also be improved as follows.

[0009] Furthermore, the human behavior action data is preprocessed to construct a sample data set including: Use filtering algorithms to denoise video data and remove noise from the video; Extract key frames from the video based on key moments of action changes; Use the posture estimation algorithm to detect the key joints of the human body, use the detection and segmentation algorithm to detect the environmental object information, and obtain the coordinate information of the key joints and the coordinate information of the objects in the environment; According to the coordinate information of the joint points and the coordinate information of the objects in the environment, the probability of the interaction between the human body and the object is calculated.

[0010] Furthermore, the interaction object is a human body or an environmental object; the motion data includes walking, running and jumping; and the interaction action data includes grabbing, extracting and putting down.

[0011] Furthermore, the generative visual large model encodes the input frames and then embeds the images in batches into a convolutional neural network for feature extraction. The mask after mask decoding outputs the prediction results of human behavior actions in several future time frames.

[0012] Furthermore, the generative visual big model includes an image marker, a video marker, a text marker, an image encoder, a video encoder, a text encoder, a Transformer network decoder, a generator and an output unit; the image marker is connected to the Transformer network decoder through the image encoder; the video marker is connected to the Transformer network decoder through the video encoder; the text marker is connected to the Transformer network decoder through the text encoder; the transformer decoder is connected to the output unit through the generator; the generative visual big model encodes the image, video stream and text description information, then performs multimodal information fusion, then decodes it through the Transformer network decoder, and finally generates the human behavior action and the corresponding text description at the next moment through the generator.

[0013] Furthermore, the long short-term memory network includes an input gate, a forget gate and an output gate; the input gate is used to store the current information in the cell state; the forget gate is used to forget the information from the hidden state of the previous time step and the current input; the output gate is used to decide to output the information from the current cell state to the next hidden state; the cell state is updated through the forget gate and the input gate, and the value of the cell state is adjusted by the hyperbolic tangent activation function.

[0014] Furthermore, the cross entropy loss function value is calculated. When the cross entropy loss function value is less than the set threshold, the target training model is obtained.

[0015] The beneficial effects of the present invention are: (1) The present invention uses a generative visual model trained on a large-scale data set, has a powerful feature extraction capability, can automatically extract high-level features from raw video data, avoids the complex manual feature design process, and improves the efficiency and accuracy of feature extraction; (2) Long short-term memory networks are used to perform time series analysis on feature vectors to capture the temporal dependency of actions and process complex time series data, thereby improving the accuracy and stability of action prediction. In addition, by performing time series modeling on feature vectors of multiple video frames, dynamic changes in actions can be captured and future action trends can be predicted. (3) The present invention improves the prediction accuracy of different individual action behaviors through generative visual model training. Through special training on individual data, the generative visual model can provide personalized predictions and suggestions; (4) The end-to-end training method can optimize the performance of the entire model, improve efficiency and robustness, and adapt to changing practical application environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a behavior prediction method based on a generative visual large model provided in Example 1 of the present invention; Figure 2 A flowchart of a behavior prediction method based on a generative visual large model; Figure 3 This is the structural diagram of the generative vision model; Figure 4 Schematic diagram of the feature extraction network for the generative visual large model; Figure 5 Schematic diagram of a behavior prediction system based on a generative visual large model provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0018] Example 1 As an example, Figure 1 As shown, in order to solve the above technical problems, this embodiment provides a behavior prediction method based on a generative visual large model, including: Collect human action video data and extract human action data; human action includes the position and posture of key joints of the human body, action data and interactive objects; action data includes motion data and interactive action data; Preprocess the human behavior data and construct a sample data set; Build a generative visual big model, use the generative visual big model to extract features from the video frames of the preprocessed video data in the sample data set, use the long short-term memory network to perform time series analysis on the feature vectors, and generate time series features; Use the sample data set to perform end-to-end training on the generative vision model, calculate the loss function of the generative vision model, adjust the parameters of the generative vision model according to the loss function, obtain the target training model, and fix the feature extraction network parameters in the target training model to obtain the personnel behavior prediction model; The human behavior prediction model is loaded into the online prediction system, and the new video data is preprocessed and input into the human behavior prediction model for prediction, and the prediction results of human behavior actions in several future time frames are output.

[0019] In actual application, a multi-camera system or a single camera is used to obtain human motion video data to ensure that the video data has a high frame rate and clarity. The collected video data includes human motion videos in different water station scenes, such as walking, grabbing, extracting and putting down videos.

[0020] Optionally, preprocess the human behavior data to construct a sample data set including: Use filtering algorithms to denoise video data and remove noise from the video; Extract key frames from the video based on key moments of action changes; Use the posture estimation algorithm to detect the key joints of the human body, use the detection and segmentation algorithm to detect the environmental object information, and obtain the coordinate information of the key joints and the coordinate information of the objects in the environment; According to the coordinate information of the joint points and the coordinate information of the objects in the environment, the probability of the interaction between the human body and the object is calculated.

[0021] In actual application, as shown in the attached Figure 2 The flowchart of the behavior prediction method based on the generative visual large model is shown. Extracting key frames from the video can reduce redundant data and improve processing efficiency. The human behavior action prediction obtains the posture of the key joints of the human body, the behavior action data and the interaction object. The interaction object is an object in the environment, such as scissors and tape. The human behavior action is to take the scissors, and the predicted human behavior action is "cut the tape with scissors".

[0022] Calculate the probability of human-object interaction to determine the impact of the corresponding object in the environment on the next action of the human body. For example, a person is walking, and there is a table and a trash can on the left and right in front of him, blocking his way. The table and the trash can are both obstacles, and the person's behavior is to cross the obstacles. Calculate the probability of collision between the human body and the two obstacles. During the crossing process, the probability of collision with the table is the highest, and the feasibility of passing is the lowest, so it is predicted that the next human behavior action is to cross the trash can.

[0023] The long short-term memory network is used to perform time series analysis on feature vectors, capture the temporal dependency of actions, process complex time series data, and improve the accuracy and stability of action prediction. In addition, by performing time series modeling on feature vectors of multiple video frames, the dynamic changes of actions can be captured and future action trends can be predicted. The end-to-end training method can optimize the performance of the entire model, improve efficiency and robustness, and adapt to the changing actual application environment.

[0024] Optionally, the interaction object is a human body or an environmental object; the motion data includes walking, running and jumping; and the interaction action data includes grabbing, extracting and putting down.

[0025] Optionally, the generative visual model encodes the input frame and then embeds the images in batches into a convolutional neural network for feature extraction, and the mask after mask decoding outputs the prediction results of human behavior actions for several future time frames.

[0026] Optional, as attached Figure 3 As shown in the figure, the generative visual model includes an image marker, a video marker, a text marker, an image encoder, a video encoder, a text encoder, a Transformer network decoder, a generator and an output unit; the image marker is connected to the Transformer network decoder through the image encoder; the video marker is connected to the Transformer network decoder through the video encoder; the text marker is connected to the Transformer network decoder through the text encoder; the transformer decoder is connected to the output unit through the generator; the generative visual model encodes the image, video stream and text description information, then performs multimodal information fusion, and then decodes it through the Transformer network decoder, and finally generates the human behavior action and the corresponding text description at the next moment through the generator.

[0027] The image data is processed by the image tagger and image encoder, the video data is processed by the video tagger and video encoder, the text data is processed by the text tagger and text encoder, and the processed data is combined in the transformer decoder and generator to finally produce the output.

[0028] As attached Figure 4As shown in the figure, the image is first passed through the image encoder to extract features; these features generate image embeddings through convolution operations; the image embeddings are processed by the hint encoder; the mask decoder uses information from the image encoder, image embedding and hint encoder to generate the final mask; the hint encoder is used to process additional input information (such as text descriptions or key point coordinates) and combine this information with the image features to help the mask decoder generate more accurate masks.

[0029] Optionally, the LSTM network includes an input gate, a forget gate, and an output gate; the input gate is used to store the current information in the cell state; the forget gate is used to forget the information from the hidden state of the previous time step and the current input; the output gate is used to determine whether to output the information from the current cell state to the next hidden state; the cell state is updated through the forget gate and the input gate, and the value of the cell state is adjusted through the hyperbolic tangent activation function, as shown in the attached figure. Figure 5 As shown, Optionally, the cross entropy loss function value is calculated. When the cross entropy loss function value is less than a set threshold, the target training model is obtained.

[0030] Generative large models can learn rich image features, and then apply these features to specific tasks through transfer learning and can perform a variety of visual tasks. Moreover, generative large models are not limited to a single task, but can also perform multiple tasks simultaneously through multi-task learning, thereby improving the generalization ability of the model. These generative visual large models are trained with large-scale data and have powerful feature extraction and pattern recognition capabilities, providing new possibilities for human motion behavior prediction.

[0031] Long short-term memory networks can avoid the gradient vanishing or gradient exploding problems when processing time series data, thereby better capturing long-term dependencies.

[0032] The loss function is a function used to measure the difference between the model's prediction and the actual result. During the training process, the model's goal is to minimize the loss function, such as calculating the cross entropy loss. Assuming the actual label is , takes a value of 0 or 1, and the predicted label is , the predicted label is the probability that the model predicts the positive class (y=1), and the value range is [0, 1]. The cross entropy loss function formula is: .

[0033] In the early stages of model training, the value of the loss function is usually large. As training progresses, if the model converges normally, the loss value gradually decreases until the loss value is less than the set threshold.

[0034] The personnel behavior prediction method based on the generative visual big model provides a powerful and efficient solution for human motion behavior prediction. It can combine environmental objects to predict human behavior in various dynamic, complex and changeable human behaviors, such as walking, grasping, extracting and putting down, to achieve human motion behavior prediction, so as to meet the accurate prediction of human motion behavior in various application scenarios.

[0035] Example 2 Based on the same principle as the method shown in Example 1 of the present invention, the embodiment of the present invention also provides a behavior prediction system based on a generative visual large model, including a collection unit, a preprocessing unit, a model building unit, a model training unit and a loading unit; The acquisition unit is used to collect human action video data and extract human action data; human action includes the posture of key joints of the human body, action data and interactive objects; the action data includes motion data and interactive action data; A preprocessing unit, used for preprocessing human behavior action data and constructing a sample data set; A model building unit, used to build a generative visual big model, use the generative visual big model to extract features from video frames of preprocessed video data in a sample data set, use a long short-term memory network to perform time series analysis on feature vectors, and generate time series features; A model training unit is used to perform end-to-end training on the generative vision model using a sample data set, calculate the loss function of the generative vision model, adjust the parameters of the generative vision model according to the loss function, obtain a target training model, and fix the feature extraction network parameters in the target training model to obtain a personnel behavior prediction model; The loading unit is used to load the human behavior prediction model into the online prediction system, pre-process the new video data and input it into the human behavior prediction model for prediction, and output the prediction results of human behavior actions in several future time frames.

[0036] Optionally, preprocess the human behavior data to construct a sample data set including: Use filtering algorithms to denoise video data and remove noise from the video; Extract key frames from the video based on key moments of action changes; Use the posture estimation algorithm to detect the key joints of the human body, use the detection and segmentation algorithm to detect the environmental object information, and obtain the coordinate information of the key joints and the coordinate information of the objects in the environment; According to the coordinate information of the joint points and the coordinate information of the objects in the environment, the probability of the interaction between the human body and the object is calculated.

[0037] Optionally, the interaction object is a human body or an environmental object; the motion data includes walking, running and jumping; and the interaction action data includes grabbing, extracting and putting down.

[0038] Optionally, the generative visual model encodes the input frame and then embeds the images in batches into a convolutional neural network for feature extraction, and the mask after mask decoding outputs the prediction results of human behavior actions for several future time frames.

[0039] Optionally, the generative visual large model includes an image marker, a video marker, a text marker, an image encoder, a video encoder, a text encoder, a Transformer network decoder, a generator and an output unit; the image marker is connected to the Transformer network decoder via the image encoder; the video marker is connected to the Transformer network decoder via the video encoder; the text marker is connected to the Transformer network decoder via the text encoder; the transformer decoder is connected to the output unit via the generator; the generative visual large model encodes the image, video stream and text description information, then performs multimodal information fusion, then decodes it through the Transformer network decoder, and finally generates the human behavior action and the corresponding text description at the next moment through the generator.

[0040] Optionally, the long short-term memory network includes an input gate, a forget gate and an output gate; the input gate is used to store current information in the cell state; the forget gate is used to forget information from the hidden state of the previous time step and the current input; the output gate is used to decide to output information from the current cell state to the next hidden state; the cell state is updated through the forget gate and the input gate, and the value of the cell state is adjusted by the hyperbolic tangent activation function.

[0041] Optionally, the cross entropy loss function value is calculated. When the cross entropy loss function value is less than a set threshold, the target training model is obtained.

[0042] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A behavior prediction method based on a generative visual large model, characterized in that: include: Collect human motion video data and extract human behavior motion data; Human behavior includes the positions of key joints, behavior data and interaction objects; Behavior action data includes motion data and interactive action data; Preprocess the human behavior data and construct a sample data set; Build a generative visual big model, use the generative visual big model to extract features from the video frames of the preprocessed video data in the sample data set, use the long short-term memory network to perform time series analysis on the feature vectors, and generate time series features; Use the sample data set to perform end-to-end training on the generative vision model, calculate the loss function of the generative vision model, adjust the parameters of the generative vision model according to the loss function, obtain the target training model, and fix the feature extraction network parameters in the target training model to obtain the personnel behavior prediction model; The human behavior prediction model is loaded into the online prediction system, and the new video data is preprocessed and input into the human behavior prediction model for prediction, and the prediction results of human behavior actions in several future time frames are output.

2. The behavior prediction method based on the generative visual large model according to claim 1 is characterized in that: Preprocess the human behavior data and construct a sample data set including: Use filtering algorithms to denoise video data and remove noise from the video; Extract key frames from the video based on the key moments of action changes; Use the posture estimation algorithm to detect the key joints of the human body, use the detection and segmentation algorithm to detect the environmental object information, and obtain the coordinate information of the key joints and the coordinate information of the objects in the environment; According to the coordinate information of the joint points and the coordinate information of the objects in the environment, the probability of the interaction between the human body and the object is calculated.

3. The behavior prediction method based on generative visual big model according to claim 1 is characterized in that: The interaction object is the human body or environmental objects; the motion data includes walking, running and jumping; the interaction action data includes grabbing, extracting and putting down.

4. The behavior prediction method based on generative visual big model according to claim 1, characterized in that: The generative visual model encodes the input frame and then embeds the images in batches into a convolutional neural network for feature extraction. The mask after mask decoding outputs the prediction results of human behavior actions in several future time frames.

5. The behavior prediction method based on generative visual big model according to claim 1, characterized in that: The generative visual big model includes an image marker, a video marker, a text marker, an image encoder, a video encoder, a text encoder, a Transformer network decoder, a generator and an output unit; the image marker is connected to the Transformer network decoder through the image encoder; the video marker is connected to the Transformer network decoder through the video encoder; the text marker is connected to the Transformer network decoder through the text encoder; the transformer decoder is connected to the output unit through the generator; the generative visual big model encodes the image, video stream and text description information, then performs multimodal information fusion, then decodes it through the Transformer network decoder, and finally generates the human behavior action and the corresponding text description at the next moment through the generator.

6. The behavior prediction method based on generative visual big model according to claim 1, characterized in that: The long short-term memory network includes an input gate, a forget gate, and an output gate; the input gate is used to store the current information in the cell state; the forget gate is used to forget the information from the hidden state of the previous time step and the current input; the output gate is used to decide to output the information from the current cell state to the next hidden state; the cell state is updated through the forget gate and the input gate, and the value of the cell state is adjusted by the hyperbolic tangent activation function.

7. The behavior prediction method based on generative visual big model according to claim 1, characterized in that: Calculate the cross entropy loss function value. When the cross entropy loss function value is less than the set threshold, the target training model is obtained.

8. A behavior prediction system based on a generative visual large model, characterized by: It includes an acquisition unit, a preprocessing unit, a model building unit, a model training unit and a loading unit; The acquisition unit is used to collect human action video data and extract human action data; human action includes the posture of key joints of the human body, action data and interactive objects; the action data includes motion data and interactive action data; A preprocessing unit, used for preprocessing human behavior action data and constructing a sample data set; A model building unit, used to build a generative visual big model, use the generative visual big model to extract features from video frames of preprocessed video data in a sample data set, use a long short-term memory network to perform time series analysis on feature vectors, and generate time series features; A model training unit is used to perform end-to-end training on the generative vision model using a sample data set, calculate the loss function of the generative vision model, adjust the parameters of the generative vision model according to the loss function, obtain a target training model, and fix the feature extraction network parameters in the target training model to obtain a personnel behavior prediction model; The loading unit is used to load the human behavior prediction model into the online prediction system, pre-process the new video data and input it into the human behavior prediction model for prediction, and output the prediction results of human behavior actions in several future time frames.

Images