Behavior recognition analysis system based on multi-modal large model
By leveraging intelligent self-learning, cross-domain knowledge transfer, and multi-task learning of multimodal large models, the problem of insufficient accuracy and real-time performance caused by single-modal data processing in existing technologies has been solved, enabling high-precision behavior recognition and real-time response in complex environments.
Patent Information
- Application Number
- CN202611257351.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-19
- Publication Date
- 2026-11-03
AI Technical Summary
Existing behavior recognition systems mostly focus on processing single-modal data, lacking deep fusion and intelligent optimization of cross-modal data, resulting in insufficient accuracy and real-time performance in complex environments.
The model is updated using intelligent self-learning and incremental learning modules, and cross-modal data fusion is achieved by combining cross-domain knowledge transfer and hybrid deep learning modules. Multi-task collaborative optimization is performed through multi-task learning modules in complex environments and real-time context awareness and behavior optimization modules, and semantic hierarchical behavior recognition and reasoning modules.
It significantly improves the model's generalization ability and behavior recognition accuracy in complex scenarios, achieves real-time response and high-precision behavior recognition, dynamically adapts to environmental changes, and improves the efficiency of multi-task collaboration.
Smart Images
Figure CN123030879A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically to a behavior recognition and analysis system based on a multimodal large model. Background Technology
[0002] With the rapid development of artificial intelligence technology, behavior recognition systems based on multimodal data such as computer vision, sensor data, and voice recognition are gradually becoming a core component of various intelligent applications. Existing behavior recognition systems mostly focus on processing single-modal data, lacking deep fusion and intelligent optimization of cross-modal data, resulting in insufficient accuracy and real-time performance in complex environments. Summary of the Invention
[0003] To address the shortcomings of existing technologies, this invention provides a behavior recognition and analysis system based on a multimodal large model. This system solves the problem that existing behavior recognition systems are mostly focused on processing single-modal data and lack deep fusion and intelligent optimization of cross-modal data, resulting in insufficient accuracy and real-time performance in complex environments.
[0004] The above-mentioned technical objective of the present invention is achieved through the following technical solution: A behavior recognition and analysis system based on a multimodal large model includes: The intelligent self-learning and incremental learning module is used to incrementally update the behavior recognition model and perform adaptive learning based on time series data. The cross-domain knowledge transfer and hybrid deep learning module is used to transfer knowledge from the source domain to the target domain through transfer learning, and to perform weighted combination of the outputs of multiple deep learning models. The multi-task learning module for complex environments is used to extract features through a shared feature learning layer, optimize multiple tasks through a weighted loss function, and adjust the priority between tasks according to their importance. The real-time context awareness and behavior optimization module is used to collect environmental data in real time through sensors, adjust the behavior recognition strategy of the behavior recognition model according to the current context, and optimize the recognition results based on dynamic context data through a context-weighted feedback mechanism. The semantically hierarchical behavior recognition and reasoning module is used to perform semantic analysis through natural language processing models, perform behavior reasoning using graph neural networks, and perform behavior reasoning by combining time and space dimensions through spatiotemporal graph modeling.
[0005] Preferably, the intelligent self-learning and incremental learning module uses an online gradient descent algorithm to update the parameters of the behavior recognition model, adjusts the behavior recognition model when new data is received, and dynamically adjusts the learning rate during the incremental update process through an adaptive time-series feedback mechanism; the intelligent self-learning and incremental learning module uses a meta-learning strategy to adapt the behavior recognition model to different tasks, and uses the MAML algorithm to learn from a small amount of new data.
[0006] Preferably, the cross-domain knowledge transfer and hybrid deep learning module is pre-trained using an autoencoder, and combines the outputs of multiple deep learning models using a weighted average method. The weights of each deep learning model are dynamically adjusted based on the similarity between the source and target domains, and the learning of the target domain is enhanced by a generative adversarial network.
[0007] Preferably, the multi-task learning module in the complex environment extracts features through shared convolutional layers and optimizes multiple tasks through a weighted loss function; the weighted loss function dynamically adjusts the learning progress and priority of each task through an adaptive task scheduling algorithm, which is based on reinforcement learning and adjusts the execution of tasks according to their importance and urgency.
[0008] Preferably, the real-time context awareness and behavior optimization module optimizes behavior recognition through a weighted loss function and dynamically adjusts the behavior recognition strategy based on environmental data through a context-weighted feedback mechanism. The behavior recognition strategy is dynamically adjusted through reinforcement learning.
[0009] Preferably, the semantically hierarchical behavior recognition and reasoning module uses the BERT model for semantic analysis and a graph neural network for behavior reasoning; the semantically hierarchical behavior recognition and reasoning module uses a spatiotemporal graph convolutional network for spatiotemporal graph modeling to combine dynamic changes in time and space for behavior reasoning.
[0010] Preferably, the cross-domain knowledge transfer and hybrid deep learning module is trained on target domain data using fine-tuning techniques, which reduce the differences between the source and target domains through a domain-adaptive loss function.
[0011] Preferably, the cross-domain knowledge transfer and hybrid deep learning module enhances its adaptability to the target domain through generative adversarial training, which utilizes a generator and a discriminator to generate virtual data similar to the target domain.
[0012] Preferably, the real-time context perception and behavior optimization module adjusts the behavior recognition strategy based on the real-time collected temperature and humidity environmental data, and optimizes the adjustment amount of the context data through a deep Q-network.
[0013] Preferably, the semantically hierarchical behavior recognition and reasoning module combines time series analysis and spatial distribution model to process spatiotemporal related data, and uses a spatiotemporal graph convolutional network for spatiotemporal modeling.
[0014] In summary, the present invention has the following main beneficial effects: I. By combining a cross-domain knowledge transfer module with a hybrid deep learning strategy, along with generative adversarial networks and adaptive weighting mechanisms, we can effectively integrate multi-source heterogeneous data, reduce domain differences, and significantly improve the model's generalization ability and behavior recognition accuracy in complex scenarios. Second, based on the online gradient descent algorithm, the temporal adaptive learning mechanism and the meta-learning strategy, the system can update the model parameters in real time, dynamically adapt to environmental changes, avoid catastrophic forgetting, and at the same time optimize task scheduling and contextual feedback through reinforcement learning to ensure real-time response and high-precision recognition in complex environments. Third, by combining semantic hierarchical analysis, spatiotemporal graph convolutional networks, and multi-task shared feature learning, the system can accurately capture the spatiotemporal correlation of behavior, dynamically adjust task priorities, and improve the inference accuracy and multi-task collaborative efficiency of complex behavior patterns. Attached Figure Description
[0015] Figure 1 Overall architecture diagram of a behavior recognition and analysis system based on a multimodal large model.
[0016] Figure 2 Detailed structural diagram of the intelligent self-learning and incremental learning module.
[0017] Figure 3 Detailed structural diagram of the cross-domain knowledge transfer and hybrid deep learning module.
[0018] Figure 4 Detailed structural diagram of the multi-task learning module in complex environments.
[0019] Figure 5 Detailed structural diagram of the real-time context awareness and behavior optimization module.
[0020] Figure 6 Detailed structural diagram of the semantically hierarchical behavior recognition and reasoning module. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] refer to Figure 1-6A behavior recognition and analysis system based on a multimodal large model, the system comprising: The intelligent self-learning and incremental learning module updates the parameters of the behavior recognition model in real time using an online gradient descent algorithm. The incremental learning module automatically adjusts the model each time new data is received, enabling the model to gradually adapt to new data and optimize behavior recognition capabilities in real time. By introducing a time-series adaptive learning mechanism and an incremental update method based on time-series data, the incremental learning module ensures that the system can learn and adapt to rapidly changing environmental data in real time. Furthermore, through a meta-learning strategy, it optimizes the model's adaptability when processing different data sources and prevents catastrophic forgetting. A cross-domain knowledge transfer and hybrid deep learning module is provided. This module transfers knowledge from the source domain to the target domain through transfer learning and combines the output results of multiple deep learning models to improve the accuracy of behavior recognition through a weighted average method. The cross-domain knowledge transfer module introduces an adaptive weighting strategy, which dynamically adjusts the transfer strategy according to the characteristics of the target domain data. The strategy enhances the learning effect of the target domain through a generative adversarial network (GAN). A multi-task learning module for complex environments optimizes the performance of multiple tasks simultaneously by sharing a feature learning layer and a weighted loss function to improve behavior recognition capabilities in complex environments. The module introduces a task adaptive scheduling mechanism that automatically adjusts the priority between tasks based on their importance. This scheduling mechanism is based on reinforcement learning and adjusts the learning progress of tasks according to their environmental adaptability. The real-time context awareness and behavior optimization module collects environmental data in real time through sensors and adjusts the behavior recognition model's strategy according to the current context to ensure that behavior recognition can adapt to changes in the environment. The module optimizes the recognition results in real time based on dynamic context data through a context-weighted feedback mechanism. The context optimization strategy is dynamically adjusted through reinforcement learning to automatically optimize the model's recognition response. The semantically hierarchical behavior recognition and reasoning module performs semantic analysis through a natural language processing model and uses a graph neural network to reason about behavior, calculating the causal relationships between nodes to further optimize behavior prediction. The reasoning process combines time and space dimensions through spatiotemporal graph modeling to perform accurate behavior reasoning. The spatiotemporal graph modeling combines a spatiotemporal graph convolutional network (ST-GCN) to improve the accuracy and spatiotemporal adaptability of behavior reasoning.
[0023] refer to Figure 1-6 The incremental learning module uses an online gradient descent algorithm for model updates, and the update formula is:
[0024] in, For learning rate, As the gradient of the loss function, the incremental learning module dynamically adjusts the learning rate in each incremental update step through an adaptive temporal feedback mechanism to better adapt to changes in data. It also enables the system to adapt quickly to different tasks through a meta-learning strategy, especially by optimizing the system's rapid learning ability on a small amount of new data through the MAML (Model-Agnostic Meta-Learning) algorithm.
[0025] refer to Figure 1-6 The cross-domain knowledge transfer and hybrid deep learning module is pre-trained using an autoencoder, and the outputs of the hybrid deep learning model are combined using a weighted average, as shown in the formula:
[0026] in, Weights for each model, For the first The output of each model, To achieve the final prediction result, the weighting strategy employs a dynamic adjustment mechanism, automatically adjusting the weights of each model based on the similarity between the source and target domains. This strategy enhances the learning effect in the target domain through a generative adversarial network (GAN).
[0027] refer to Figure 1-6 The multi-task learning module extracts features by sharing convolutional layers and optimizes multiple tasks using the following weighted loss function:
[0028] in, For the first The loss of each sub-task The weighted loss function is implemented using an adaptive task scheduling algorithm, which dynamically adjusts the learning progress and priority of each task. The scheduling mechanism is based on reinforcement learning and automatically optimizes task execution according to the importance and urgency of the task.
[0029] refer to Figure 1-6 The context awareness and behavior optimization module optimizes behavior recognition through a weighted loss function, the formula of which is:
[0030] in, For context-related weights, Using a conventional loss function, the context-aware module dynamically adjusts the behavior recognition strategy based on environmental data through a context-weighted feedback mechanism to improve adaptability in changing environments. The optimization strategy is dynamically adjusted through reinforcement learning to ensure the system's real-time response and accuracy.
[0031] refer to Figure 1-6 The semantically hierarchical behavior recognition and reasoning module uses the BERT model for semantic analysis, and the reasoning process is performed through a graph neural network, as shown in the formula:
[0032] in, For nodes In the Layer representation, For the weight of the layer, For nodes The reasoning module innovatively introduces spatiotemporal graph modeling to the neighboring nodes of the neighboring nodes, combining the dynamic changes of time and space to perform accurate behavioral reasoning. The spatiotemporal graph modeling adopts a spatiotemporal graph convolutional network (ST-GCN) to enhance the accuracy and spatiotemporal adaptability of behavioral reasoning.
[0033] refer to Figure 1-6 The cross-domain knowledge transfer and hybrid deep learning module trains the model on target domain data through fine-tuning techniques, enabling the model to adapt to specific tasks in the target domain. The fine-tuning technique is implemented through a domain-adaptive loss function to reduce the difference between the source domain and the target domain. Furthermore, the model's adaptability is enhanced through generative adversarial training, which uses a generator and a discriminator to generate virtual data similar to the target domain, further improving the model's learning ability in the target domain.
[0034] refer to Figure 1-6 The real-time context awareness and behavior optimization module adjusts the behavior recognition strategy based on real-time collected environmental data such as temperature and humidity. The strategy optimization formula is as follows:
[0035] in, For adjustment coefficients, The adjustment amount is the contextual data. The adjustment amount dynamically adjusts the recognition strategy based on the real-time collected environmental data to optimize the adaptability in environmental changes. The adjustment amount is optimized through a deep Q-network (DQN) to further improve the system's behavior prediction accuracy in dynamic environments.
[0036] refer to Figure 1-6The semantically hierarchical behavior recognition and reasoning module combines time series analysis and spatial distribution models to process spatiotemporal related data, further enhancing the ability to recognize complex behavior patterns. The spatiotemporal modeling uses Spatiotemporal Graph Convolutional Networks (ST-GCN) to improve the accuracy of behavior reasoning.
[0037] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A behavior recognition and analysis system based on a multimodal large model, characterized in that, include: The intelligent self-learning and incremental learning module is used to incrementally update the behavior recognition model and perform adaptive learning based on time series data. The cross-domain knowledge transfer and hybrid deep learning module is used to transfer knowledge from the source domain to the target domain through transfer learning, and to perform weighted combination of the outputs of multiple deep learning models. The multi-task learning module for complex environments is used to extract features through a shared feature learning layer, optimize multiple tasks through a weighted loss function, and adjust the priority between tasks according to their importance. The real-time context awareness and behavior optimization module is used to collect environmental data in real time through sensors, adjust the behavior recognition strategy of the behavior recognition model according to the current context, and optimize the recognition results based on dynamic context data through a context-weighted feedback mechanism. The semantically hierarchical behavior recognition and reasoning module is used to perform semantic analysis through natural language processing models, perform behavior reasoning using graph neural networks, and perform behavior reasoning by combining time and space dimensions through spatiotemporal graph modeling.
2. The behavior recognition and analysis system based on a multimodal large model according to claim 1, characterized in that: The intelligent self-learning and incremental learning module uses an online gradient descent algorithm to update the parameters of the behavior recognition model, adjusts the behavior recognition model when new data is received, and dynamically adjusts the learning rate during the incremental update process through an adaptive time-series feedback mechanism. The intelligent self-learning and incremental learning module uses a meta-learning strategy to adapt the behavior recognition model to different tasks, and uses the MAML algorithm to learn from a small amount of new data.
3. The behavior recognition and analysis system based on a multimodal large model according to claim 2, characterized in that: The cross-domain knowledge transfer and hybrid deep learning module is pre-trained using an autoencoder and combines the outputs of multiple deep learning models using a weighted average method. It dynamically adjusts the weights of each deep learning model based on the similarity between the source and target domains and enhances the learning of the target domain through a generative adversarial network.
4. The behavior recognition and analysis system based on a multimodal large model according to claim 3, characterized in that: The multi-task learning module in the complex environment extracts features through shared convolutional layers and optimizes multiple tasks through a weighted loss function. The weighted loss function dynamically adjusts the learning progress and priority of each task through an adaptive task scheduling algorithm. The adaptive task scheduling algorithm is based on reinforcement learning and adjusts the execution of tasks according to their importance and urgency.
5. The behavior recognition and analysis system based on a multimodal large model according to claim 4, characterized in that: The real-time context awareness and behavior optimization module optimizes behavior recognition through a weighted loss function and dynamically adjusts the behavior recognition strategy based on environmental data through a context-weighted feedback mechanism. The behavior recognition strategy is dynamically adjusted through reinforcement learning.
6. The behavior recognition and analysis system based on a multimodal large model according to claim 5, characterized in that: The semantically hierarchical behavior recognition and reasoning module uses the BERT model for semantic analysis and graph neural networks for behavior reasoning; the semantically hierarchical behavior recognition and reasoning module uses a spatiotemporal graph convolutional network for spatiotemporal graph modeling to combine dynamic changes in time and space for behavior reasoning.
7. The behavior recognition and analysis system based on a multimodal large model according to claim 6, characterized in that: The cross-domain knowledge transfer and hybrid deep learning module trains on target domain data through fine-tuning techniques, which reduce the differences between the source and target domains through a domain-adaptive loss function.
8. The behavior recognition and analysis system based on a multimodal large model according to claim 7, characterized in that: The cross-domain knowledge transfer and hybrid deep learning module enhances the adaptability to the target domain through generative adversarial training, which uses a generator and a discriminator to generate virtual data similar to the target domain.
9. The behavior recognition and analysis system based on a multimodal large model according to claim 8, characterized in that: The real-time context awareness and behavior optimization module adjusts the behavior recognition strategy based on real-time collected temperature and humidity environmental data, and optimizes the adjustment amount of context data through a deep Q network.
10. The behavior recognition and analysis system based on a multimodal large model according to claim 9, characterized in that: The semantically hierarchical behavior recognition and reasoning module combines time series analysis and spatial distribution model to process spatiotemporal related data, and uses spatiotemporal graph convolutional networks for spatiotemporal modeling.