A complex scene situation understanding method, system, device and medium

Abstract

This invention discloses a method, system, device, and medium for situational understanding in complex scenarios, relating to the field of behavior understanding technology. The method provides a network model suitable for behavioral associations of multimodal objects, enabling complex association analysis between different modalities. It constructs a unified multimodal processing method to handle arbitrary multimodal association situations, realizing the construction of an enhanced situational understanding model based on the concept of lifelong continuous learning to address situational understanding of different novel data. It utilizes a pre-trained large language model to mine effective general representations, while simultaneously capturing graph-structured prior knowledge and dynamic input correlations, guiding the powerful feature extraction and generalization capabilities of the large language model to complete various downstream situational understanding tasks. This solves the problems of low automation and insufficient intelligence in current situational awareness and understanding.

Description

Technical Field

[0001] This invention relates to the field of behavior understanding technology, and in particular to a method, system, device and medium for situational understanding in complex scenarios. Background Technology

[0002] The fundamental model for situational understanding enables technicians to analyze massive amounts of multi-source, heterogeneous information from complex urban scenarios more efficiently. It allows for general feature extraction from multimodal information, facilitating the identification, classification, and prediction of target behavior even under random noise interference. In recent years, deep learning technology, with its data-driven nature and powerful feature extraction and integration capabilities, has effectively accomplished situational understanding tasks in complex scenarios.

[0003] For example, Lee et al. proposed a self-supervised feature enhancement network structure. This method does not rely on manually labeled data. During training, the network randomly eliminates feature values ​​from the high-level feature matrix of scene situation and then predicts the eliminated feature values ​​by aggregating neighboring values. This method achieves fine-grained segmentation of scene targets in noisy environments.

[0004] Zhao et al. combined a cascaded residual auto-programmer architecture with a method based on cyclic consistency learning to learn a general representation among multimodal information in different complex scenarios. This method can help humans understand the semantic information contained in images when modalities are missing.

[0005] Liu et al. explored a method for decoupling situation time series, which achieves interpolation and prediction of missing situation information in the time dimension by studying the periodic representation, trend representation, and local deviation representation contained in situation information.

[0006] Zhang et al. proposed a human behavior recognition method based on scene semantic perception. This method captures the global correlation between the semantic information of human joints and the spatiotemporal information of nodes, and finally achieves effective understanding of human behavior in the scene.

[0007] Qin et al. proposed using graph neural networks to perform high-level representation of the target's semantic, temporal, and spatial information contained in the scene, thereby mining the deep relationship between the target's motion trajectory and behavior category in three-dimensional space and achieving efficient prediction of the target's behavior in space.

[0008] Gao et al. proposed a multimodal graph neural network structure that constructs a structure from both visual and linguistic perspectives. This structure represents image content as three graph knowledge structures: visual, semantic, and numerical. It also guides an information aggregator to achieve feature transfer between graphs, ultimately realizing cross-modal scene understanding from vision to text.

[0009] Xian et al. proposed a method that uses a pre-trained large language model to generate scene understanding text descriptions and constructs a text encoder to generate feature vectors between target behaviors, thereby improving the accuracy of understanding target actions by utilizing target behavior descriptions and semantic relationships between behaviors.

[0010] Azuma et al. proposed a scanning question-answering model, which consists of a 3D position and language information encoder, a position and language information feature fusion layer, and a target localization and question-answering module. Ultimately, it associates 3D position features with descriptor language features to achieve 3D situational information understanding.

[0011] In summary, current situational understanding models suffer from a number of limitations, such as a singular situational understanding scenario objective, a limited range of modal information categories to process, reliance on high-quality sensing data, and model structure design tailored only to specific situational understanding tasks. However, complex scenarios involve diverse scene types, target types, and data modalities, and urban situations present complex situational understanding tasks. These limitations render current situational understanding models inadequate for handling situational understanding tasks in complex urban sensing scenarios. Summary of the Invention

[0012] In view of the above problems, the present invention provides a method, system, device and medium for complex scene situation understanding to overcome the above problems or at least partially solve the above problems.

[0013] This invention provides the following solution:

[0014] A method for situational understanding in complex scenarios, comprising:

[0015] Acquire multimodal data related to the scene situation, wherein the multimodal data includes at least text language data, target key point data, target movement trajectory data, electromagnetic spectrum data, and graph structure prior knowledge data;

[0016] The multimodal data is input into a situation understanding base model for implementing unified situation understanding deep learning, so that the situation understanding base model outputs the behavior understanding results of the target, the behavior understanding results including at least the target action recognition results, the target action classification results, and the target action prediction results;

[0017] The situational understanding basic model includes a data serialization unified representation module, a cross-modal extraction shared representation module, a behavior prior knowledge fusion module, and a downstream specific task decoder.

[0018] After receiving the multimodal data, the situational understanding base model uses the data serialization unified representation module to perform unified serialization processing on the various modal data and embed them into a shared manifold space.

[0019] The cross-modal extraction shared representation module is used to extract a common representation among different feature representations;

[0020] The behavioral prior knowledge fusion module is used to mine the correlation between the current behavioral situation information and the graph structure prior knowledge data;

[0021] The downstream task-specific decoder is used to decode the general representation to obtain the target behavior understanding results required by the downstream task.

[0022] Preferably, the unified serialization process includes:

[0023] For the text language data, the original sentence is segmented into clauses using an embedding method based on double-byte encoding to achieve natural language encoding and ensure the semantic information contained in the language learned by the network.

[0024] For the target key point data, the entire image is reshaped into a flat sequence of information, and the pixels containing key points are assigned specified semantic information;

[0025] For the target movement trajectory data, trajectory position encoding at different time scales is constructed using the target time window;

[0026] For the electromagnetic spectrum data, the spectrum representation under different windows is obtained by combining the filter overlapping window mechanism, realizing the conversion from spectrum expansion representation to sequence representation.

[0027] Preferably, the cross-modal extraction shared representation module uses a Transformer-based pre-trained visual large language model to extract features from sequence information from different modalities.

[0028] Preferably, the behavior prior knowledge fusion module uses a multi-head attention mechanism to obtain the association features between the semantics of different modal behaviors and graph prior knowledge, and fuses the extracted semantic general and association features between modalities to obtain the association relationship.

[0029] Preferably, the downstream specific task decoder performs model fine-tuning by understanding downstream tasks through diverse situational awareness.

[0030] Preferably, the prior knowledge data of the graph structure is obtained by analyzing prior multimodal data using a behavioral association analysis model; the association analysis model includes a cross-modal association graph encoding module and a behavioral association recognition module.

[0031] The cross-modal association graph encoding module learns the semantic correspondence between different modalities by constructing a heterogeneous cross-modal graph fusion model;

[0032] The behavior association recognition module processes the heterogeneous graph network through differentiable pooling and graph attention network to identify semantic associations within and across modalities, generate node embeddings, and realize the representation of information of different modalities by combining cross-modal graphs. It also uses the interactive graph encoder module to realize semantic association recognition, so as to provide the behavioral characteristics of multimodal targets in different contexts and their association strength.

[0033] Preferably, a lifelong continuous learning mechanism is used to correct the parameters of the situation understanding base model in order to obtain an enhanced situation understanding model; the lifelong continuous learning mechanism includes a knowledge transfer module, a human-in-the-loop evaluation module, and a self-optimizing iterative module;

[0034] The knowledge transfer module extracts the global and local correlations of model parameters for a single scene by combining graph neural networks and convolutional neural networks, and then superimposes the results to obtain the model initialization parameters for any current scene.

[0035] The human-in-the-loop evaluation module evaluates the situational understanding prediction results output by the situational understanding network after the parameters are updated to obtain the human evaluation results;

[0036] The self-optimizing iterative module optimizes and iterates the human evaluation results to obtain quantified human judgment experience, so as to use the human judgment experience to optimize and update the parameters of the situation understanding basic model in the next round of iteration.

[0037] A complex scene situation understanding system, characterized in that it is used to execute the above-described complex scene situation understanding method, the system comprising:

[0038] A multimodal feature acquisition unit is used to acquire scene situation-related multimodal data, which includes at least text language data, target key point data, target movement trajectory data, electromagnetic spectrum data, and graph structure prior knowledge data.

[0039] The situation understanding execution unit is used to input the multimodal data into a situation understanding base model for implementing unified situation understanding deep learning, so that the situation understanding base model outputs the behavior understanding result of the target, the behavior understanding result including at least the target action recognition result, the target action classification result and the target action prediction result;

[0040] The situational understanding basic model includes a data serialization unified representation module, a cross-modal extraction shared representation module, a behavior prior knowledge fusion module, and a downstream specific task decoder.

[0041] After receiving the multimodal data, the situational understanding base model uses the data serialization unified representation module to perform unified serialization processing on the various modal data and embed them into a shared manifold space.

[0042] The cross-modal extraction shared representation module is used to extract a common representation among different feature representations;

[0043] The behavioral prior knowledge fusion module is used to mine the correlation between the current behavioral situation information and the graph structure prior knowledge data;

[0044] The downstream task-specific decoder is used to decode the general representation to obtain the target behavior understanding results required by the downstream task.

[0045] A complex scene situation understanding device, the device including a processor and a memory:

[0046] The memory is used to store program code and transmit the program code to the processor;

[0047] The processor is used to execute the aforementioned complex scenario situation understanding method according to the instructions in the program code.

[0048] A computer-readable storage medium for storing program code for executing the above-described complex scenario situational understanding method.

[0049] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0050] This application provides a method, system, device, and medium for situational understanding in complex scenarios. The method offers a network model suitable for multimodal object behavior association, enabling complex association analysis between different modalities. It constructs a unified multimodal processing method to handle arbitrary multimodal association situations, and implements a situational understanding enhancement model based on the concept of lifelong continuous learning to address situational understanding of different novel data. It utilizes a pre-trained large language model to mine effective general representations, while simultaneously capturing graph-structured prior knowledge and dynamic input correlations, guiding the powerful feature extraction and generalization capabilities of the large language model to complete various downstream situational understanding tasks. This solves the problems of low automation and insufficient intelligence in current situational awareness and understanding.

[0051] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0053] Figure 1 This is a flowchart of a complex scene situation understanding method provided by an embodiment of the present invention;

[0054] Figure 2 This is the overall architecture diagram of the multimodal behavior correlation analysis model provided in the embodiments of the present invention;

[0055] Figure 3 This is a structural diagram of the basic model for situational understanding provided in this embodiment of the invention;

[0056] Figure 4 This is a structural diagram of the situational understanding model based on lifelong continuous learning provided in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of a complex scene situation understanding system provided in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of a complex scene situation understanding device provided in an embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0060] See Figure 1 This invention provides a method for understanding the situation in complex scenarios, such as... Figure 1 As shown, the method may include:

[0061] S101: Acquire multimodal data related to the scene situation, wherein the multimodal data includes at least text language data, target key point data, target movement trajectory data, electromagnetic spectrum data, and graph structure prior knowledge data;

[0062] S102: Input the multimodal data into a situational understanding base model for implementing unified situational understanding deep learning, so that the situational understanding base model outputs the behavior understanding results of the target, the behavior understanding results including at least the target action recognition results, the target action classification results, and the target action prediction results;

[0063] The situational understanding basic model includes a data serialization unified representation module, a cross-modal extraction shared representation module, a behavior prior knowledge fusion module, and a downstream specific task decoder.

[0064] After receiving the multimodal data, the situational understanding base model uses the data serialization and unified representation module to perform unified serialization processing on the various modal data and embed it into a shared manifold space; in specific implementation, the unified serialization processing in this application embodiment may include:

[0065] The text language data is segmented into clauses using a double-byte encoding-based embedding method to achieve natural language encoding and ensure the semantic information contained in the language learned by the network.

[0066] The target key point data is used to reshape the entire image into a flat sequence of information, and the pixels containing key points are assigned the specified semantic information.

[0067] For the target movement trajectory data, a target time window is used to construct trajectory position encoding at different time scales;

[0068] By combining the electromagnetic spectrum data with the filter overlap window mechanism, the spectrum representation under different windows is obtained, realizing the conversion from spectrum expansion representation to sequence representation.

[0069] The cross-modal extraction shared representation module is used to extract a common representation among different feature representations; the cross-modal extraction shared representation module uses a Transformer-based pre-trained visual large language model to extract features from sequence information from different modalities.

[0070] The behavior prior knowledge fusion module is used to mine the correlation between the current behavior situation information and the graph structure prior knowledge data. The behavior prior knowledge fusion module uses a multi-head attention mechanism to obtain the correlation features between the semantics of different modal behaviors and the graph prior knowledge, and fuses the extracted semantic general and correlation features between modalities to obtain the correlation.

[0071] The downstream task-specific decoder is used to decode the general representation to obtain the target's behavioral understanding results required by the downstream task. The downstream task-specific decoder performs model fine-tuning by understanding diverse situations in downstream tasks.

[0072] Furthermore, embodiments of this application can provide that the prior knowledge data of the graph structure is obtained by analyzing prior multimodal data using a behavioral association analysis model; the association analysis model includes a cross-modal association graph encoding module and a behavioral association recognition module;

[0073] The cross-modal association graph encoding module is used to learn the semantic correspondence between different modalities by constructing a heterogeneous cross-modal graph fusion model;

[0074] The behavior association recognition module is used to process heterogeneous graph networks through differentiable pooling and graph attention networks to identify semantic associations within and across modalities, generate node embeddings, and realize the representation of information of different modalities by combining cross-modal graphs. It also uses the interactive graph encoder module to realize semantic association recognition, so as to provide the behavioral characteristics of multimodal targets in different contexts and their association strength.

[0075] To address the issue of varying understanding capabilities and focuses of situational understanding models across different application scenarios, embodiments of this application can provide a lifelong continuous learning mechanism to correct the parameters of the basic situational understanding model, thereby obtaining an enhanced situational understanding model. The lifelong continuous learning mechanism includes a knowledge transfer module, a human-in-the-loop evaluation module, and a self-optimizing iterative module.

[0076] The knowledge transfer module is used to extract the global and local correlations of model parameters for a single scene by combining graph neural networks and convolutional neural networks, and then superimpose the results to obtain the model initialization parameters for any current scene.

[0077] The human-in-the-loop evaluation module is used to evaluate the situation understanding prediction results output by the situation understanding network after parameter updates to obtain human evaluation results.

[0078] The self-optimizing iteration module is used to optimize and iterate the human evaluation results to obtain quantified human judgment experience, so as to use the human judgment experience to optimize and update the parameters of the situation understanding basic model in the next iteration.

[0079] The complex scenario situational understanding method provided in this application offers a network model applicable to multimodal object behavior association, enabling complex correlation analysis between different modalities. A unified multimodal processing method is constructed to handle arbitrary multimodal correlation situations, realizing the construction of an enhanced situational understanding model based on the concept of lifelong continuous learning to address situational understanding of different novel data. A pre-trained large language model is used to mine effective general representations, while simultaneously capturing graph-structured prior knowledge and dynamic input correlations, guiding the powerful feature extraction and generalization capabilities of the large language model to complete various downstream situational understanding tasks. This solves the problems of low automation and insufficient intelligence in current situational awareness and understanding.

[0080] The following is a detailed description of the complex scenario situation understanding method provided in the embodiments of this application.

[0081] In situational understanding tasks using multimodal data, the cross-modal data acquired through target recognition and motion tracking suffer from heterogeneity and temporal asynchrony, making direct data fusion extremely complex. To address this challenge, a behavior association analysis network is constructed. This network first unifies multi-source modal data into a consistent feature representation through a feature extraction layer, then captures complex semantic correspondences through a feature interaction layer, ultimately achieving accurate association analysis of target behaviors.

[0082] The model is a behavior knowledge-enhanced attention fusion network for multimodal data analysis. It performs unified feature representation of multimodal information, then uses a multi-head attention mechanism through a feature interaction module to capture the correlation between different modalities, and uses behavior knowledge priors to guide attention allocation. Finally, the feature fusion layer uses a feature weighted attention fusion module to fuse the feature representations of different modal data, and finally outputs the object behavior recognition and prediction results.

[0083] Considering the variability of complex environments and the complexity of scenarios, single-scenario situational understanding models often exhibit poor generalization ability, leading to low confidence in downstream situational prediction tasks based on situational understanding results. To address this issue, this method employs a knowledge transfer-based model parameter update method to achieve adaptive transfer of the situational understanding model under dynamic scenarios; it utilizes a human-in-the-loop (HIL) evaluation method to enhance the generalization ability of the situational understanding model, thereby accurately evaluating the situational understanding results to guide the next round of training iterations; and it uses deep reinforcement learning to iteratively update network parameters and optimize the iterative model to enhance its understanding capabilities. Based on these solutions, a lifelong continuous learning-enhanced situational understanding model is constructed.

[0084] (1) Multimodal object behavior association analysis in heterogeneous graph network learning.

[0085] To address the challenges of directly fusing and collaboratively processing multimodal data acquired in real-world environments, this method constructs a behavior association analysis model based on heterogeneous networks. Utilizing a constructed cross-modal heterogeneous graph network, nodes and edges are encoded through a differentiable pooling graph attention network. The encoder first extracts intra- and inter-modal dependencies through graph interaction attention learning. Then, node clustering and assignment are performed to determine the relevance of each node to a specific behavior, constructing prior knowledge data of the graph structure.

[0086] The behavioral association analysis model comprises two key modules: a cross-modal association graph encoding module and a behavioral association recognition module.

[0087] The cross-modal graph encoding module learns semantic correspondences between different modalities by constructing a heterogeneous cross-modal graph fusion model. This module includes modeling behavioral nodes in different modalities and internal / inter-modal edges, aiming to identify semantic correspondences between nodes.

[0088] The behavior association recognition module processes heterogeneous graph networks through differentiable pooling and graph attention networks to identify intramodal and cross-modal semantic associations, generate node embeddings, and achieve effective representation of information from different modalities by combining cross-modal graphs. It then utilizes an interactive graph encoder module to realize semantic association recognition, ultimately providing behavioral features of multimodal targets in different contexts and their association strengths. The overall model architecture is as follows: Figure 2 As shown.

[0089] The module behavior interaction graph in the cross-modal association graph encoding module contains two types of nodes: object nodes and behavior nodes. In this method, object nodes correspond to the N objects of interest in the input modal data, such as drones and armored vehicles. Behavior nodes represent L labeled text annotations, such as offensive, defensive, or danger warning statements. The total number of nodes formed is... There are T = N + L behavioral interaction graphs (DIPs) for learning semantic correspondences between them. DIPs aim to capture dependencies between nodes within the same modality and interactions between nodes across different modalities in a scene. For example, when traffic lights and pedestrians are identified in a scene (e.g., when a car is stopped at an intersection), intramodal edges between related nodes within the object category are learned. Simultaneously, intermodal edges are associated between these nodes and behavioral nodes (e.g., the "turn left" node). Using these edges, the DIPs are formulated as follows: The adjacency matrix represents these intra-modal / inter-modal edges, and behavioral associations are derived from the text annotations. In this process, The graph is considered as a weighted graph, where each matrix element ca[i, j] ∈ [0, 1] represents the semantic correspondence between nodes i and j in the labeled concept. When two nodes have a strong semantic correspondence, ca[i, j] approaches 1; otherwise, it approaches 0. In model initialization, the edge weight between nodes i and j is set to ca[i, j] = 1.

[0090] The purpose of the interaction graph encoder module is to identify the most important neighboring nodes within and across modalities of the behavioral interaction graph to generate differentiable graph attention networks for node embeddings. The detailed design includes graph feature encoding and interaction attention, as well as node cluster assignment and embedding generation. Graph feature encoding and interaction attention refer to initializing the node features of the behavioral interaction graph using a Transformer-based bidirectional encoder (BERT), and then extracting node features and edge weights by stacking multiple graph attention layers. Each graph attention layer calculates the graph interaction attention weights between two nodes using the Softmax (normalization exponent) function, capturing their semantic correspondence in the relevant behavioral scenario. Therefore, za(0) first represents the initial node features of Ga, inputting the multimodal data information Q(a) into the BERT bidirectional encoder (Transformer-based bidirectional encoder) representation, and using its output word embeddings as the initial node features, i.e., za(0) = BERT(Qa). Furthermore, since BERT is pre-trained on a large corpus, it can generate context.

[0091] To extract semantic correspondences, a total of Bi graph attention layers are stacked in the network. Based on the adjacency matrix ca, assuming Np is the neighborhood of node p in the network graph, i.e., nodes with intramodal / intermodal edges with p and p itself, each graph interaction attention layer (denoted as GIA) obtains the graph interaction attention weight between two nodes p and q in the network using the softmax function, denoted as va, i.e.:

[0092]

[0093] Wherein, the edge feature embedding score ψ between node p and node q is:

[0094]

[0095] in, and They represent the first The and the first The characteristics of each node and It is a trainable weight matrix. For nodes in the network graph The nodes in the neighborhood Np.

[0096] in addition, ∥ represents the activation function, specifically the Leaky ReLU activation function, and ∥ represents the concatenation operation. This represents the matrix transpose operation. The above formula describes the nodes. and The importance of interdependencies in relevant network scenarios was identified, thereby capturing their semantic correspondences. Subsequently, graph convolution operations were performed based on va(l).

[0097] Differentiable pooling operations are used to identify strongly correlated node clusters, followed by Laplacian smoothing to generate node embeddings. The probability of a node being assigned to each cluster is calculated using the Softmax function, and then updated node embeddings are generated by combining node cluster assignment scores and intermediate node embeddings through a fully connected layer. This allows the model to aggregate relevant features between different modalities, contributing to the final node embeddings and behavior recognition. This process iteratively extracts node features and edge weights by repeatedly using multiple graph attention layers and cluster assignment operations, ultimately obtaining the correlation weight coefficients between different modalities.

[0098] (2) Construction of a dynamic knowledge-driven situational understanding basic model.

[0099] In modern complex environments, the diverse types of target objects and the multi-source, heterogeneous nature of the massive amounts of sensor-perceived information have led to an explosive growth in holographic big data, the cumulative effect of which far exceeds the acquisition, analysis, and intelligence capabilities of existing unmanned systems. Traditional situational understanding models consider only single-modal information, limiting their ability to understand situations in complex scenarios. Furthermore, in multimodal information scenarios, different network model architectures need to be carefully designed based on different combinations of modal information and specific tasks. This method introduces prior knowledge of the situational scenario using a graph structure, constructs a unified modal encoder to acquire high-dimensional multimodal information representations, and utilizes the feature extraction and generalization capabilities of a pre-trained large language model to achieve situational understanding tasks such as target object behavior recognition, classification, and prediction.

[0100] This method employs a unified situational understanding deep learning model based on the Transformer architecture for multimodal and multi-task learning under unpaired data. The model mainly consists of four parts: a data serialization and unified representation module for uniformly serializing and embedding various modal data into a shared manifold space; a behavior prior knowledge fusion module for mining the correlation between current situational information and prior knowledge; a pre-trained cross-modal shared representation extraction module for extracting common representations among different feature representations; and a downstream task-specific decoder for decoding shared features to obtain the features required by downstream tasks. The model architecture is shown in Figure 3.

[0101] To achieve unified serialization representation of multimodal data, this method employs a unique serialization representation method based on the structural characteristics of multimodal data. Multimodal information includes message information, target key point information, target movement trajectory, electromagnetic spectrum, and prior knowledge of the scene situation in the graph structure. First, the multimodal data undergoes serialization processing. For example, for message information, this method proposes to segment the original sentence into clauses using WordPiece embedding based on double-byte encoding, achieving natural language encoding to ensure the network learns the semantic information contained within the language. For key point information in images, this method reshapes the entire image into a flattened sequence of information and assigns special semantic information to pixels containing key points, facilitating the network model to discover the intrinsic correlation between key point pixels and the entire image. For target movement trajectory, this method proposes to construct trajectory position encoding at different time scales using specific time windows to ensure the model can capture movement states at different time scales in complex environments. For speech data, this method proposes to use filters to obtain spectrogram information and, combined with a filter overlapping window mechanism, obtain spectral representations under different windows, achieving the conversion from spectral expansion representation to sequence representation.

[0102] This method employs a Transformer-based pre-trained visual large-scale language model to extract features from sequence information from different modalities. The model uses ViT as its backbone network structure and is pre-trained on massive amounts of language image data. The multimodal information that has completed sequence representation is input into this large-scale model to extract general high-level representations between modalities; in this process, only the deep layers of the pre-trained model are trained.

[0103] Simultaneously, a behavioral knowledge prior fusion module is constructed. This module utilizes a multi-head attention mechanism to obtain the correlation features between behavioral semantics and graph prior knowledge of different modalities, and finally fuses the extracted high-level semantic general and correlation features between modalities.

[0104] Finally, we plan to build a downstream decoder to fine-tune the model through various situational understanding downstream tasks, such as target classification, behavior relationship judgment, and behavior similarity judgment, in order to improve the model's semantic feature extraction capability.

[0105] To achieve situational understanding in modern complex environments, and addressing the problem that current methods cannot effectively utilize multimodal information and that designing separate situational understanding task networks is too energy-intensive, we propose a unified situational understanding deep learning-based basic model based on the Transformer structure. This model is combined with modules such as unified sequence encoding and pre-trained large language models to achieve the construction of a unified large-scale model for multimodal and multi-task situational understanding.

[0106] (3) Construction of a situational understanding enhancement model for lifelong continuous learning.

[0107] To address the varying understanding capabilities and focuses of situational understanding models across different application scenarios, this method employs a lifelong learning approach to construct an enhanced situational understanding model based on the fundamental situational understanding model. The model structure is shown in Figure 4, and it primarily comprises a knowledge transfer module, a human-in-the-loop evaluation module, and a self-optimizing iterative module. The specific implementation is as follows:

[0108] Single-scene information is used to obtain situational understanding predictions through a situational understanding model driven by real-time online data. However, considering the diversity and complexity of complex scene environments, the situational understanding model for a single scene lacks generalization ability for dynamic scenes. This method introduces lifelong continuous learning theory to correct the situational understanding model parameters, constructing an enhanced situational understanding model that more closely integrates human and artificial intelligence. Human expert knowledge is incorporated into the training process and is responsible for evaluating the prediction results. Due to the specific nature of the situational understanding task based on lifelong continuous learning in complex scene environments, this method does not require pre-constructing a dataset. The situational understanding results output by the enhanced model are evaluated by human experts, and the evaluation results are used as part of the parameter iterative optimization to iteratively correct the model parameters. The following sections will describe in detail the important modules of the lifelong continuous learning-based enhanced situational understanding model in this method.

[0109] First, a knowledge transfer module is introduced, which uses a combination of graph neural networks and convolutional neural networks to extract the global and local correlations of model parameters for a single scenario. The results are then superimposed to obtain the model initialization parameters for any given scenario. Since the model task is the same, the initialization of model parameters only considers updating the output layer parameters of the previous situation understanding model.

[0110] Furthermore, due to the complexity and incompleteness of complex scene data, the "Human-in-the-Loop" theory is introduced, incorporating relevant complex scene analysis experts into the model training: combining the data correlation obtained from a multi-source heterogeneous data knowledge base and a perception model, a situational understanding model that completes model parameter transfer is used to achieve the situational understanding task of the current scene data; the situational understanding results are then evaluated by an expert-led human judgment module, and an iterative optimization system based on deep reinforcement learning is used to quantify the expert evaluation results; the quantified evaluation results are used for parameter optimization and updates in the next iteration. Because the understanding results of artificial intelligence in situational understanding suffer from confidence inconsistencies—for example, while AI models can understand results that humans consider correct, their confidence is low, while some non-generally accepted results have high confidence—the introduction of the "Human-in-the-Loop" evaluation module and the optimization iterative module for quantifying human evaluations can greatly improve this problem, enabling the situational understanding model to output results that conform to human logic and understanding.

[0111] The loss function of the situational awareness enhancement model based on lifelong continuous learning is proposed to be defined as follows:

[0112]

[0113] in, The penalty coefficient is... To address the parameter bias in knowledge transfer networks, by changing... Used to control the penalty for changing the parameters of the knowledge transfer module. This represents the human evaluation results obtained by the situational understanding model through the "human-in-the-loop" evaluation module and the quantified human evaluation results after the optimization iteration module. This method uses a weighted sum of the L2 norms of each result as the quantization part of the optimization iteration module, where the weights are also defined by experts in the "human-in-the-loop" module.

[0114] Considering the complexity and uncertainty of complex scenarios, the "human-in-the-loop" evaluation module and the self-optimization iteration module are not essential modules during the model application phase. When situations arise where the model's situational understanding results cannot be evaluated in a timely manner due to data transmission delays, the model can directly output results through the situational understanding model. Unevaluated results will be automatically stored and subsequently updated via the open interfaces of the "human-in-the-loop" evaluation module and the self-optimization iteration module.

[0115] See Figure 5 This application embodiment can also provide a complex scene situation understanding system, such as Figure 5 As shown, the system used to perform the above-described complex scenario situational understanding method may include:

[0116] The multimodal feature acquisition unit 501 is used to acquire multimodal data related to the scene situation. The multimodal data includes at least text language data, target key point data, target movement trajectory data, electromagnetic spectrum data, and graph structure prior knowledge data.

[0117] The situation understanding execution unit 502 is used to input the multimodal data into a situation understanding base model for implementing unified situation understanding deep learning, so that the situation understanding base model outputs the behavior understanding result of the target, the behavior understanding result including at least the target action recognition result, the target action classification result and the target action prediction result;

[0118] The situational understanding basic model includes a data serialization unified representation module, a cross-modal extraction shared representation module, a behavior prior knowledge fusion module, and a downstream specific task decoder.

[0119] After receiving the multimodal data, the situational understanding base model uses the data serialization unified representation module to perform unified serialization processing on the various modal data and embed them into a shared manifold space.

[0120] The cross-modal extraction shared representation module is used to extract a common representation among different feature representations;

[0121] The behavioral prior knowledge fusion module is used to mine the correlation between the current behavioral situation information and the graph structure prior knowledge data;

[0122] The downstream task-specific decoder is used to decode the general representation to obtain the target behavior understanding results required by the downstream task.

[0123] This application embodiment can also provide a complex scene situation understanding device, the device including a processor and a memory:

[0124] The memory is used to store program code and transmit the program code to the processor;

[0125] The processor is used to execute the steps of the complex scenario situation understanding method described above according to the instructions in the program code.

[0126] like Figure 6 As shown in the figure, a complex scene situational understanding device provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0127] In this embodiment, the processor 10 may be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0128] The processor 10 can call programs stored in the memory 11. Specifically, the processor 10 can execute operations in the embodiments of the complex scene situation understanding method.

[0129] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0130] Acquire multimodal data related to the scene situation, wherein the multimodal data includes at least text language data, target key point data, target movement trajectory data, electromagnetic spectrum data, and graph structure prior knowledge data;

[0131] The multimodal data is input into a situation understanding base model for implementing unified situation understanding deep learning, so that the situation understanding base model outputs the behavior understanding results of the target, the behavior understanding results including at least the target action recognition results, the target action classification results, and the target action prediction results;

[0132] The situational understanding basic model includes a data serialization unified representation module, a cross-modal extraction shared representation module, a behavior prior knowledge fusion module, and a downstream specific task decoder.

[0133] After receiving the multimodal data, the situational understanding base model uses the data serialization unified representation module to perform unified serialization processing on the various modal data and embed them into a shared manifold space.

[0134] The cross-modal extraction shared representation module is used to extract a common representation among different feature representations;

[0135] The behavioral prior knowledge fusion module is used to mine the correlation between the current behavioral situation information and the graph structure prior knowledge data;

[0136] The downstream task-specific decoder is used to decode the general representation to obtain the target behavior understanding results required by the downstream task.

[0137] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0138] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0139] Communication interface 12 can be an interface for a communication model, used to connect with other devices or systems.

[0140] Of course, it should be noted that, Figure 6 The structure shown does not constitute a limitation on the complex scene situation understanding device in the embodiments of this application. In practical applications, the complex scene situation understanding device may include more than Figure 6 More or fewer components as shown, or combinations of certain components.

[0141] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the above-described complex scenario situation understanding method.

[0142] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0143] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0144] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0145] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for situational understanding in complex scenarios, characterized in that, include: Acquire multimodal data related to the scene situation, wherein the multimodal data includes at least text language data, target key point data, target movement trajectory data, electromagnetic spectrum data, and graph structure prior knowledge data; The multimodal data is input into a situation understanding base model for implementing unified situation understanding deep learning, so that the situation understanding base model outputs the behavior understanding results of the target, the behavior understanding results including at least the target action recognition results, the target action classification results, and the target action prediction results; The situational understanding basic model includes a data serialization unified representation module, a cross-modal extraction shared representation module, a behavior prior knowledge fusion module, and a downstream specific task decoder. After receiving the multimodal data, the situational understanding base model uses the data serialization unified representation module to perform unified serialization processing on the various modal data and embed them into a shared manifold space. The cross-modal extraction shared representation module is used to extract a common representation among different feature representations; The behavioral prior knowledge fusion module is used to mine the correlation between the current behavioral situation information and the graph structure prior knowledge data; The downstream task-specific decoder is used to decode the general representation to obtain the behavioral understanding results of the target required by the downstream task.

2. The method for understanding complex scenarios according to claim 1, characterized in that, The unified serialization process includes: The text language data is segmented into clauses using a double-byte encoding-based embedding method to achieve natural language encoding and ensure the semantic information contained in the language learned by the network. For the target key point data, the entire image is reshaped into a flat sequence of information, and the pixels containing key points are assigned specified semantic information; For the target movement trajectory data, trajectory position encoding at different time scales is constructed using the target time window; By combining the electromagnetic spectrum data with the filter overlap window mechanism, the spectrum representation under different windows is obtained, realizing the conversion from spectrum expansion representation to sequence representation.

3. The method for understanding complex scenarios according to claim 1, characterized in that, The cross-modal extraction shared representation module uses a Transformer-based pre-trained visual large language model to extract features from sequence information from different modalities.

4. The method for understanding complex scenarios according to claim 1, characterized in that, The behavior prior knowledge fusion module uses a multi-head attention mechanism to obtain the association features between the semantics of different modal behaviors and graph prior knowledge, and fuses the extracted semantic general and association features between modalities to obtain the association relationship.

5. The method for understanding complex scenarios according to claim 1, characterized in that, The downstream task-specific decoder fine-tunes the model by understanding the downstream tasks through diverse situations.

6. The method for understanding complex scenarios according to claim 1, characterized in that, The prior knowledge data of the graph structure is obtained by analyzing the prior multimodal data using a behavioral association analysis model; the association analysis model includes a cross-modal association graph encoding module and a behavioral association recognition module; The cross-modal association graph encoding module learns the semantic correspondence between different modalities by constructing a heterogeneous cross-modal graph fusion model; The behavior association recognition module processes the heterogeneous graph network through differentiable pooling and graph attention network to identify semantic associations within and across modalities, generate node embeddings, and realize the representation of information of different modalities by combining cross-modal graphs. It also uses the interactive graph encoder module to realize semantic association recognition, so as to provide the behavioral characteristics of multimodal targets in different contexts and their association strength.

7. The method for understanding complex scenarios according to claim 1, characterized in that, A lifelong continuous learning mechanism is used to correct the parameters of the basic situational understanding model in order to obtain an enhanced situational understanding model; the lifelong continuous learning mechanism includes a knowledge transfer module, a human-in-the-loop evaluation module, and a self-optimization iteration module. The knowledge transfer module extracts the global and local correlations of model parameters for a single scene by combining graph neural networks and convolutional neural networks, and then superimposes the results to obtain the model initialization parameters for any current scene. The human-in-the-loop evaluation module evaluates the situational understanding prediction results output by the situational understanding network after the parameters are updated to obtain the human evaluation results; The self-optimizing iterative module optimizes and iterates the human evaluation results to obtain quantified human judgment experience, so as to use the human judgment experience to optimize and update the parameters of the situation understanding basic model in the next round of iteration.

8. A situational understanding system for complex scenarios, characterized in that, The system is used to perform the complex scene situation understanding method according to any one of claims 1-7, the system comprising: A multimodal feature acquisition unit is used to acquire scene situation-related multimodal data, which includes at least text language data, target key point data, target movement trajectory data, electromagnetic spectrum data, and graph structure prior knowledge data. The situation understanding execution unit is used to input the multimodal data into a situation understanding base model for implementing unified situation understanding deep learning, so that the situation understanding base model outputs the behavior understanding result of the target, the behavior understanding result including at least the target action recognition result, the target action classification result and the target action prediction result; The situational understanding basic model includes a data serialization unified representation module, a cross-modal extraction shared representation module, a behavior prior knowledge fusion module, and a downstream specific task decoder. After receiving the multimodal data, the situational understanding base model uses the data serialization unified representation module to perform unified serialization processing on the various modal data and embed them into a shared manifold space. The cross-modal extraction shared representation module is used to extract a common representation among different feature representations; The behavioral prior knowledge fusion module is used to mine the correlation between the current behavioral situation information and the graph structure prior knowledge data; The downstream task-specific decoder is used to decode the general representation to obtain the target behavior understanding results required by the downstream task.

9. A situational understanding device for complex scenarios, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the complex scene situation understanding method according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the complex scenario situation understanding method according to any one of claims 1-7.

Images