Complex scene situation understanding method, system, equipment and medium

Abstract

The invention discloses a complex scene situation understanding method, system and device and a medium, and relates to the technical field of behavior understanding, the method provides a network model suitable for multi-modal object behavior association, and realizes complex association analysis among different modals. A unified multi-modal processing method is constructed to cope with any multi-modal association condition, and a situation understanding enhancement model is constructed based on a lifelong continuous learning concept to cope with situation understanding of different novel data. A pre-trained large language model is used for mining effective general representation, meanwhile, graph structured priori knowledge and dynamic input relevance are captured, and the powerful feature extraction ability and generalization ability of the large language model are guided to complete various downstream situation understanding tasks. The problems that current situation awareness and understanding are low in automation degree and insufficient in intelligence are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of behavior understanding, and in particular to a complex scene situation understanding method, system, device and medium. BACKGROUND

[0002] The situation understanding basic model makes it more efficient for technicians to analyze massive multi-source heterogeneous information from complex urban scenes, and the general feature extraction of multi-modal information helps to realize the behavior recognition, classification and prediction of targets in the target scene under random noise interference. In recent years, deep learning technology has the characteristics of data-driven and strong feature extraction and integration capabilities, which can effectively complete the situation understanding task in complex scenes.

[0003] For example, Lee et al. proposed a self-supervised feature enhancement network structure, which does not rely on manually labeled data. In the training process, the network randomly eliminates feature values from the scene situation high-level feature matrix, and then predicts the eliminated feature values by aggregating the adjacent values. This method realizes fine-grained division of scene targets in a noisy environment.

[0004] Zhao et al. combined a cascaded residual self-encoder architecture and a method based on cycle consistency learning to learn the general representation between multi-modal information in different complex scenes. This method can assist humans in understanding the semantic information contained in the image in the case of missing modalities.

[0005] Liu et al. explored a method for decoupling situation time series. This method realizes the interpolation and prediction of missing situation information in the time dimension by studying the periodic representation, trend representation and local bias representation contained in the situation information.

[0006] Zhang et al. proposed a human behavior recognition method based on scene semantic perception. This method captures the global correlation between human joint semantic information and spatio-temporal information of nodes, and finally realizes effective understanding of human behavior in the scene.

[0007] Qin et al. proposed to represent the target semantics, time, space and other information contained in the scene in the form of a graph neural network, to mine the deep relationship between the motion trajectory and behavior category of the target in three-dimensional space, and to realize efficient prediction of the target behavior in space.

[0008] Gao et al. proposed to construct a multi-modal graph neural network structure from the aspects of vision and language. This structure represents the image content as three graph knowledge structures of vision, semantics and numerical value, and guides the information aggregator to realize feature transmission between graphs, and finally realizes cross-modal scene understanding from vision to text.

[0009] Xian et al. proposed a method of generating scene understanding text description by using pre-training large language model, and constructing text encoder to generate feature vectors between target behaviors, so as to realize the use of target behavior description and semantic relationship between behaviors, and improve the accuracy of understanding target action.

[0010] Azuma et al. proposed a scanning question and answer model, which is composed of three-dimensional position and language information encoder, position and language information feature fusion layer, target positioning and question and answer module, finally associates three-dimensional position features and descriptor language features, and realizes three-dimensional situation understanding.

[0011] As can be seen from the above, the current situation understanding model is subject to a series of limitations, such as single situation understanding scene target, less processing modal information category and dependence on high-quality perception data, and model structure design only for special situation understanding task. But in complex scenes, the scene type, target type, data modal are various, and the city complex scene situation understanding task is complicated. Therefore, the current situation understanding model cannot cope with the situation understanding task in complex city perception scene. SUMMARY

[0012] In view of the above problems, the present application provides a complex scene situation understanding method, system, device and medium for overcoming the above problems or at least partially solving the above problems.

[0013] The present application provides the following scheme: A complex scene situation understanding method, comprising: acquiring multi-modal data related to scene situation, the multi-modal data at least including text language data, target key point data, target moving trajectory data, electromagnetic spectrum data and graph structure priori knowledge data; inputting the multi-modal data into a situation understanding basic model for realizing unified situation understanding deep learning, so that the situation understanding basic model outputs behavior understanding result of the target, the behavior understanding result at least including target action recognition result, target action classification result and target action prediction result; wherein, the situation understanding basic model includes data serialization unified representation module, cross-modal extraction shared representation module, behavior priori knowledge fusion module and downstream specific task decoder; after the situation understanding basic model receives the multi-modal data, the data serialization unified representation module is used to uniformly serialize and embed various modal data to a shared manifold space; the cross-modal extraction shared representation module is used to extract common representation between different feature representations; the behavior priori knowledge fusion module is used to mine the correlation between current behavior situation information and the graph structure priori knowledge data; The behavior understanding result of the target required by the downstream task is obtained by decoding the general representation using the downstream specific task decoder.

[0014] Preferably, the unified serialization processing includes: For the text language data, the original sentence is segmented into sub-sentences in a double-byte encoding-based embedding manner, and natural language coding is implemented to ensure that the network learns the semantic information contained in the language; For the target key point data, the entire image is reshaped into a flattened sequence information, and the pixels containing the key points are assigned with designated semantic information; For the target moving trajectory data, a target time window is used to construct trajectory position coding under different time scales; For the electromagnetic spectrum data, a filter overlap window mechanism is combined to obtain frequency spectrum representation under different windows, and conversion from spectrum expansion representation to sequence representation is implemented.

[0015] Preferably, the cross-modal extraction shared representation module uses a Transformer-based pre-trained visual large language model to extract features from sequence information from different modalities.

[0016] Preferably, the behavior prior knowledge fusion module uses a multi-head attention mechanism to obtain associated features of different modal behavior semantics and graph prior knowledge, and fuses the extracted inter-modal semantic general and associated features to obtain the associated relationship.

[0017] Preferably, the downstream specific task decoder performs model fine-tuning through various situation understanding downstream tasks.

[0018] Preferably, the graph structure prior knowledge data is obtained by analyzing the prior multi-modal data using a behavior association analysis model; the association analysis model includes a cross-modal association graph encoding module and a behavior association recognition module. The cross-modal association graph encoding module learns the semantic correspondence between different modalities by constructing a heterogeneous cross-modal graph fusion model. The behavior association recognition module processes the heterogeneous graph network through differentiable pooling and graph attention network, identifies the semantic association within and across modalities, generates node embedding, represents different modal information in a cross-modal graph manner, and uses an interaction graph encoder module to implement semantic association recognition, so as to provide behavior features and their associated strengths of multi-modal targets in different situations.

[0019] Preferably, a lifelong continuous learning mechanism is used to correct the parameters of the situation understanding base model to obtain a situation understanding enhanced model; the lifelong continuous learning mechanism includes a knowledge transfer module, a human-in-the-loop evaluation module, and a self-optimization iteration module. The knowledge transfer module extracts global correlation and local correlation of model parameters for a single scene in a combination of graph neural networks and convolutional neural networks, and superimposes the results to obtain model initialization parameters for any current scene; The human-in-the-loop evaluation module evaluates the situation understanding prediction results output by the situation understanding network after updating the parameters to obtain artificial evaluation results; The self-optimization iteration module optimizes and iterates the artificial evaluation results to obtain quantized artificial discrimination experience, so as to use the artificial discrimination experience for optimizing and updating the parameters of the situation understanding base model in the next iteration.

[0020] A complex scene situation understanding system, characterized by being used to execute the complex scene situation understanding method described above, the system comprises: A multi-modal feature acquisition unit is configured to acquire multi-modal data related to a scene situation, wherein the multi-modal data at least includes text language data, target key point data, target moving trajectory data, electromagnetic spectrum data, and graph structure prior knowledge data; A situation understanding execution unit is configured to input the multi-modal data into a situation understanding base model for realizing unified situation understanding deep learning, so that the situation understanding base model outputs a behavior understanding result of a target, wherein the behavior understanding result at least includes a target action recognition result, a target action classification result, and a target action prediction result; The situation understanding base model comprises a data serialization unified representation module, a cross-modal shared representation extraction module, a behavior prior knowledge fusion module, and a downstream specific task decoder. After the situation understanding base model receives the multi-modal data, the data serialization unified representation module is used to uniformly serialize and embed various modal data into a shared manifold space; The cross-modal shared representation extraction module is used to extract a general representation between different feature representations; The behavior prior knowledge fusion module is used to mine the association between current behavior situation information and the graph structure prior knowledge data; The downstream specific task decoder is used to decode the general representation to obtain a behavior understanding result of a target required by a downstream task.

[0021] A complex scene situation understanding device, the device comprises a processor and a memory: The memory is configured to store program code and transmit the program code to the processor; The processor is configured to execute the complex scene situation understanding method described above according to the instructions in the program code.

[0022] A computer-readable storage medium for storing program code for performing the complex scene situation understanding method described above.

[0023] According to the specific embodiments provided by the present application, the following technical effects are disclosed: The complex scene situation understanding method, system, device and medium provided by the embodiments of the present application provide a network model suitable for multi-modal object behavior association, and realize complex association analysis between different modalities. A unified multi-modal processing method is constructed to cope with any multi-modal association situation, and a situation understanding enhancement model is constructed based on the concept of lifelong continuous learning to cope with different new data situation understanding. The pre-training large language model is used to mine effective general-purpose representation, and the association between graph structured prior knowledge and dynamic input is captured to guide the powerful feature extraction capability and generalization capability of the large language model to complete multiple downstream situation understanding tasks. The problems of low automation and insufficient intelligence of current situation awareness and understanding are solved.

[0024] Of course, implementing any product of the present application does not necessarily require all the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0026] Figure 1 is a flowchart of a complex scene situation understanding method provided by an embodiment of the present application; Figure 2 is a general architecture diagram of a multi-modal behavior association analysis model provided by an embodiment of the present application; Figure 3 is a situation understanding base model structure diagram provided by an embodiment of the present application; Figure 4 is a situation understanding model structure diagram based on lifelong continuous learning provided by an embodiment of the present application; Figure 5 is a schematic diagram of a complex scene situation understanding system provided by an embodiment of the present application; Figure 6 is a schematic diagram of a complex scene situation understanding device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0028] Referring to Figure 1 A complex scene situation understanding method provided by the embodiments of the present application, as shown in Figure 1 The method can include: S101: acquiring multi-modal data related to a scene situation, wherein the multi-modal data at least includes text language data, target key point data, target moving trajectory data, electromagnetic spectrum data, and graph structure priori knowledge data; S102: inputting the multi-modal data into a situation understanding basic model for realizing unified situation understanding deep learning, so that the situation understanding basic model outputs a behavior understanding result of a target, wherein the behavior understanding result at least includes a target action recognition result, a target action classification result, and a target action prediction result; The situation understanding basic model includes a data serialization unified representation module, a cross-modal extraction shared representation module, a behavior priori knowledge fusion module, and a downstream specific task decoder. After the situation understanding basic model receives the multi-modal data, the data serialization unified representation module is used to uniformly serialize and embed various modal data into a shared manifold space; in a specific implementation, the present application embodiment can provide that the uniform serialization processing includes: For the text language data, a double-byte coding based embedding method is used to divide the original sentence into sub-sentences, and natural language coding is realized to ensure that the network learns the semantic information contained in the language; For the target key point data, the entire image is reshaped into a flattened sequence information, and the pixels containing the key points are assigned with specified semantic information; For the target moving trajectory data, a target time window is used to construct trajectory position coding under different time scales; For the electromagnetic spectrum data, a filter overlapping window mechanism is used to obtain spectrum representation under different windows, and conversion from spectrum expansion representation to sequence representation is realized.

[0029] The cross-modal extraction shared representation module is used to extract a general representation between different feature representations; the cross-modal extraction shared representation module uses a Transformer based pre-training visual large language model to extract features from sequence information from different modalities.

[0030] The behavior prior knowledge fusion module is used to mine an association relationship between current behavior situation information and the graph structure prior knowledge data; the behavior prior knowledge fusion module uses a multi-head attention mechanism to obtain an association feature of different modal behavior semantics and graph prior knowledge, and fuses the extracted inter-modal semantic common and association features to obtain the association relationship.

[0031] The downstream specific task decoder is used to decode the general representation to obtain a target behavior understanding result required by a downstream task. The downstream specific task decoder performs model fine-tuning for a downstream task through various situation understanding.

[0032] Further, the embodiment of the present application can provide that the graph structure prior knowledge data uses a behavior association analysis model to analyze prior multi-modal data; the association analysis model includes a cross-modal association graph encoding module and a behavior association recognition module. The cross-modal association graph encoding module is used to learn semantic correspondence between different modalities by constructing a heterogeneous cross-modal graph fusion model; The behavior association recognition module is used to process a heterogeneous graph network through a differentiable pooling and a graph attention network, recognize semantic associations within and across modalities, generate node embeddings, represent different modal information in a cross-modal graph manner, and use an interaction graph encoder module to implement semantic association recognition, so as to provide behavior features and their association strengths of multi-modal targets in different contexts.

[0033] To solve the problem of different understanding abilities and focuses of situation understanding models in different use scenarios, the embodiment of the present application can provide that a lifelong continuous learning mechanism is used to correct parameters of the situation understanding base model to obtain a situation understanding enhanced model; the lifelong continuous learning mechanism includes a knowledge transfer module, a human-in-the-loop evaluation module, and a self-optimization iteration module. The knowledge transfer module is used to extract global relevance and local relevance of model parameters for a single scene in a manner of combining a graph neural network and a convolutional neural network, and superimposes the results to obtain model initialization parameters for a current arbitrary scene; The human-in-the-loop evaluation module is used to evaluate a situation understanding prediction result output by the situation understanding network after updating the parameters to obtain a manual evaluation result; The self-optimization iteration module is used to optimize and iterate the manual evaluation result to obtain quantized manual discrimination experience, so as to use the manual discrimination experience for optimization and update of the parameters of the situation understanding base model in the next iteration.

[0034] The complex scene situation understanding method provided by the embodiments of the application provides a network model suitable for multi-modal object behavior association, and realizes complex association analysis between different modalities. A unified multi-modal processing method is constructed to cope with any multi-modal association situation, and a situation understanding enhancement model is constructed based on the concept of lifelong continuous learning to cope with different new data situation understanding. The pre-trained large language model is used to mine effective general-purpose representation, and the association between graph structured prior knowledge and dynamic input is captured to guide the powerful feature extraction capability and generalization capability of the large language model to complete multiple downstream situation understanding tasks. The problems of low automation and insufficient intelligence of current situation awareness and understanding are solved.

[0035] The complex scene situation understanding method provided by the embodiments of the application is described in detail below.

[0036] In the situation understanding task of multi-modal data, the cross-modal data obtained by target recognition and motion tracking has problems such as heterogeneity and time sequence asynchronization, and direct data fusion becomes quite complex. To solve this problem, a behavior association analysis network is constructed. The network first unifies multi-source modal data into consistent feature representation through a feature extraction layer, then captures complex semantic correspondence through a feature interaction layer, and finally realizes accurate association analysis of target behavior.

[0037] The behavior knowledge enhanced attention fusion network for model used in multi-modal data analysis unifies multi-modal information into consistent feature representation, then uses a multi-head attention mechanism through a feature interaction module to capture the association between different modalities, and uses behavior knowledge prior to guide attention allocation, finally uses a feature weighting attention fusion module through a feature fusion layer to fuse the feature representation of different modal data, and finally outputs object behavior recognition and prediction results.

[0038] Considering the variability of complex environment and the complexity of scene, the phenomenon of poor generalization of single scene situation understanding model is caused, which leads to low confidence of downstream situation prediction task based on situation understanding result. To solve the above problems, the method uses a model parameter updating method based on knowledge transfer to realize adaptive transfer of situation understanding model in dynamic scene; uses a "man-in-loop" evaluation method to enhance the generalization ability of the situation understanding model, so as to accurately evaluate the situation understanding result to guide the training of the next round of iteration; uses the way of deep reinforcement learning to update network parameters cyclically, and optimizes the iterative model to realize the understanding ability of the situation understanding model. The situation understanding enhancement model based on lifelong continuous learning is constructed based on the above scheme.

[0039] (1) Multi-modal object behavior association analysis of heterogeneous graph network learning.

[0040] To address the difficulties of directly integrating and collaboratively processing multimodal data acquired in real-world environments, this method constructs a behavioral association analysis model based on heterogeneous networks. Leveraging the constructed cross-modal heterogeneous graph network, a differentiable pooled graph attention network is used to encode nodes and edges. The encoder first extracts intra-modal and inter-modal dependencies through interactive attention learning. Nodes are then clustered and assigned to determine the relevance of each node for a specific behavior, building upon prior knowledge of the graph structure.

[0041] The behavioral association analysis model includes two key modules: a cross-modal association graph encoding module and a behavioral association recognition module.

[0042] The cross-modal association graph encoding module learns the semantic correspondences between different modalities by building a heterogeneous cross-modal graph fusion model. This module includes modeling behavior nodes under different modalities and intra- and inter-modal edges, aiming to identify semantic correspondences between nodes.

[0043] The behavioral association recognition module processes heterogeneous graph networks through differentiable pooling and graph attention networks, identifies semantic associations within and across modalities, generates node embeddings, and effectively represents information from different modalities by combining cross-modal graphs. It also uses the interaction graph encoder module to recognize semantic associations, ultimately providing behavioral characteristics and association strengths of multimodal targets in different contexts. The overall architecture of the model is as follows: Figure 2 shown.

[0044] The module behavior interaction graph in the cross-modal association graph encoding module contains two types of nodes, namely object and behavior nodes. In this method, the object node corresponds to the N objects of interest in the input modal data, such as drones, armored vehicles, etc. The behavior node represents the L labeled text annotation labels, such as attack, defense, danger warning, and other behavior statements. The total number of nodes formed is There are T = N + L behavioral interaction graphs for learning semantic correspondences between them. The behavioral interaction graph aims to capture the dependencies of nodes within the same modality and the interactions between nodes in different modalities in the scene. For example, when traffic lights and pedestrians are identified in the scene (e.g., when a car stops at an intersection), intra-modal edges are learned between related nodes within the object category. At the same time, inter-modal edges are associated with the above nodes and behavioral nodes (e.g., the "turn left" node). Using these edges, the behavioral interaction graph is formulated as The adjacency matrix of represents these intra-modal / inter-modal edges, and thus the behavioral associations are inferred from the text annotations. In this process, is considered as a weighted graph, where each matrix element ca[i, j] e [0, 1] represents the semantic correspondence between nodes i and j in the marked concept, when two nodes have strong semantic correspondence, ca[i, j] tends to 1, otherwise tends to 0. In the model initialization, the edge weight between nodes i and j is set to ca[i, j] = 1.

[0045] The interaction graph encoder module aims to identify the most important neighboring nodes within the behavioral interaction graph and across modalities to generate different differentiable graph attention networks for node embeddings. Detailed design includes graph feature encoding and interaction attention, and node cluster assignment and embedding generation. Graph feature encoding and interaction attention refer to initializing the node features of the behavioral interaction graph using a transformer-based bidirectional encoder (BERT), and then extracting node features and edge weights by stacking multiple graph attention layers. Each graph attention layer calculates the graph interaction attention weight between two nodes through the Softmax (normalized exponential) function, capturing their semantic correspondence in the relevant behavioral scenario. Therefore, first, za(0) represents the node features of the initialized Ga, and the input of the multi-modal data information Q(a) is input into the bidirectional encoder of BERT, and the output word embedding is used as the initial node feature, that is, za(0) = BERT(Qa). In addition, since BERT is pre-trained on a large corpus, it can generate context.

[0046] To extract semantic correspondence, a total of Bi graph attention layers are stacked in the network, according to the adjacency matrix ca, assuming Np is the neighborhood of node p in the network graph, that is, the nodes with intra-modal / inter-modal edges with p and p itself, each graph interaction attention layer (denoted as GIA) obtains the graph interaction attention weight between two nodes p and q in the network according to the softmax function, denoted as va, that is: where the edge feature embedding score ψ between node p and node q is: where, and represent the features of the th and the th node, and are trainable weight matrices, is the node in the neighborhood Np of the node in the network graph.

[0047] In addition, denotes an activation function, i.e., a LeakyReLU activation function, denotes a concatenation operation, denotes a matrix transpose operation. The above equation describes the node and The importance of the interdependence in the related network scenario, thereby capturing their semantic correspondence. Subsequently, the graph convolution operation is performed based on va(l).

[0048] Determine the node cluster with strong correlation using different differentiable pooling operations, perform Laplace smoothing and generate node embedding. The probability of the node being assigned to each cluster is calculated by the Softmax function, and then the node cluster assignment score and the intermediate node embedding are combined through a fully connected layer to generate an updated node embedding. Enable the model to aggregate relevant features between different modalities and contribute to the final node embedding and behavior recognition. This process gradually extracts node features and edge weights by repeatedly using multiple graph attention layers and cluster assignment operations, and finally obtains the correlation weight coefficient between different modalities.

[0049] (2) Dynamic knowledge-driven situation understanding basic model construction.

[0050] In view of the fact that in modern complex environment, the target object is various, and the information sensed by the mass sensor presents the characteristics of multi-source heterogeneity, which brings an explosive growth of holographic big data information, and its accumulation effect has far exceeded the acquisition, analysis, and intelligence capabilities of existing unmanned systems. The traditional situation understanding model only considers single modal information, and its situation understanding ability in complex scenarios is limited, and in multi-modal information, different network model architectures need to be carefully designed according to different modal information combinations and specific tasks. This method introduces the prior knowledge of the graph structure situation scenario, constructs a unified modal encoder to obtain high-dimensional multi-modal information representation, and uses the feature extraction and generalization ability of the pre-trained large language model to realize the identification, classification, prediction and other situation understanding tasks of the target object behavior.

[0051] This method adopts a unified situation understanding deep learning situation understanding basic model based on the Transformer structure for multi-modal multi-task learning under unpaired data. The model mainly consists of four parts: a data serialization unified representation module for various modal data to be serialized and embedded into a shared manifold space; a behavior prior knowledge fusion module for mining the correlation between current situation information and prior knowledge; a pre-trained cross-modal shared representation extraction module for extracting common representations between different feature representations; and a downstream specific task decoder for decoding shared features to obtain the features required by the downstream task. The model architecture is shown in FIG. 3.

[0052] To realize the unified serialization representation of multi-modal data, the method adopts a multi-modal data structure feature to construct a separate serialization unified representation method, and the multi-modal information includes message information, target key point information, target moving track, electromagnetic spectrum, and graph structure scene situation prior knowledge, etc. First, the multi-modal data is serialized, for example, for message information, the method adopts a double-byte coding based (WordPiece) embedding method to divide the original sentence into sub-sentences, realizes natural language coding to ensure that the network learns the semantic information contained in the language; for key point information in the image, the method remodels the whole image into a flattened sequence information, and assigns special semantic information to the pixels containing key points, which facilitates the network model to mine the internal relevance between key point pixels and the whole image; for target moving track, the method adopts a specific time window to construct track position coding under different time scales to ensure that the model can capture the moving state under different time scales in complex environments; for voice data, the method uses a filter to obtain spectrum information, and combines the filter overlap window mechanism to obtain spectrum representation under different windows, realizing the conversion from spectrum expansion representation to sequence representation.

[0053] The method adopts a pre-training visual large language model based on Transformer to extract features from sequence information from different modalities. The model adopts ViT as the backbone network structure, and completes pre-training in a large amount of language image data. The multi-modal information with completed sequence representation is input into the large model to extract general high-level representations between modalities. Only the deep layer of the pre-training model is trained in this process.

[0054] Meanwhile, a behavior knowledge prior fusion module is constructed, which uses a multi-head attention mechanism to obtain the association features of different modal behavior semantics and graph prior knowledge, and finally fuses the extracted high-level semantic general and association features between modalities.

[0055] Finally, a downstream decoder is constructed to understand downstream tasks such as target classification, behavior relationship judgment, and behavior similarity judgment, etc., to improve the semantic feature extraction capability of the model.

[0056] To realize the situation understanding of modern complex scene environment, in view of the problem that the current method cannot effectively utilize multi-modal information and the separate situation understanding task network consumes energy, a unified situation understanding deep learning model based on Transformer structure is proposed, and combined with unified sequence coding, pre-training large language model, etc. Module, realize the construction of multi-modal multi-task unified situation understanding field large model.

[0057] (3) Construction of situation understanding enhancement model with lifelong continuous learning.

[0058] In view of the understanding ability and understanding focus of the situation understanding model in different use scenarios, the method uses the idea of lifelong continuous learning to build a situation understanding enhanced model based on the basic situation understanding model. The model structure is shown in FIG. 4, which mainly includes a knowledge transfer module, a human-in-the-loop evaluation module, and a self-optimization iteration module. The specific scheme is as follows: Single scene information is used to drive the situation understanding model in real time to obtain situation understanding prediction values. However, considering the diversity and complexity of complex scene environments, the situation understanding model in a single scene does not have generalization for dynamic scenes. The method introduces the theory of lifelong continuous learning to modify the situation understanding model parameters and build a situation understanding enhanced model, which closely combines artificial intelligence and human expertise. Human expert knowledge is incorporated into the training process and is responsible for evaluating the prediction results. Due to the special nature of the situation understanding task of lifelong continuous learning for complex scene environments, the method does not need to construct a dataset in advance. The situation understanding results output by the enhanced model are evaluated by human experts, and the evaluation results are used as part of the parameter iterative optimization to iteratively modify the model parameters. The following content will describe the important modules in the situation understanding enhanced model based on lifelong continuous learning in this method.

[0059] First, the knowledge transfer module is introduced to extract the global and local relevance of the model parameters for a single scene in the form of a combination of graph neural networks and convolutional neural networks, and the results are superimposed to obtain the model initialization parameters for the current arbitrary scene. Since the model tasks are the same, the initialization model parameters only consider updating the output layer parameters of the previous situation understanding model.

[0060] In addition, due to the complexity and incompleteness of complex scene data, the theory of Human-in-the-Loop is introduced, which incorporates relevant complex scene analysis experts into the training of the model: combining the data correlation obtained from the multi-source heterogeneous data knowledge base and the perception model, the situation understanding model that completes the model parameter transfer is used to implement the situation understanding task for the current scene data; the situation understanding results will be evaluated by the artificial judgment evaluation module performed by experts, and an iterative optimization system based on deep reinforcement learning is used to quantify the evaluation results of experts; the quantified evaluation results are used for parameter optimization update in the next iteration. Since the understanding results of artificial intelligence in situation understanding have confidence confusion problems, the introduction of the Human-in-the-Loop evaluation module and the optimization iteration module for quantifying human evaluation can greatly improve this problem, so that the situation understanding model can output results that conform to human logic according to human understanding thinking.

[0061] The loss function of the situation understanding enhancement model based on lifelong continuous learning is defined as follows: wherein, is a penalty coefficient, is a parameter deviation of the knowledge transfer network, and the change is used to control the penalty for changing the parameters of the knowledge transfer module. represents the artificial evaluation results obtained by the "man-in-the-loop" evaluation module and the quantitative artificial evaluation results after the optimization iteration module. This method uses the L2 norm weighted sum of each result as the quantitative part of the optimization iteration module, and the weight here is also defined by the experts in the "man-in-the-loop" module.

[0062] Considering the complexity and uncertainty of complex scene environments, the "man-in-the-loop" evaluation module and the self-optimization iteration module are non-essential modules in the application stage of the model. When there is a condition that the model situation understanding result cannot be evaluated in time due to data transmission delay, etc., the model can directly output the result through the situation understanding model. The results not evaluated will be automatically stored and the model will be updated through the open interface of the "man-in-the-loop" evaluation module and the self-optimization iteration module in the future.

[0063] Referring to Figure 5 , the embodiments of the present application can also provide a complex scene situation understanding system, as shown in Figure 5 , for executing the complex scene situation understanding method described above, which can include: A multi-modal feature acquisition unit 501 is configured to acquire multi-modal data related to a scene situation, wherein the multi-modal data at least includes text language data, target key point data, target moving trajectory data, electromagnetic spectrum data, and graph structure priori knowledge data. A situation understanding execution unit 502 is configured to input the multi-modal data into a situation understanding basic model for realizing unified situation understanding deep learning, so that the situation understanding basic model outputs a behavior understanding result of a target, wherein the behavior understanding result at least includes a target action recognition result, a target action classification result, and a target action prediction result. The situation understanding basic model includes a data serialization unified representation module, a cross-modal extraction shared representation module, a behavior priori knowledge fusion module, and a downstream specific task decoder. After the situation understanding basic model receives the multi-modal data, the data serialization unified representation module is used to serialize and embed various modal data into a shared manifold space. The cross-modal extraction shared representation module is used to extract a general representation between different feature representations. The behavior prior knowledge fusion module is used to mine an association relationship between current behavior situation information and the graph structure prior knowledge data. The general representation is decoded by the downstream specific task decoder to obtain a target behavior understanding result required by a downstream task.

[0064] Embodiments of the present application can also provide a complex scene situation understanding device, which comprises a processor and a memory: The memory is configured to store program code and transmit the program code to the processor. The processor is configured to execute the steps of the complex scene situation understanding method according to instructions in the program code.

[0065] As shown in Figure 6 The complex scene situation understanding device provided by the embodiments of the present application can comprise a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 can communicate with each other through the communication bus 13.

[0066] In the embodiments of the present application, the processor 10 can be a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0067] The processor 10 can call a program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiments of the complex scene situation understanding method.

[0068] The memory 11 is configured to store one or more programs, which can include program code comprising computer operation instructions. In the embodiments of the present application, the memory 11 at least stores programs for implementing the following functions: Obtain multi-modal data related to a scene situation, which at least includes text language data, target key point data, target moving track data, electromagnetic spectrum data and graph structure prior knowledge data; Input the multi-modal data into a situation understanding basic model for realizing unified situation understanding deep learning, so that the situation understanding basic model outputs a target behavior understanding result, which at least includes a target action recognition result, a target action classification result and a target action prediction result; The situation understanding basic model comprises a data serialization unified representation module, a cross-modal extraction shared representation module, a behavior prior knowledge fusion module and a downstream specific task decoder. After receiving the multimodal data, the situation understanding basic model uses the data serialization unified representation module to perform unified serialization processing on the various modal data and embed them into a shared manifold space; Extracting common representations between different feature representations using the cross-modal shared representation extraction module; Utilizing the behavior prior knowledge fusion module to mine the correlation between the current behavior situation information and the graph structure prior knowledge data; The downstream specific task decoder is used to decode the general representation to obtain the target behavior understanding result required by the downstream task.

[0069] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0070] In addition, the memory 11 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0071] The communication interface 12 may be an interface of a communication model, used for connecting to other devices or systems.

[0072] Of course, it needs to be explained that Figure 6 The structure shown does not constitute a limitation on the complex scene situation understanding device in the embodiment of the present application. In actual applications, the complex scene situation understanding device may include Figure 6 More or fewer components than shown, or combinations of certain components.

[0073] An embodiment of the present application may also provide a computer-readable storage medium, which is used to store program code, and the program code is used to execute the steps of the above-mentioned complex scene situation understanding method.

[0074] It should be noted that, in the specification, relational terms such as first and second, and the like, can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0075] Those skilled in the art can clearly understand that the application can be implemented by means of software plus a necessary universal hardware platform from the description of the above embodiments. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments of the application.

[0076] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts of each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system or system embodiments, since it is basically similar to the method embodiments, it is described more simply, and the relevant parts can be referred to the part of the description of the method embodiments. The above-described system and system embodiments are merely illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to the actual needs. Those skilled in the art can understand and implement it without creative labor.

[0077] The above only describes the preferred embodiments of the application and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application is included in the protection scope of the application.

Claims

1. A complex scene situation understanding method, characterized by: include: Acquire multimodal data related to the scene situation, wherein the multimodal data includes at least text language data, target key point data, target movement trajectory data, electromagnetic spectrum data, and graph structure prior knowledge data; Inputting the multimodal data into a situation understanding basic model for implementing unified situation understanding deep learning, so that the situation understanding basic model outputs a target behavior understanding result, the target behavior understanding result including at least a target action recognition result, a target action classification result, and a target action prediction result; The situation understanding basic model includes a data serialization unified representation module, a cross-modal extraction shared representation module, a behavior priori knowledge fusion module, and a downstream specific task decoder. After receiving the multimodal data, the situation understanding basic model uses the data serialization unified representation module to perform unified serialization processing on the various modal data and embed them into a shared manifold space; Extracting common representations between different feature representations using the cross-modal shared representation extraction module; Utilizing the behavior prior knowledge fusion module to mine the correlation between the current behavior situation information and the graph structure prior knowledge data; The downstream specific task decoder is used to decode the general representation to obtain the target behavior understanding result required by the downstream task.

2. The complex scene situation understanding method according to claim 1, characterized in that: The unified serialization process includes: For the text language data, the original sentence is divided into clauses by adopting an embedding method based on double-byte encoding, and natural language encoding is implemented to ensure the semantic information contained in the network learning language; For the target key point data, the entire image is reshaped into a flat sequence information, and the pixels containing the key points are assigned with specified semantic information; For the target movement trajectory data, a target time window is used to construct trajectory position codes at different time scales; The electromagnetic spectrum data is combined with a filter overlapping window mechanism to obtain spectrum representations under different windows, thereby realizing the conversion from spectrum expansion representation to sequence representation.

3. The complex scene situation understanding method according to claim 1, characterized in that: The cross-modal shared representation extraction module uses a Transformer-based pre-trained visual language model to extract features from sequence information from different modalities.

4. The complex scene situation understanding method according to claim 1, characterized in that: The behavior prior knowledge fusion module uses a multi-head attention mechanism to obtain the association features of different modal behavior semantics and graph prior knowledge, and fuses the extracted semantic commonality and association features between modalities to obtain the association relationship.

5. The complex scene situation understanding method according to claim 1, characterized in that: The downstream task-specific decoder is fine-tuned through a variety of situation understanding downstream tasks.

6. The complex scene situation understanding method according to claim 1, characterized in that: The graph structure prior knowledge data is obtained by analyzing prior multimodal data using a behavior association analysis model; the association analysis model includes a cross-modal association graph encoding module and a behavior association recognition module; The cross-modal association graph encoding module learns the semantic correspondence between different modalities by constructing a heterogeneous cross-modal graph fusion model; The behavioral association recognition module processes heterogeneous graph networks through differentiable pooling and graph attention networks, identifies semantic associations within and across modalities, generates node embeddings, combines cross-modal graphs to represent information from different modalities, and uses an interaction graph encoder module to realize semantic association recognition, so as to provide behavioral characteristics and association strength of multimodal targets in different contexts.

7. The complex scene situation understanding method according to claim 1, characterized in that: A lifelong continuous learning mechanism is used to modify the parameters of the situation understanding basic model to obtain a situation understanding enhanced model; the lifelong continuous learning mechanism includes a knowledge transfer module, a human-in-the-loop evaluation module, and a self-optimization iteration module; The knowledge transfer module extracts the global and local correlations of model parameters for a single scenario by combining graph neural networks and convolutional neural networks, and superimposes the results to obtain the model initialization parameters for any current scenario; The human-in-the-loop evaluation module evaluates the situation understanding prediction result output by the situation understanding network after the parameters are updated to obtain a manual evaluation result; The self-optimization iteration module optimizes and iterates the manual evaluation results to obtain quantified manual discrimination experience, so as to utilize the manual discrimination experience for optimizing and updating the parameters of the situation understanding basic model in the next iteration.

8. A complex scene situation understanding system, characterized by: The system for executing the complex scene situation understanding method according to any one of claims 1 to 7 comprises: A multimodal feature acquisition unit, configured to acquire multimodal data related to scene status, wherein the multimodal data includes at least text language data, target key point data, target movement trajectory data, electromagnetic spectrum data, and graph structure prior knowledge data; a situation understanding execution unit, configured to input the multimodal data into a situation understanding basic model for implementing unified situation understanding deep learning, so that the situation understanding basic model outputs a target behavior understanding result, the target behavior understanding result including at least a target action recognition result, a target action classification result, and a target action prediction result; The situation understanding basic model includes a data serialization unified representation module, a cross-modal extraction shared representation module, a behavior priori knowledge fusion module, and a downstream specific task decoder. After receiving the multimodal data, the situation understanding basic model uses the data serialization unified representation module to perform unified serialization processing on the various modal data and embed them into a shared manifold space; Extracting common representations between different feature representations using the cross-modal shared representation extraction module; Utilizing the behavior prior knowledge fusion module to mine the correlation between the current behavior situation information and the graph structure prior knowledge data; The downstream specific task decoder is used to decode the general representation to obtain the target behavior understanding result required by the downstream task.

9. A complex scene situation understanding device, characterized in that: The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the complex scene situation understanding method described in any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the complex scene situation understanding method described in any one of claims 1-7.

Images