Explanatable target group behavior discrimination method

Abstract

The invention belongs to the technical field of situation cognition, and particularly discloses an interpretable target group behavior discrimination method, which comprises the following steps: acquiring multi-source situation information data of a detection target group; based on the multi-source situation information data, obtaining a spatial-temporal distribution feature vector of the detection target; constructing a to-be-learned double-layer graph structure based on the multi-source situation information data and the spatial-temporal distribution feature vector; on the basis of the spatial and temporal distribution feature vectors and the to-be-learned graph structure, the behavior category of the detection target group is judged through a relational graph neural network; and through a deep structure causal model SCM, the discrimination result of the relational graph neural network can be explained.

Description

Technical Field

[0001] The present application belongs to the technical field of situation recognition, and more specifically, relates to an explainable target group behavior discrimination method. Background Art

[0002] In situational awareness, limiting the analysis of individual behaviors will waste combat resources and easily be confused and deceived by the opponent's tactical behaviors. Considering that individual behaviors are usually guided by the group, the comprehensive behavior analysis of the target group is often more important than the analysis of a single target. The recognition of group behavior is the premise and foundation for selecting core targets and achieving key defense.

[0003] As detection methods tend to be diversified, the identification of target group behavior has expanded from relying solely on track data to relying on multiple modal data including target text descriptions, optical images and even videos. The introduction of multimodal data helps to more accurately identify target group behavior. However, due to the large differences in the attributes of multi-source heterogeneous multimodal data, deep learning analysis methods must be introduced to effectively use multimodal data. However, the widely used "end-to-end" deep learning method requires a large amount of data to support training, and the quality of the method depends heavily on data quality. In complex environments, it is very common to have information missing in the target perception process and incomplete target information acquisition. Both the amount of data and the quality of data are difficult to effectively meet the training requirements of the model. At the same time, although the application of this type of deep learning method can improve the accuracy of group behavior analysis, it is unexplainable because its model training is data-driven and its internal structure is a "black box". For combat missions, unexplainability means that there are decision-making risks, which is a more serious defect. Summary of the invention

[0004] In view of the defects of the prior art, the purpose of this application is to provide an interpretable target group behavior discrimination method, aiming to solve the problem that the behavior discrimination results of the existing multimodal methods are unexplainable and lack reliability.

[0005] To achieve the above objectives, in a first aspect, the present application provides an interpretable target group behavior identification method, comprising:

[0006] Acquire multi-source situation information data of the detection target group;

[0007] Based on the multi-source situation information data, obtaining a spatiotemporal distribution feature vector of the detection target;

[0008] Based on the multi-source situation information data and the spatiotemporal distribution feature vector, construct a two-layer graph structure to be learned;

[0009] Based on the spatiotemporal distribution feature vector and the graph structure to be learned, the behavior category of the detection target group is discriminated through a relational graph neural network;

[0010] The discrimination results of the relationship graph neural network are made explainable through a deep structural causal model (SCM).

[0011] This application obtains cross-modal multi-source situation information data of the detected target group, extracts the spatiotemporal distribution feature vectors from it as the input of the relationship graph neural network, and combines the two-layer graph structure for group behavior prediction learning and structural causal learning, so that the relationship graph neural network learns the overall structural characteristics of the spatiotemporal data and gives the target group behavior category judgment results. Then, a deep structural causal model is used to obtain explainable causal relationships for the discrimination results, making the behavior discrimination results explainable and improving reliability.

[0012] According to an interpretable target group behavior identification method provided by the present invention, the step of obtaining multi-source situation information data of the detected target group includes:

[0013] The multi-source situation information data of the detection target group is obtained, and the multi-source situation information data is stored and managed in the form of a double index "Key-Value" key-value pair.

[0014] This application uses dual-index "Key-Value" key-value pairs to store and manage data, which can effectively organize and manage large-scale and complex target detection data, and realize fast storage and efficient use of data.

[0015] According to an interpretable target group behavior discrimination method provided by the present invention, the spatiotemporal distribution feature vector of the detected target is obtained based on the multi-source situation information, including:

[0016] A pre-trained deep learning multimodal encoder is used to encode multi-source situation information data of different modalities using different initialization methods to obtain the initial vector;

[0017] The time information and location information of the multi-source situation information data are spliced ​​to the end of the initial vector, and the spliced ​​vector is used as the input of the pre-trained multimodal Chinese open source large model. A Linear Layer is spliced ​​at the output position of the large model to adjust the distribution of the data, so as to obtain the spatiotemporal distribution feature vector output by the large model.

[0018] This application adopts a pre-trained deep learning multimodal encoder, and uses different initialization methods to encode multi-source situation information data of different modalities, and splices the time information and location information to the end of the initial vector. The spliced ​​vector is used as the input of the pre-trained multimodal Chinese open source large model, and a Linear Layer is spliced ​​at the output position of the large model to adjust the distribution of the data, which can fully reflect the temporal and spatial characteristics of the data.

[0019] According to an interpretable target group behavior discrimination method provided by the present invention, the double-layer graph structure to be learned is constructed based on the multi-source situation information data and the spatiotemporal distribution feature vector, including:

[0020] Based on the multi-source situation information data, construct a lower instance layer in the graph structure to be learned;

[0021] Based on the spatiotemporal distribution feature vector and the lower instance layer, an upper attribute layer in the graph structure to be learned is constructed.

[0022] This application takes all the collected situation-related data including environment, location, track, image, etc. as examples, and incorporates them into the graph structure according to their mutual correlation. Then, based on the specific instances of the lower layer, the attribute nodes of the upper layer are abstracted to construct the upper attribute layer graph structure, so that the relational graph neural network can better predict the behavior category of the detection target group.

[0023] According to an interpretable target group behavior discrimination method provided by the present invention, based on the spatiotemporal distribution feature vector and the graph structure to be learned, the behavior category of the detected target group is discriminated through a relational graph neural network, including:

[0024] Based on the spatiotemporal distribution feature vector and the graph structure to be learned, encoding the nodes in the relational graph neural network to obtain multiple group targets;

[0025] Under the premise of maintaining the learning complexity of the relational graph neural network, the calculation form of vector addition is directly adopted for multiple group targets, and the features of the same group targets are grouped together to obtain the feature set vector;

[0026] The feature set vector is passed through a multi-layer neural network to determine the behavior category of the detection target group.

[0027] According to an interpretable target group behavior discrimination method provided by the present invention, the discrimination result of the relationship graph neural network is made interpretable through the deep structure causal model SCM, including:

[0028] Through the deep structured causal model SCM, the counterfactual logic judgment method is adopted to learn the causal relationship between the attribute layer nodes according to the relationship mapped from the instance layer to the attribute layer, so that the judgment result of the relationship graph neural network can be explained.

[0029] This application uses a deep structural causal model and a counterfactual logic judgment method to learn the causal relationship between attribute layer nodes based on the relationship between the instance layer and the attribute layer, thereby gaining an in-depth and comprehensive understanding of the target behavior type and its causes.

[0030] According to an interpretable target group behavior discrimination method provided by the present invention, after making the discrimination result of the relationship graph neural network interpretable through the deep structure causal model SCM, the method further includes:

[0031] Completing the lower instance layer in the two-layer graph structure to be learned based on the causal relationship;

[0032] The relationship graph neural network is fine-tuned based on the completed lower instance layer.

[0033] This application uses the learned causal relationship to guide the fine-tuning of the relationship graph neural network to alleviate the decrease in the accuracy of target group behavior discrimination caused by missing and incomplete information.

[0034] In a second aspect, the present application provides an interpretable target group behavior identification device, comprising:

[0035] The first acquisition module is used to acquire multi-source situation information data of the detection target group;

[0036] A second acquisition module is used to obtain a spatiotemporal distribution feature vector of the detection target based on the multi-source situation information data;

[0037] A construction module, used to construct a two-layer graph structure to be learned based on the multi-source situation information data and the spatiotemporal distribution feature vector;

[0038] A discrimination module, used to discriminate the behavior category of the detection target group through a relational graph neural network based on the spatiotemporal distribution feature vector and the graph structure to be learned;

[0039] The interpretable module is used to make the discrimination result of the relationship graph neural network interpretable through a deep structural causal model SCM.

[0040] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the interpretable target group behavior identification method described in the first aspect or any possible implementation method of the first aspect.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the interpretable target group behavior discrimination method described in the first aspect or any possible implementation of the first aspect.

[0042] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the interpretable target group behavior identification method described in the first aspect or any possible implementation of the first aspect.

[0043] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0044] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0045] The cross-modal multi-source situation information data of the detected target group is obtained, and the spatiotemporal distribution feature vector is extracted from it as the input of the relational graph neural network. The double-layer graph structure is combined for group behavior prediction learning and structural causal learning, so that the relational graph neural network learns the overall structural characteristics of the spatiotemporal data and gives the target group behavior category judgment results. Then, the deep structural causal model is used to obtain explainable causal relationships for the discrimination results, making the behavior discrimination results explainable and improving reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 This is one of the flow charts of an interpretable target group behavior discrimination method provided in an embodiment of the present application;

[0048] Figure 2 This is the second flow chart of the explainable target group behavior discrimination method provided in the embodiment of the present application;

[0049] Figure 3 It is a structural schematic diagram of an interpretable target group behavior discrimination device provided in an embodiment of the present application;

[0050] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0053] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0054] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0055] Next, combine Figure 1-Figure 2 An explainable target group behavior discrimination method provided in an embodiment of the present application is introduced.

[0056] Figure 1 is one of the flow charts of an interpretable target group behavior identification method provided in the embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0057] Step 100, obtaining multi-source situation information data of the detection target group;

[0058] Optionally, the multi-source situation information data may be situation information obtained by fusing the time position information of the detected target group, the multi-modal information of the target group, and the like.

[0059] Optionally, the present application does not limit the method of acquiring and fusing multi-source situation information data.

[0060] Step 110, obtaining a spatiotemporal distribution feature vector of the detection target based on the multi-source situation information data;

[0061] After obtaining multi-source situation information data, the distribution feature vector of the spatiotemporal trajectory data is constructed, and these spatiotemporal data are converted into data that can be understood by the relational graph neural network to facilitate the subsequent category judgment and division. It is used as the input variable of the relational graph neural network, and the distribution features are extracted from the constructed time series feature vector as the classification criterion of the deep learning network to capture the distribution characteristics of the complete trajectory data.

[0062] Step 120, constructing a two-layer graph structure to be learned based on multi-source situation information data and spatiotemporal distribution feature vectors;

[0063] The present application constructs a two-layer graph structure to be learned based on multi-source situation information data and spatiotemporal distribution feature vectors, and the graph structure includes a lower-layer instance graph and an upper-layer attribute layer that are mutually related.

[0064] Step 130, based on the spatiotemporal distribution feature vector and the graph structure to be learned, the behavior category of the detection target group is discriminated through a relational graph neural network;

[0065] For graph structures containing multiple types of relationships, the Relational Graph Convolutional Network (RGCN), which can distinguish multiple relationships in the field of graph representation learning, is more suitable for learning complex graph relationships.

[0066] The core of RGCN lies in the encoding of nodes. This application encodes the nodes of RGCN based on the spatiotemporal distribution feature vector and the graph structure to be learned, and then distinguishes the behavior category of the detection target group.

[0067] Step 140, using the deep structural causal model SCM, makes the discrimination results of the relational graph neural network explainable.

[0068] In the field of interpretable learning, Structural Causal Model (SCM) is widely used. SCM is a generalization and improvement of Bayesian network. It has the advantages of traditional Bayesian network and adds Structural Equation Model (SEM) commonly used in engineering field. When dealing with a large number of unknown variable values, SCM can also use marginalization integral operation to remove the influence, so as to carry out probability-based causal inference without obstacles.

[0069] In the field of image recognition, some studies have built the DeepSCM model based on the principles of SCM and the method of deep learning. DeepSCM can not only accurately identify image categories, but also realize the causal relationship mining that affects the image presentation effect, which provides a reference for the causal reasoning research of deep models in other fields.

[0070] This application applies deep SCM to perform causal reasoning learning at the attribute layer, so as to achieve the purpose of making counterfactual logical judgments in the target task interpretation, deeply and comprehensively understand the target behavior type and its inducements, and make the judgment results of the relational graph neural network explainable.

[0071] The present application provides an interpretable target group behavior discrimination method, which first obtains cross-modal multi-source situation information data of the detected target group, extracts the spatiotemporal distribution feature vector from it as the input of the relationship graph neural network, and combines it with a two-layer graph structure for group behavior prediction learning and structural causal learning, so that the relationship graph neural network learns the overall structural characteristics of the spatiotemporal data and gives the target group behavior category judgment result, and then uses a deep structural causal model to obtain an interpretable causal relationship for the discrimination result, so that the behavior discrimination result can be interpreted and the reliability is improved.

[0072] In some embodiments, step 100 specifically includes:

[0073] Step 1001, obtain multi-source situation information data of the detection target group, and store and manage the multi-source situation information data in the form of dual index "Key-Value" key-value pairs.

[0074] Since situation information has typical temporal and spatial attributes, the detection cycle of mainstream detection equipment is generally 1 second. In the detection task, as time goes on, the scale of detection data will become very large. Moreover, situation information mainly includes the time location information of the target. Depending on the detection method, it often includes text reports of the target, target detection images and other information. The detection data of the target contains multiple modes and the data composition is relatively complex. At the same time, there are multiple correlations between these data, which further increases the complexity of the data.

[0075] In order to effectively organize and manage large-scale and complex target detection data, this application uses dual-index "Key-Value" key-value pairs to store and manage data, thereby achieving fast storage and efficient use of data.

[0076] Specifically, firstly, a high-dimensional hash function is used to hash all the data to obtain a unique hash code for each data; then, the first-level index structure is constructed, with "[time]-[location]-[data hash value]" as the underlying index key value and "[data]" as the value to store all the data. The time and location are stored in the form of "year-month-day hour-minute-second" and "precision latitude-altitude" respectively, with the time accurate to seconds and the location accurate to six decimal places (i.e. accurate to meters).

[0077] Next, we build the second index according to the type of relationship. First, we classify the relationships between data. Here, we use rules to distinguish different types of relationships by communicating with the experts who generate the model for the previous data relationship.

[0078] For example, the commonly existing chronological relationship between target track point data, the correlation relationship between target track points and target images, the correlation relationship between target track points and target reports, and the mutual correlation relationship between target track points and other target track points. Like data, relationships have time and space attributes. However, the difference is that its spatial attributes involve two objects. Therefore, for the index storage of "relationship C between data A->data B", "[time]-[relationship type]-[hash value of data A]" is used as the index Key value, and "[hash value of data B-relationship C]" is used as the Value to realize the storage of all data.

[0079] By building dual indexes, you can efficiently query data at any time and any location and its related relationships. At the same time, you can directly append new data and relationships through queries. To delete a piece of data, you can also delete it by querying based on the index key value.

[0080] In some embodiments, step 110 specifically includes:

[0081] Step 1101, using a pre-trained deep learning multi-modal encoder, encoding multi-source situation information data of different modalities using different initialization methods to obtain an initial vector;

[0082] Step 1102, splice the time information and location information of the multi-source situation information data to the end of the initial vector, use the spliced ​​vector as the input of the pre-trained multimodal Chinese open source large model, and splice the Linear Layer at the output position of the large model to adjust the distribution of the data, and obtain the spatiotemporal distribution feature vector output by the large model.

[0083] This application adopts an open source pre-trained deep learning multimodal encoder and uses the collected multi-source situation information data to construct a distribution feature vector of data with spatiotemporal characteristics.

[0084] The Transformer-based big model itself has the potential to process spatiotemporal feature data. In recent years, based on general open source big models, multimodal time series big models represented by VideoLLM, GPT4STM, etc. have performed excellently in fields such as video processing and financial data processing.

[0085] In one embodiment of the present application, based on Chinese open source large models such as GLM and VisCPM, and drawing on the time series processing architecture of GPT4STM, different initialization methods are used for data of different modalities. For time series data such as track, Linear Embedding is first used to encode to obtain the initial vector; for text type data, BERT Embedding is first used to encode to obtain the initial vector; for image type data, Transformer Embedding is used to encode to obtain the initial vector.

[0086] Furthermore, in order to fully reflect the spatiotemporal characteristics of the data, it is necessary to directly concatenate the "[time][location]" information of the data to the end of the initial vector, and use the concatenated vector as the input of the pre-trained multimodal Chinese open source large model, and concatenate the Linear Layer at the output position of the large model to adjust the distribution of the data. During the entire model training process, the parameters of the large model remain fixed, and the large model's general understanding of multi-modal data is used to encode the above-mentioned multi-temporal and multi-modal data.

[0087] In some embodiments, step 120 specifically includes:

[0088] Step 1201, constructing a lower instance layer in the graph structure to be learned based on multi-source situation information data;

[0089] Step 1202: construct an upper attribute layer in the graph structure to be learned based on the spatiotemporal distribution feature vector and the lower instance layer.

[0090] This application first constructs the instance layer graph structure of the lower layer, and takes the collected multi-source situation information data, including all situation-related data such as environment, location, track, image, etc. as instances, and incorporates them into the graph structure based on their mutual correlation. Then, based on the specific instances of the lower layer, the attribute nodes of the upper layer are abstracted to construct the upper attribute layer graph structure.

[0091] Specifically, the effective identification of group behavior requires the support of relatively basic data such as the relative positions between targets, the equipment carried by targets, and the historical activities of targets. Therefore, it is necessary to include all situation-related data such as environment, position, track, image, etc. into the data source of the graph structure required for behavior identification. Based on the multi-source situation information data, a retrieval method is used to connect all the relationships contained therein to construct the lower-level instance graph.

[0092] According to the specific examples of the lower layer, the form of unsupervised clustering and rule induction is used in combination to summarize and abstract the attribute nodes of the upper layer and construct the upper attribute layer graph structure. Specifically, the initial vector obtained by encoding (the vector has not been processed by the large model) is first clustered using unsupervised clustering to cluster data with similar features. It should be noted that since time and space information can be easily used as a basis for clustering, but this type of information is irrelevant to the attribute value, the initial vector used here does not carry the "[time][place]" information.

[0093] Next, similar to the construction of the attribute layer and relationship classification, unsupervised clustering is guided by domain experts and summarized based on existing examples.

[0094] In some embodiments, step 130 specifically includes:

[0095] Step 1301, encoding nodes in the relationship graph neural network based on the spatiotemporal distribution feature vector and the graph structure to be learned, to obtain multiple group targets;

[0096] Step 1302, under the premise of maintaining the learning complexity of the relationship graph neural network, directly adopt the calculation form of vector addition for multiple group targets, and set the features of the same group targets together to obtain a feature set vector;

[0097] Step 1303, the feature set vector is passed through a multi-layer neural network to determine the behavior category of the detection target group.

[0098] This application is made through m (·) function encodes the nodes in the relational graph neural network. In the standard RGCN, g m (·) is usually defined as a simple linear transformation with normalization capability, that is, Among them, e i and e j They are graph nodes e i and e j The graph input vector, that is, the spatial and temporal distribution feature vector, For node e i For j The Laplace matrix of , W is the neural network transformation weight matrix.

[0099] When the graph structure is more complex, the use of the neural network transformation weight matrix W and autoregression will greatly increase the complexity of graph representation learning, thereby seriously affecting the performance of the graph neural network model. At the same time, the negative impact of the complexity of the model will become more serious when there is a lot of node data.

[0100] The number of spatiotemporal graph neural network nodes involved in this application is very large, so this application will use the standard gm (·) is usually simplified to It can improve the model effect while significantly reducing the complexity of node representation learning.

[0101] Based on the above improvements, from the perspective of training the model in the form of matrix update, the update method of RGCN adopted in this application is as follows:

[0102]

[0103] in, Represents graph node e i In graph convolution, we learn the vector of layer l, Represents node e under relationship r i The neighbor index set of It is a normalization constant for different relationships, which can be obtained through learning or pre-set. Through the above RGCN learning that distinguishes various types of relationships, the representation vector of each node carrying relationship information can be learned on the topological structure of the instance layer graph.

[0104] In particular, during the initialization of graph network nodes, for nodes that can be encoded, such as images and tracks, their codes of the same length are used as the initial vectors for graph learning; for other instance layer nodes, they are initialized in a random manner.

[0105] In the standard data sample (Ground Truth Sample) of group target behavior discrimination, a group of targets constitutes a group, and the group is marked with a certain behavior towards one or more targets outside the group, where the behavior of the target group towards other targets can be patrolling, tracking, surveillance, air defense, anti-submarine, etc. However, due to the uncertainty of the number of targets in the group, it is impossible to directly use multi-layer neural network (MLP) to achieve pattern matching.

[0106] Considering that after the graph neural network representation learning, the target vectors are all located in the same vector space, which has the premise of simple vector calculation. Therefore, under the premise of maintaining the complexity of graph neural network learning, the multiple group targets are directly calculated in the form of vector addition, and the features of the same group targets are grouped together to obtain the feature set vector. Then the obtained feature set vector is passed through a multi-layer neural network (Multilayer Perceptron, MLP) to realize the discrimination of the target group behavior.

[0107] In some embodiments, step 140 specifically includes:

[0108] Step 1401, through the deep structural causal model SCM, adopting the counterfactual logic judgment method, according to the relationship mapped from the instance layer to the attribute layer, learn the causal relationship between the attribute layer nodes, so that the judgment result of the relationship graph neural network can be explained.

[0109] Since the graph convolutional network group behavior recognition works on the instance layer, it completes the representation learning of the instance layer graph nodes through parameter learning within the network, and then applies the learned representation combined with MLP to predict the type of target group behavior. This process has the characteristics of a typical deep learning method, and its inference process is difficult for humans to understand.

[0110] This application uses deep SCM, a causal reasoning learning method that can adopt counterfactual logic judgment to learn the causal relationship between attribute layer nodes based on the relationship between the instance layer and the attribute layer, thereby gaining an in-depth and comprehensive understanding of the target behavior type and its causes, making the judgment results of the relationship graph neural network explainable.

[0111] Deep SCM is composed of MLP combined with an attribute layer graph structure. The training uses a reward mechanism and stochastic gradient descent technology to automatically learn the parameters in the MLP.

[0112] Furthermore, this technology uses the MLP structure to model independent causal mechanisms. The input layer of each MLP contains M×N neurons. Specifically, it is M N-dimensional One-Hot vectors. The middle hidden layer can be selected arbitrarily. Generally speaking, it is composed of Max(4M,4N) neurons. The subsequent activation layer selects the LeakyReLU structure. The final output layer contains N neurons to represent the logarithmic likelihood of each task classification.

[0113] In some embodiments, after step 140, the method further includes:

[0114] Step 150, completing the lower instance layer in the two-layer graph structure to be learned based on the causal relationship;

[0115] Step 160, fine-tune the relationship graph neural network based on the completed lower instance layer.

[0116] Figure 2 This is the second flow chart of the interpretable target group behavior identification method provided in the embodiment of the present application. Figure 2As shown, in one embodiment of the present application, both the expert knowledge and the learned causal relationship are stored in a tree structure, and then a heuristic search method is used to match the causal relationship with the existing expert knowledge to further improve the causal analysis chain. Finally, a feedback learning method is used to use the obtained causal relationship rules to try to infer the lower instance layer of the graph structure, and based on the trained RGCN model parameters, the new instance layer data is used for fine-tuning to achieve more accurate group behavior prediction.

[0117] Combine the following Figure 3 An interpretable target group behavior identification device provided by the present invention is described. The interpretable target group behavior identification device described below and the interpretable target group behavior identification method described above can correspond to each other.

[0118] Figure 3 is a schematic diagram of the structure of an interpretable target group behavior identification device provided in an embodiment of the present application, such as Figure 3 As shown, the device includes a first acquisition module 310, a second acquisition module 320, a construction module 330, a determination module 340 and an interpretable module 350, wherein:

[0119] A first acquisition module 310 is used to acquire multi-source situation information data of a detection target group;

[0120] A second acquisition module 320 is used to obtain a spatiotemporal distribution feature vector of a detection target based on multi-source situation information data;

[0121] A construction module 330, for constructing a two-layer graph structure to be learned based on multi-source situation information data and spatiotemporal distribution feature vectors;

[0122] A discrimination module 340 is used to discriminate the behavior category of the detection target group through a relational graph neural network based on the spatiotemporal distribution feature vector and the graph structure to be learned;

[0123] The interpretable module 350 is used to make the discrimination results of the relational graph neural network interpretable through the deep structural causal model SCM.

[0124] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0125] Based on the method in the above embodiment, Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, an embodiment of the present application provides an electronic device, which may include: a processor (processor) 410, a communication interface (CommunicationsInterface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 430 to execute the interpretable target group behavior discrimination method in the above embodiment.

[0126] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the interpretable target group behavior discrimination method described in each embodiment of the present application.

[0127] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the interpretable target group behavior identification method in the above embodiment.

[0128] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the interpretable target group behavior identification method in the above embodiment.

[0129] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0130] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0131] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0132] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0133] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An interpretable target group behavior identification method, characterized in that: include: Acquire multi-source situation information data of the detection target group; Based on the multi-source situation information data, obtaining a spatiotemporal distribution feature vector of the detection target; Based on the multi-source situation information data and the spatiotemporal distribution feature vector, construct a two-layer graph structure to be learned; Based on the spatiotemporal distribution feature vector and the graph structure to be learned, the behavior category of the detection target group is discriminated through a relational graph neural network; Through the deep structural causal model SCM, the discrimination results of the relationship graph neural network are made explainable.

2. The interpretable target group behavior identification method according to claim 1, characterized in that: The step of acquiring multi-source situation information data of the detection target group includes: Acquire multi-source situation information data of the detection target group, and store and manage the multi-source situation information data in the form of dual index "Key-Value" key-value pairs.

3. The interpretable target group behavior identification method according to claim 1, characterized in that: The step of obtaining the spatiotemporal distribution feature vector of the detection target based on the multi-source situation information includes: A pre-trained deep learning multimodal encoder is used to encode multi-source situation information data of different modalities using different initialization methods to obtain the initial vector; The time information and location information of the multi-source situation information data are spliced ​​to the end of the initial vector, and the spliced ​​vector is used as the input of the pre-trained multimodal Chinese open source large model. A LinearLayer is spliced ​​at the output position of the large model to adjust the distribution of the data, so as to obtain the spatiotemporal distribution feature vector output by the large model.

4. The interpretable target group behavior identification method according to claim 1, characterized in that: The step of constructing a two-layer graph structure to be learned based on the multi-source situation information data and the spatiotemporal distribution feature vector includes: Based on the multi-source situation information data, construct a lower instance layer in the graph structure to be learned; Based on the spatiotemporal distribution feature vector and the lower instance layer, an upper attribute layer in the graph structure to be learned is constructed.

5. The interpretable target group behavior identification method according to claim 1, characterized in that: The step of discriminating the behavior category of the detection target group through a relational graph neural network based on the spatiotemporal distribution feature vector and the graph structure to be learned includes: Based on the spatiotemporal distribution feature vector and the graph structure to be learned, encoding the nodes in the relational graph neural network to obtain multiple group targets; Under the premise of maintaining the learning complexity of the relational graph neural network, the calculation form of vector addition is directly adopted for multiple group targets, and the features of the same group targets are grouped together to obtain the feature set vector; The feature set vector is passed through a multi-layer neural network to determine the behavior category of the detection target group.

6. The interpretable target group behavior identification method according to claim 4, characterized in that: The deep structural causal model SCM is used to make the discrimination result of the relationship graph neural network explainable, including: Through the deep structured causal model SCM, the counterfactual logic judgment method is adopted to learn the causal relationship between the attribute layer nodes according to the relationship mapped from the instance layer to the attribute layer, so that the judgment result of the relationship graph neural network can be explained.

7. The interpretable target group behavior identification method according to claim 6, characterized in that: After making the discrimination result of the relationship graph neural network interpretable through the deep structural causal model SCM, the method further includes: Completing the lower instance layer in the two-layer graph structure to be learned based on the causal relationship; The relationship graph neural network is fine-tuned based on the completed lower instance layer.

8. An interpretable target group behavior identification device, characterized in that: include: The first acquisition module is used to acquire multi-source situation information data of the detection target group; A second acquisition module is used to obtain a spatiotemporal distribution feature vector of the detection target based on the multi-source situation information data; A construction module, used to construct a two-layer graph structure to be learned based on the multi-source situation information data and the spatiotemporal distribution feature vector; A discrimination module, used to discriminate the behavior category of the detection target group through a relational graph neural network based on the spatiotemporal distribution feature vector and the graph structure to be learned; The interpretable module is used to make the discrimination result of the relationship graph neural network interpretable through a deep structural causal model SCM.

9. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the interpretable target group behavior identification method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is enabled to execute the interpretable target group behavior identification method as described in any one of claims 1 to 7.

Images