A police situation analysis method and system based on cross-modal data

Abstract

The present invention relates to the field of big data analysis, and discloses a method and system for analyzing police situations based on cross-modal data, the method comprising obtaining character map data and clue text data. Feature extraction is performed on the character map data and clue text data to obtain character map features and clue text features. The character map features and the clue text features are encoded respectively to obtain character map codes and clue text codes. The character map codes and the clue text codes are fused to obtain a fused coding vector. The fused coding vector is input into a trained classification network for processing, and the classification network outputs the probability of the police situation category. The present invention makes full use of data from multiple modalities, avoids the need to obtain a large amount of expert knowledge, and can improve the precision, accuracy and efficiency of police situation analysis.

Description

Technical Field

[0001] The present invention relates to the field of big data analysis, and more specifically, to a police situation analysis method and system based on cross-modal data. Background Art

[0002] Public security systems require massive amounts of data to analyze police incidents and determine their types and probabilities. Currently, police incident analysis relies primarily on manual data analysis. However, with the accelerating growth of data, manual data processing is inefficient and can easily overlook or delay important incidents.

[0003] At present, scholars such as Li Yunxuan have proposed an automatic processing method for traffic warning information based on multi-task transfer learning. After using the natural language processing model in artificial intelligence to extract features from the warning text, the method uses the features to perform classification tasks, obtains the warning type and executes corresponding strategies.

[0004] However, when the above method performs police situation analysis on single text modal data, the police situation analysis results obtained have the defects of low precision and inaccuracy. Summary of the Invention

[0005] In order to overcome the defects of low precision and inaccurate analysis results in existing police situation analysis technologies, the present invention provides a police situation analysis method and system based on cross-modal data.

[0006] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0007] In a first aspect, the present invention proposes a police situation analysis method based on cross-modal data, comprising:

[0008] Get character graph data and clue text data.

[0009] Feature extraction is performed on the character graph data and clue text data to obtain character graph features and clue text features.

[0010] The character graph features and the clue text features are encoded respectively to obtain a character graph code and a clue text code.

[0011] The character graph encoding and the clue text encoding are fused to obtain a fused encoding vector.

[0012] The fused coding vector is input into a trained classification network for processing, and the classification network outputs the probability of the alarm category.

[0013] As a preferred technical solution, the encoder in the standard Transformer model is used to encode the character graph features and the clue text features respectively to obtain the character graph encoding and the clue text encoding. The specific steps include:

[0014] Performing linear transformation on the character graph features and the clue text features respectively to obtain a query matrix Q, a key matrix K, and a value matrix V corresponding to the character graph features and the clue text features;

[0015] The query matrix Q, key matrix K, and value matrix V are input into the multi-head attention module for self-attention mechanism learning. The self-attention matrices corresponding to the character graph features and clue text features are obtained respectively. The expressions are as follows:

[0016]

[0017] Among them, K T represents the transpose of the key matrix K, d K represents the dimension of the key matrix K;

[0018] The self-attention matrices corresponding to the character graph features and clue text features are summed and normalized in turn, and the normalized results are input into the fully connected network for calculation to obtain the character graph encoding and clue text encoding.

[0019] As a preferred technical solution, the fully connected network includes a fully connected layer, an activation layer, a fully connected layer and a normalization layer connected in sequence; the activation function of the activation layer is expressed as follows:

[0020] Relu(x)=max(0,x)

[0021] Here, x represents a vector in the self-attention matrix.

[0022] As a preferred technical solution, the character map code and the clue text code are fused to obtain a fused code vector. The specific steps include:

[0023] Defining m sets of learnable z vectors; the dimensions of the z vectors are the same as the dimensions of the character atlas encoding and the clue text encoding;

[0024] Using m groups of z vectors to perform attention mechanism learning on the character graph encoding and the clue text encoding respectively, and summing and normalizing the attention mechanism learning results in sequence to obtain m groups of fusion information;

[0025] Performing self-attention mechanism learning on the character graph encoding, and normalizing the self-attention mechanism learning result of the character graph encoding to obtain a first self-attention vector;

[0026] Performing self-attention mechanism learning on the clue text encoding, and normalizing the self-attention mechanism learning result of the clue text encoding to obtain a second self-attention vector;

[0027] Using m groups of fusion information to perform attention mechanism learning on the first self-attention vector, and summing and normalizing the attention mechanism learning results in sequence to obtain a first self-attention fusion vector;

[0028] Using m groups of fusion information to perform attention mechanism learning on the second self-attention vector, and summing and normalizing the attention mechanism learning results in sequence to obtain a second self-attention fusion vector;

[0029] The first self-attention fusion vector and the second self-attention fusion vector are summed and averaged to obtain a fused encoding vector.

[0030] As an optimal technical solution, the classification network includes a fully connected layer and a Softmax activation layer connected in sequence; the fused coding vector is transmitted to the fully connected layer for classification, and the classification result is transmitted to the Softmax activation layer for activation to obtain the alarm category probability.

[0031] As a preferred technical solution, after obtaining the character graph features and the clue text features, the method further includes: mapping the character graph features and the clue text features to the same dimension.

[0032] As a preferred technical solution, after obtaining the fused coding vector, the method further includes: training the classification network until the loss function of the classification network converges, thereby obtaining a trained classification network.

[0033] As a preferred technical solution, the loss function is a cross entropy loss function.

[0034] As a preferred technical solution, the expression of the cross entropy loss function is as follows:

[0035]

[0036] Among them, M is the total number of police categories, y c represents the probability of the true label of category c, p c represents the predicted probability of category c.

[0037] In a second aspect, the present invention further proposes a police situation analysis system based on cross-modal data, comprising:

[0038] The acquisition module is used to obtain character graph data and clue text data.

[0039] The extraction module is used to perform feature extraction on the character graph data and clue text data to obtain character graph features and clue text features.

[0040] The encoding module is used to encode the character graph features and the clue text features respectively to obtain the character graph code and the clue text code.

[0041] A fusion module is used to fuse the character graph code and the clue text code to obtain a fused code vector.

[0042] The classification module is used to input the fusion coding vector into a trained classification network for processing, and the classification network outputs the probability of the alarm category.

[0043] Compared with the prior art, the beneficial effect of the technical solution of the present invention is: the present invention processes two cross-modal data, namely, face maps and case clue texts, and encodes the character map data and clue text data respectively, and then fuses the character map codes and clue text codes of the two modalities into a fusion coding vector, and uses the obtained fusion coding vector representing the joint information to obtain the corresponding police situation type through a classification network, making full use of data from multiple modalities, avoiding the need to obtain a large amount of expert knowledge, and being able to improve the precision, accuracy and efficiency of police situation analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of a method for analyzing police situations based on cross-modal data according to an embodiment of the present application.

[0045] Figure 2 This is a flowchart of obtaining a fused coding vector according to an embodiment of the present application.

[0046] Figure 3 This is an architectural diagram of a police situation analysis system based on cross-modal data according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0048] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0049] Example 1

[0050] See also Figure 1 This embodiment proposes a police situation analysis method based on cross-modal data, including:

[0051] Get character graph data and clue text data.

[0052] Feature extraction is performed on the character graph data and clue text data to obtain character graph features and clue text features.

[0053] The character graph features and the clue text features are encoded respectively to obtain a character graph code and a clue text code.

[0054] The character graph encoding and the clue text encoding are fused to obtain a fused encoding vector.

[0055] The fused coding vector is input into a trained classification network for processing, and the classification network outputs the probability of the alarm category.

[0056] In a specific implementation, the classification network outputs the probabilities of criminal cases, civil cases, administrative cases, etc. Alternatively, the classification network outputs the probabilities of extremely urgent, moderately urgent, and ordinary police situations.

[0057] The police situation analysis method based on cross-modal data proposed in this embodiment processes two types of cross-modal data, namely facial maps and case clue texts, and encodes the character map data and clue text data respectively. Then, the character map codes and clue text codes of the two modalities are fused into a coding vector. The obtained fused coding vector representing the joint information is used to obtain the corresponding police situation type through a classification network. This makes full use of data from multiple modalities, avoids the need to obtain a large amount of expert knowledge, and can improve the precision, accuracy and efficiency of police situation analysis.

[0058] Example 2

[0059] This embodiment improves the police situation analysis method based on cross-modal data proposed in the first embodiment.

[0060] In this embodiment, after obtaining the character graph features and the clue text features, the character graph features and the clue text features are mapped to the same dimension.

[0061] In the specific implementation process, because other models such as CNN are used to extract features when obtaining character graph features and clue text features, other models are independent of this model and each feature needs to be mapped to the same dimension. Assume that the input character graph features are where a i represents the eigenvector, N a Represents the number of eigenvectors, whose dimension is d a Similarly, the input clue text features are defined as Among them, b i represents the eigenvector, N b Represents the number of eigenvectors, whose dimension is d bAfter the character graph features and clue text features pass through a layer of fully connected network respectively, each feature vector can be mapped to the same dimension d.

[0062] In this embodiment, the encoder in the standard Transformer model is used to encode the character graph features and the clue text features respectively to obtain the character graph encoding and the clue text encoding. The specific steps include:

[0063] The character graph features and the clue text features are linearly transformed respectively to obtain the query matrix Q, key matrix K and value matrix V corresponding to the character graph features and the clue text features.

[0064] The query matrix Q, key matrix K, and value matrix V are input into the multi-head attention module for self-attention mechanism learning. The self-attention matrices corresponding to the character graph features and clue text features are obtained respectively. The expressions are as follows:

[0065]

[0066] Among them, K T represents the transpose of the key matrix K, d K represents the dimension of the key matrix K.

[0067]

[0068] The self-attention matrices corresponding to the character graph features and clue text features are summed and normalized in turn, and the normalized results are input into the fully connected network for calculation to obtain the character graph encoding and clue text encoding.

[0069] In practice, the multi-head attention module simply splits each vector into different dimensions. Assuming a vector has a dimension of 256 and the number of heads is set to 8, each vector is split into eight 32-dimensional vectors, which are then divided into eight groups and subjected to the single-head attention mechanism's operations. In the self-attention mechanism, assuming the input is a person's graph features, Q, K, and V are all set to the person's graph features. That is, the three matrices Q, K, and V are equal, and each row of the matrix represents a feature graph vector. The Softmax function of the single-head attention mechanism is shown in Equation 2. Equation 1 is the Softmax definition of the multi-head attention mechanism. The Softmax in Equation 1 indicates that the Softmax operation in Equation 1 is performed on each row of the matrix.

[0070] In this embodiment, the fully connected network includes a fully connected layer, an activation layer, a fully connected layer, and a normalization layer connected in sequence; the activation function of the activation layer is expressed as follows:

[0071] Relu(x)=max(0,x)

[0072] Here, x represents a vector in the self-attention matrix.

[0073] like Figure 2 As shown, in this embodiment, the character map code and the clue text code are fused to obtain a fused code vector. The specific steps include:

[0074] Defining m sets of learnable z vectors; the dimensions of the z vectors are the same as the dimensions of the character atlas encoding and the clue text encoding;

[0075] Using m groups of z vectors to perform attention mechanism learning on the character graph encoding and the clue text encoding respectively, and summing and normalizing the attention mechanism learning results in sequence to obtain m groups of fusion information;

[0076] Performing self-attention mechanism learning on the character graph encoding, and normalizing the self-attention mechanism learning result of the character graph encoding to obtain a first self-attention vector;

[0077] Performing self-attention mechanism learning on the clue text encoding, and normalizing the self-attention mechanism learning result of the clue text encoding to obtain a second self-attention vector;

[0078] Using m groups of fusion information to perform attention mechanism learning on the first self-attention vector, and summing and normalizing the attention mechanism learning results in sequence to obtain a first self-attention fusion vector;

[0079] Using m groups of fusion information to perform attention mechanism learning on the second self-attention vector, and summing and normalizing the attention mechanism learning results in sequence to obtain a second self-attention fusion vector;

[0080] The first self-attention fusion vector and the second self-attention fusion vector are summed and averaged to obtain a fused encoding vector.

[0081] In this embodiment, the classification network includes a fully connected layer and a Softmax activation layer connected in sequence; the fused coding vector is transmitted to the fully connected layer for classification, and the classification result is transmitted to the Softmax activation layer for activation to obtain the alarm category probability.

[0082] In this embodiment, the classification network is trained until the loss function of the classification network converges to obtain a trained classification network. The loss function is a cross entropy loss function, which is expressed as follows:

[0083]

[0084] Among them, M is the total number of police categories, yc represents the probability of the true label of category c, p c represents the predicted probability of category c.

[0085] In the police situation analysis method proposed in this embodiment, both the character graph and the case clues are usually contextually relevant. The self-attention mechanism can obtain the contextual information of the corresponding character graph features and clue text features, thereby obtaining more accurate results.

[0086] Example 3

[0087] See Figure 3 This embodiment proposes a police situation analysis system based on cross-modal data, including:

[0088] The acquisition module is used to obtain character graph data and clue text data.

[0089] The extraction module is used to perform feature extraction on the character graph data and clue text data to obtain character graph features and clue text features.

[0090] The encoding module is used to encode the character graph features and the clue text features respectively to obtain the character graph code and the clue text code.

[0091] A fusion module is used to fuse the character graph code and the clue text code to obtain a fused code vector.

[0092] The classification module is used to input the fusion coding vector into a trained classification network for processing, and the classification network outputs the probability of the alarm category.

[0093] The police situation analysis system based on cross-modal data proposed in this embodiment processes two types of cross-modal data, namely facial maps and case clue texts, and encodes the character map data and clue text data respectively. Then, the character map codes and clue text codes of the two modalities are fused into a coding vector. The obtained fused coding vector representing the joint information is used to obtain the corresponding police situation type through a classification network, which makes full use of data from multiple modalities, avoids the need to obtain a large amount of expert knowledge, and can improve the precision, accuracy and efficiency of police situation analysis.

[0094] The terms used in the drawings to describe positional relationships are for illustrative purposes only and should not be construed as limiting this patent;

[0095] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A police situation analysis method based on cross-modal data, characterized in that: include: Obtain character graph data and clue text data; Performing feature extraction on the character graph data and clue text data to obtain character graph features and clue text features; The encoder in the standard Transformer model is used to encode the character graph features and the clue text features respectively to obtain the character graph encoding and the clue text encoding, including: Perform linear transformation on the character graph features and the clue text features respectively to obtain the query matrix corresponding to the character graph features and the clue text features Q , key matrix K Sum Matrix V ; The query matrix Q , key matrix K Sum Matrix V Input the multi-head attention module to learn the self-attention mechanism, and obtain the self-attention matrices corresponding to the character graph features and clue text features respectively. The expressions are as follows: in, Represents the bond matrix K The transpose of Represents the bond matrix K Dimensions; The self-attention matrices corresponding to the character graph features and clue text features are summed and normalized in sequence, and the normalized results are input into the fully connected network for calculation to obtain the character graph encoding and clue text encoding; The character graph code and the clue text code are fused to obtain a fused code vector, including: definition m Group learnable z vector; z The dimension of the vector is the same as the dimension of the character map code and the clue text code; use m Group z The vector performs attention mechanism learning on the character map encoding and the clue text encoding respectively, and sums and normalizes the attention mechanism learning results in turn to obtain m Group fusion information; Performing self-attention mechanism learning on the character graph encoding, and normalizing the self-attention mechanism learning result of the character graph encoding to obtain a first self-attention vector; Performing self-attention mechanism learning on the clue text encoding, and normalizing the self-attention mechanism learning result of the clue text encoding to obtain a second self-attention vector; use m The group fusion information performs attention mechanism learning on the first self-attention vector, and sums and normalizes the attention mechanism learning results in sequence to obtain a first self-attention fusion vector; use m The group fusion information performs attention mechanism learning on the second self-attention vector, and sums and normalizes the attention mechanism learning results in sequence to obtain a second self-attention fusion vector; Summing the first self-attention fusion vector and the second self-attention fusion vector and taking the average value to obtain a fusion encoding vector; The fused coding vector is input into a trained classification network for processing, and the classification network outputs the probability of the alarm category.

2. The cross-modal data-based police situation analysis method according to claim 1, characterized in that: The fully connected network includes a fully connected layer, an activation layer, a fully connected layer, and a normalization layer connected in sequence; the activation function of the activation layer is expressed as follows: in, Represents a vector in the self-attention matrix.

3. The cross-modal data-based police situation analysis method according to claim 1, characterized in that: The classification network includes a fully connected layer and Softmax Activation layer; the fusion coding vector is transmitted to the fully connected layer for classification, and the classification result is transmitted to the Softmax The activation layer is activated to obtain the probability of the alarm category.

4. The method for analyzing police situations based on cross-modal data according to claim 1, characterized in that: After obtaining the character graph features and the clue text features, the method further includes: mapping the character graph features and the clue text features to the same dimension.

5. The method for analyzing police situations based on cross-modal data according to claim 1, characterized in that: After obtaining the fused coding vector, the method further includes: training the classification network until the loss function of the classification network converges, thereby obtaining a trained classification network.

6. The method for analyzing police situations based on cross-modal data according to claim 5, characterized in that: The loss function is a cross entropy loss function.

7. The method for analyzing police situations based on cross-modal data according to claim 6, characterized in that: The expression of the cross entropy loss function is as follows: in, M is the total number of police incident categories, y c Representation category c The probability of the true label, Representation category c The predicted probability of .

8. A police situation analysis system based on cross-modal data, applied to the police situation analysis method based on cross-modal data according to any one of claims 1 to 7, characterized in that: include: The acquisition module is used to obtain character map data and clue text data; An extraction module, configured to extract features from the character graph data and clue text data to obtain character graph features and clue text features; An encoding module, configured to encode the character graph features and the clue text features respectively to obtain a character graph code and a clue text code; A fusion module, configured to fuse the character graph code and the clue text code to obtain a fused code vector; The classification module is used to input the fusion coding vector into a trained classification network for processing, and the classification network outputs the probability of the alarm category.

Images