Police file and image association reasoning method of multi-modal large model

Abstract

The invention discloses a multi-mode large model police file and image association reasoning method, and belongs to the technical field of artificial intelligence and police information processing. The method comprises the following steps: preprocessing police affair texts and images; respectively extracting word-level and sentence-level features and local and global visual features by using a text encoder and an image encoder; realizing alignment and joint representation through a cross-modal fusion layer and an attention mechanism; generating a preliminary reasoning result by adopting a task output head; a heterogeneous evidence graph is constructed, and priori knowledge is injected in combination with the knowledge graph; and finally, joint reasoning is carried out through the graph neural network, and a cross-modal reasoning result and a complete evidence chain are output. According to the method, deep correlation analysis of police affair texts and images is realized, and the accuracy and interpretability of case research and judgment are improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and police information processing technology, specifically relating to a multimodal large-scale model of police documents and image association reasoning method. Background Technology

[0002] With the development of informatization and intelligentization, a large amount of multi-source heterogeneous data is generated in police work, including police documents, case records, surveillance images, and on-site evidence photos. How to efficiently process and intelligently analyze this multimodal data to uncover potential case elements and evidence chains has become an important requirement for improving police combat capabilities and case analysis levels.

[0003] In existing technologies, natural language processing models are typically used to structure police documents, or convolutional neural networks are used to extract features from images. These methods can achieve good representation capabilities within their respective modalities, but in cross-modal scenarios, they often rely on simple feature concatenation or similarity calculations for association analysis, lacking an effective cross-modal alignment mechanism.

[0004] However, the above methods still have several shortcomings: First, the lack of deep integration between text and image modalities makes it difficult to construct a unified semantic space, resulting in insufficient cross-modal association capabilities; second, existing methods fail to fully integrate police knowledge graphs, lack the utilization of domain prior knowledge, and have insufficient interpretability of the reasoning process; third, the organization of evidence information is fragmented, lacking a structured framework that can integrate multi-source evidence and support joint reasoning. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, the present invention aims to provide a multimodal large-scale model for police documents and image association reasoning, in order to solve the problems of lack of deep integration between text and image modalities, insufficient cross-modal alignment capabilities, failure to fully integrate police knowledge graphs, incomplete evidence chain construction, and insufficient interpretability of reasoning in the existing technologies, thereby achieving efficient association analysis and joint reasoning of police documents and image data.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for inferring the association between police documents and images in a multimodal large-scale model, comprising the following steps:

[0008] Step S1: Collect text and image files related to police work and perform data preprocessing;

[0009] Step S2: Encode the preprocessed text and image files using a multimodal large model;

[0010] Step S3: Align text files and image files across modalities and construct a heterogeneous evidence map;

[0011] Step S4: Perform joint reasoning and output the reasoning results and evidence chain.

[0012] Furthermore, the data preprocessing involves performing layout analysis, text correction, and structured extraction on police documents, quality assessment, noise reduction, and key area extraction on police images, and desensitization of sensitive personal information.

[0013] Furthermore, the multimodal large model includes: an input layer, a processing layer, an encoding layer, a cross-modal fusion layer, a policing adaptive reasoning layer, and an output layer;

[0014] The input layer is used to receive text files and image files;

[0015] The processing layer is used to process the input data;

[0016] The encoding layer includes a text encoder and an image encoder. The text encoder uses a pre-trained model based on the BERT architecture and extracts word-level and sentence-level feature representations from the text file through a 12-layer Transformer structure. The image encoder uses ResNet and utilizes a deep residual network structure to extract visual features from the image, including local region features and global semantic features.

[0017] The cross-modal fusion layer is used to perform cross-modal alignment and fusion of text files and image files;

[0018] The police adaptive reasoning layer is used to receive the output of the cross-modal fusion layer, generate preliminary reasoning results through the police task output header, and construct a heterogeneous evidence graph based on the preliminary reasoning results; the heterogeneous evidence graph is enhanced by injecting domain prior knowledge through a police knowledge graph, and then joint reasoning is performed based on the enhanced heterogeneous evidence graph.

[0019] The output layer is used to output the joint inference results.

[0020] Furthermore, the BERT architecture includes an input layer, an encoding layer, and an output layer; the encoding layer adopts a 12-layer Transformer encoder stacked structure, each layer containing: a multi-head attention mechanism, a feedforward neural network, residual connections, and layer normalization.

[0021] Furthermore, the word-level feature representation maps the preprocessed police text sequence into a set of related word vectors. The specific steps are as follows:

[0022] The input text file was processed to obtain the following text sequence:

[0023]

[0024] Where L represents the length of the text sequence, This represents each word in the text sequence;

[0025]

[0026] in, For word embedding matrix, For position matrix, For paragraph matrix, The paragraph markers indicated by the words; The input is processed by the text encoder to obtain the final word-level representation matrix. :

[0027]

[0028] in, express Corresponding word-level feature representation;

[0029] The sentence-level feature representation is obtained by using an aggregation mechanism to obtain a global semantic vector based on the word-level feature representation. The aggregation mechanism uses an attention-weighted approach to sum the word-level representations, as shown in the formula:

[0030]

[0031]

[0032] Where s is the sentence-level feature representation, It is a queryable vector used to measure the importance of words. The projection matrix projects the word vectors onto the attention space. It is the importance weight of the i-th word. It is a non-linear activation function. This indicates the matrix transpose.

[0033] The ResNet further includes an input layer, a residual stacking layer, a global average pooling layer, and a fully connected layer; wherein, the input layer consists of a 7×7 convolutional layer and a 3×3 max pooling layer; the residual stacking layer is composed of four stages of residual modules in sequence.

[0034] Furthermore, the aforementioned local region feature representation involves processing the input image I through layers of ResNet convolution, pooling, and residual concatenation to obtain a feature map F, which is then flattened into a set of region features.

[0035]

[0036] in, This represents the visual features of the nth local region in the image. This represents the total number of local regions into which the image is divided;

[0037] The global language feature representation described above is based on local region features, and weighted aggregation of local region features is performed through an attention mechanism:

[0038]

[0039]

[0040] in, It is a global semantic feature representation of the image. It is a queryable vector used to measure the importance of local features, and U is a projection matrix that projects the local features onto the attention space. It is the attention weight of the nth local region feature.

[0041] Furthermore, the cross-modal alignment and fusion include:

[0042] Text modality includes word-level and sentence-level feature representations of text files, while image modality includes local region features and global semantic features of image files.

[0043] The text modality and the image modality are mapped into a unified cross-modal semantic space, and cross-modal alignment and fusion are achieved through an attention mechanism;

[0044] No. The word-level features are represented as follows: Sentence-level features are represented as , No. The characteristics of each local region are The global semantic features are The text modality and the image modality are mapped to a cross-modal semantic space through linear projection, as shown in the following formula:

[0045]

[0046]

[0047] in, , , and For learnable projection matrices, , , and For bias terms; Represents global features of the text. Represents global features of an image; Represents word-level mapping features; Indicates the characteristics of the region mapping;

[0048] Then, alignment is achieved through an attention mechanism, as shown in the following formula:

[0049]

[0050] in, This represents the relationship between the i-th word in the word-level mapping feature and the i-th region mapping feature. Weights between regions;

[0051] The fusion representation is calculated using the following formula:

[0052]

[0053] in, This represents the visual context vector of the i-th word. Represents the visual context matrix;

[0054] Then, the global features of the text are concatenated with the global features of the image, as shown in the following formula:

[0055]

[0056] in, This represents the concatenated global union representation. It is a fusion function;

[0057] The final cross-modal joint representation is obtained. This is used to generate preliminary reasoning results in the police adaptive reasoning layer, and to construct a heterogeneous evidence graph and perform joint reasoning.

[0058] Furthermore, the task output head is used to receive the joint representation after cross-modal fusion and generate corresponding inference results according to different police task types; the task output head includes a classification subhead, a matching subhead, and a generation subhead;

[0059] The classifier subheader is used to classify target objects, and outputs a class probability distribution based on a fully connected layer and a softmax function. The specific form of the probability distribution is a vector. Where C represents the total number of case categories distinguished by the model. This represents the predicted probability that the input belongs to c categories, and satisfies the following conditions: The category corresponding to the maximum value in the probability distribution. As a preliminary case type classification result;

[0060] The expression for the fully connected layer is:

[0061]

[0062] in, These are unnormalized logits (each category corresponds to a score). It is the input feature vector. and These are learnable parameters, and then the logits are transformed into an unclassified probability distribution using the softmax function:

[0063]

[0064] in, Predicting probabilities It is the logit corresponding to the i-th category. It is the logit for all classes.

[0065] The matching subheader is used to achieve correlation matching between police text and images, and to calculate the cosine similarity score between the global features of the text and the global features of the image, as shown in the following formula:

[0066]

[0067] in, This represents the L2 magnitude of the vector. The formula for calculating similarity is given; the similarity score serves as a preliminary matching result for the correlation between images and text.

[0068] The generated subheader uses a Transformer-based decoder to generate an initial inference text sequence in a regression manner; during the decoding process, a cross-modal joint representation is queried through a cross-attention mechanism. Proceed to output word sequences As a preliminary result of reasoning, among which Indicates the length of the word sequence;

[0069] The preliminary classification results output by the classification subheader, the similarity scores output by the matching subheader, and the preliminary reasoning text output by the generated subheader are used to construct a heterogeneous evidence graph.

[0070] Furthermore, the heterogeneous evidence graph is derived from the cross-modal joint representation based on the preliminary classification result of the classification subhead output, the similarity score of the matching subhead output, and the preliminary inference text of the generated subhead output. The heterogeneous evidence graph is constructed by selectively extracting information from a police knowledge graph; the heterogeneous evidence graph comprises two parts: a set of nodes and a set of edges; the heterogeneous evidence graph is represented as follows:

[0071]

[0072] in, For a set of nodes, The set of edges; the set of nodes The node set is composed of evidence information of different modalities, including text nodes ( Image nodes ) and knowledge nodes ( ); edge set The edge set is used to characterize the semantic and evidence relationships between nodes, including same-modal edges, cross-modal edges, and knowledge edges.

[0073] Furthermore, the joint reasoning takes a heterogeneous evidence graph as input, uses a graph neural network to realize information interaction and evidence fusion between nodes, and finally outputs a case reasoning text; the joint reasoning includes a reasoning input layer, a reasoning propagation layer, and a reasoning output layer;

[0074] The inference input layer receives a heterogeneous evidence graph composed of text nodes, image nodes, and knowledge nodes. ,in, For a set of nodes, The set is an edge set; the initial feature representations of each node are derived from the following sources:

[0075] Text node features originate from word-level visual context vectors Image node features originate from image region features. Knowledge node features are derived from the embedding of knowledge entities retrieved from the knowledge graph. ;

[0076] The inference propagation layer, through a graph neural network architecture, transmits and updates features among text nodes, image nodes, and knowledge nodes in the heterogeneous evidence graph. Its update formula is as follows:

[0077]

[0078] in, This represents the feature representation of node v at layer l. Denotes the set of neighbors of node v. For edge weights, and For learnable parameters, It is a non-linear activation function;

[0079] The inference output layer receives the passed... The enhanced node representation is obtained after layered inference propagation. A global case reasoning representation vector is generated through aggregation functions. The calculation formula is:

[0080]

[0081] in, Represents aggregate functions, Indicates the total number of inference propagation layers; Used for joint inference results in the output layer.

[0082] Furthermore, the joint reasoning result is a global case reasoning representation vector. The joint result is output through three task output heads; the three task output heads include: classification output, matching output, and generation output.

[0083] The classification output is... Input a fully connected layer and a softmax function, output case type classification labels;

[0084] The matching output will The image-text matching score output is converted into a probability value using the sigmoid function.

[0085] The generated output will As the initial state input decoder, autoregression generates structured case reasoning chain text.

[0086] Compared with the prior art, the advantages of the present invention are as follows:

[0087] (1) This invention extracts features from text and images separately, and achieves deep alignment and joint representation through cross-modal fusion layer and attention mechanism, which overcomes the problem of traditional methods only performing simple splicing or similarity calculation, and improves the ability of association reasoning between text and images.

[0088] (2) The present invention introduces police knowledge graph into heterogeneous evidence graph, injects domain prior knowledge, and realizes the enhancement, association and correction of evidence information, thereby improving the accuracy and interpretability of case reasoning results.

[0089] (3) The present invention organizes multi-source evidence information in the form of nodes and edges, and uses graph neural networks for propagation and fusion, which effectively solves the shortcomings of fragmented evidence information and lack of a systematic reasoning framework, thereby supporting the generation of complete evidence chains and joint reasoning in cases. Attached Figure Description

[0090] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0091] Figure 1 This is a flowchart of the method of the present invention.

[0092] Figure 2This is a structural diagram of the multimodal large model in the method of the present invention. Detailed Implementation

[0093] To facilitate understanding by those skilled in the art, the present invention will be further described below in conjunction with embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0094] like Figure 1 As shown, the present invention provides a method for multimodal large-scale model-based reasoning of police documents and images, the steps of which are as follows:

[0095] Step S1: Collect text and image files related to police work and perform data preprocessing;

[0096] Step S2: Encode the preprocessed text and image files using a multimodal large model;

[0097] Step S3: Align text files and image files across modalities and construct a heterogeneous evidence map;

[0098] Step S4: Perform joint reasoning and output the reasoning results and evidence chain analysis.

[0099] Police documents and image files are collected and preprocessed. This preprocessing involves layout analysis of the police documents, requiring the segmentation of text and table areas. OCR technology is then used to extract the text, identifying typos and noise. Finally, the text is mapped to case elements (triplets), such as suspect-crime tool-location.

[0100] The sharpness, brightness, and noise of the police images were evaluated, with sharpness expressed as Laplace variance:

[0101]

[0102] in, It is the input image set; the brightness is calculated using the pixel average.

[0103] The noise reduction process uses convolutional filtering to remove noise.

[0104]

[0105] in, It is a convolution kernel; This represents the denoised image. Next, key region extraction is performed on the image, locating candidate key regions using YOLOv5. Finally, the police documents and images undergo desensitization processing. For text, the entity set S, including names, ID numbers, and addresses, is identified using the following desensitization function:

[0106]

[0107] For images, detect face regions. Blur processing

[0108]

[0109] in, Original image in coordinates Pixel value at that location, Blurred image in coordinates Pixel value at that location, This is the detected face region, with the coordinates of its upper left corner being... ,width ,high , express The surrounding neighborhood pixel set, This indicates the number of pixels in the neighborhood. By averaging the pixels in the face region, the original sharp details are destroyed to avoid leaking personal privacy; however, general regional information (such as location and size) is still preserved, ensuring the contextual integrity of the case analysis.

[0110] like Figure 2 As shown, the multimodal large model includes: an input layer, a processing layer, an encoding layer, a cross-modal fusion layer, a policing adaptive reasoning layer, and an output layer;

[0111] The input layer is used to receive text files and image files;

[0112] The processing layer is used to process the input data;

[0113] The encoding layer includes a text encoder and an image encoder. The text encoder uses a pre-trained model based on the BERT architecture and extracts word-level and sentence-level feature representations from the text file through a 12-layer Transformer structure. The image encoder uses ResNet and utilizes a deep residual network structure to extract visual features from the image, including local region features and global semantic features.

[0114] The cross-modal fusion layer is used to perform cross-modal alignment and fusion of text files and image files;

[0115] The police adaptive reasoning layer is used to receive the output of the cross-modal fusion layer, generate preliminary reasoning results through the police task output header, and construct a heterogeneous evidence graph based on the preliminary reasoning results; the heterogeneous evidence graph is enhanced by injecting domain prior knowledge through a police knowledge graph, and then joint reasoning is performed based on the enhanced heterogeneous evidence graph.

[0116] The input police text data uses a pre-trained model based on the BERT architecture, which includes an input layer, an encoding layer, and an output layer. The encoding layer uses a 12-layer Transformer encoder stacked structure, with each layer containing: a multi-head attention mechanism, a feedforward neural network, residual connections, and layer normalization.

[0117] First, the input raw police texts (such as case descriptions, interrogation records, police reports, etc.) undergo standardization preprocessing. Then, a domain-adaptive word segmentation tool is used to segment the text into continuous text sequences, represented as:

[0118]

[0119] Where L represents the length of the text sequence, This represents each word in the text sequence;

[0120] To fully represent the rich information of each word, the BERT model assigns a unique identifier to each word in the sequence. An embedding vector that integrates three types of information was constructed:

[0121]

[0122] in, For word embedding matrix, For position matrix, For paragraph matrix, Paragraph markers indicating words;

[0123] Stack the fused vectors of all words in order to form the initial input matrix of the model:

[0124]

[0125] The coding layer is a sequence of input vectors Deep, bidirectional contextual semantic encoding is performed. This model adopts a stacked structure of 12 Transformer encoders, with each layer containing two core sub-modules, supplemented by residual connections and layer normalization to ensure training stability and the performance of the deep model.

[0126] After layers of processing by the depth encoder, the initial input matrix... It was transformed into a word-level representation matrix containing rich contextual semantic information. :

[0127]

[0128] in, express Corresponding word-level feature representation

[0129] In obtaining a word-level representation matrix containing rich contextual information Subsequently, to complete downstream tasks requiring holistic semantic understanding, such as text classification and sentiment analysis, it is necessary to aggregate the variable-length word vector sequence into a fixed-length vector representation that can represent the global semantics of the entire sentence. This study employs an attention mechanism as the aggregation mechanism, which can dynamically and discriminatively fuse information from all words, rather than simply averaging or directly using [CLS] labels. This approach allows the model to automatically focus on information segments within the sentence that are more critical to the current task, and is particularly effective for police texts with prominent "case keywords."

[0130] The generation of the sentence-level feature representation is a two-stage process: first, the attention weight of each word in the sequence is calculated, and then the word-level representations are weighted and summed based on these weights. Its mathematical formalization is as follows:

[0131] First, each... Importance score. The network consists of a linear transformation layer and a nonlinear activation function:

[0132]

[0133] in, It is a trainable weight matrix. It is a non-linear activation function. It is a queryable vector used to measure the importance of words.

[0134] Then, the importance score of all words was obtained using the Softmax function. After normalization, we obtain the final attention weight distribution with a sum of 1:

[0135]

[0136] in, The projection matrix projects the word vectors onto the attention space. It is the importance weight of the i-th word. It is a non-linear activation function. This indicates the matrix transpose.

[0137] The final sentence-level feature representation s is derived from all word-level feature representations. With its attention weight This is obtained by linear weighted sum of the coefficients:

[0138]

[0139] The ResNet includes an input layer, a residual stacking layer, a global average pooling layer, and a fully connected layer; the input layer consists of a 7×7 convolutional layer and a 3×3 max pooling layer; the residual stacking layer consists of four stages of residual modules in sequence.

[0140] The input is the original police image. The size is (Height, width, and RGB three channels).

[0141] First, the image Standardized preprocessing, including size scaling and pixel value normalization, is performed to meet the input requirements of the pre-trained model. The preprocessed image is then fed into a ResNet network. After forward propagation, a 3D feature map F is output from the last convolutional layer of the network. This feature map can be viewed as an abstract representation of the image after it has been meshed. Each spatial location on the feature map F... Each corresponds to a local receptive field region in the original input image, which can be considered as a set of local region features:

[0142]

[0143] in, This represents the visual features of the nth local region in the image. This represents the total number of local regions into which the image is divided;

[0144] Directly using the set of all local features can introduce a large amount of redundant or irrelevant background information. To obtain a compact and semantically rich overall image representation, an attention mechanism is introduced to automatically filter and aggregate these local features. This mechanism mimics the way humans observe images, focusing not on every part of the image equally, but on the salient regions most relevant to the task. Its mathematical form is as follows:

[0145] First, a parameterized neural network is used to define the features of each local region. Calculate an importance score. The network projects each feature vector into a common attention subspace and evaluates its relevance to the task:

[0146]

[0147] in, The projection matrix projects local features into the attention space. It is a queryable vector used to measure the importance of local features.

[0148] Then, the Softmax function is used to score the importance of all regions. Normalization is performed to transform it into a probability distribution that sums to 1, and then weighted aggregation is applied to the local region features:

[0149]

[0150] in, It is the first Attention weights for local region features.

[0151] Finally, the global semantic feature representation of the image Through all local region features With its attention weight This is obtained by linear weighted sum of the coefficients:

[0152]

[0153] in, It is a global semantic feature representation of an image.

[0154] After obtaining the word-level representation matrices of the text modalities respectively After combining sentence-level feature representations s with local region feature maps F and global semantic features g of the image modality, a hierarchical cross-modal alignment and fusion module is used, and an attention mechanism is employed to achieve refined mutual information retrieval and fusion.

[0155] Police data is inherently multimodal. A case description, such as "a man wearing a black shirt is driving a white sedan," often contains complementary and mutually verifiable information with a related surveillance image. Simply concatenating or averaging features from the two modalities is insufficient to capture the complex, fine-grained correspondences between them. For example, "black shirt" corresponds to a specific area in the image, while "white sedan" corresponds to another area. Therefore, the goal of this module is:

[0156] By establishing fine-grained semantic correspondences between text words and image regions, the aligned information can be effectively integrated to form a joint representation that can simultaneously contain textual and image information.

[0157] Since text and image features originate from different networks (BERT and ResNet), their original feature spaces differ in distribution and scale. Directly calculating their similarity is suboptimal. Therefore, the first step is to project all features into a completely new, common semantic space through a linear transformation. The word-level representation matrix is ​​described above. Sentence-level feature representation Local region feature map F and global semantic features of the image Different modalities are mapped to a cross-modal semantic space through linear projection:

[0158]

[0159]

[0160] in, , , and For learnable projection matrices, , , and For bias terms;

[0161] To achieve fine-grained alignment between text words and image regions, a soft attention mechanism is employed, which calculates the correlation between each word in the text and all regions in the image.

[0162] Features of the i-th word in the text sequence Calculate its features with respect to the i-th region in the image. The relevance score uses the vector dot product as the similarity because it is computationally efficient and performs well in high-dimensional spaces. The mathematical expression is:

[0163]

[0164] The higher the score, the more relevant the semantics of word i are to the visual content of image region n.

[0165] To obtain the attention distribution of the i-th word across all image regions, the softmax function is used to normalize the above scores:

[0166]

[0167] in, and It precisely quantifies "the extent to which the model should focus on the nth region of the image in order to understand the i-th word in the text".

[0168] Based on attention weights, a corresponding image context vector is calculated for each word in the text. . It is a weighted sum of features from all regions of the image, and its weights are determined by the relevance of the word to each region:

[0169]

[0170] After generating the corresponding image context vector for each word in the text sequence, a visual context matrix representation is obtained:

[0171]

[0172] in, Each line integrates the semantics of a word in the text and its most relevant visual information in the image, achieving fine-grained cross-modal fusion.

[0173] Building upon fine-grained alignment, global semantic information from text and images is fused to capture overall intermodal relationships. The projected global text vector... and image global vector The fusion is performed using a fusion strategy that involves splicing followed by nonlinear transformation. :

[0174]

[0175] in, It is a weight matrix. It is a bias term. It is an activation function, which ultimately yields... It is a joint representation vector that integrates global information from both modes.

[0176] The final output of the cross-modal alignment and fusion module is a cross-modal joint representation. Z is a visual context matrix table that preserves the structural information of the text and injects relevant visual evidence; It is a coarse-grained global joint representation that contains the overall semantic relationship between text and images, and is used to construct evidence graphs and joint reasoning in subsequent police adaptive reasoning layers.

[0177] In obtaining cross-modal joint fine-grained alignment information and global semantics Then, an adaptive multi-subheader output module is used. This module contains three independent subheaders: a classification subheader, a matching subheader, and a generation subheader.

[0178] The classification subheader is primarily used for discrimination tasks, with the goal of mapping the joint representation of the input to a discrete set of class labels. Examples include case type classification, risk level assessment, and identification of suspects. This subheader typically consists of one or more fully connected layers, terminated by a softmax classifier.

[0179] The classifier subheader is used to classify target objects. Based on a fully connected layer and a softmax function, it outputs a probability distribution of categories, which is specifically a vector. Where C represents the total number of case categories distinguished by the model. This represents the predicted probability that the input belongs to c categories, and satisfies the following conditions: The category corresponding to the maximum value in the probability distribution. As a preliminary case type classification result;

[0180] The expression for the fully connected layer is:

[0181]

[0182] in, These are unnormalized logits (each category corresponds to a score). It is the input feature vector. and These are learnable parameters, and then the logits are transformed into an unclassified probability distribution using the softmax function:

[0183]

[0184] in, Predicting probabilities It is the logit corresponding to the i-th category. It is the logit for all classes.

[0185] Matching subheaders are primarily used for measurement tasks, aiming to calculate the relevance or similarity score between text descriptions and image content. In law enforcement, this includes text-based image retrieval and verifying text-image matching. The core of this subheader is defining a robust similarity metric function.

[0186] The matching subheader is used to achieve correlation matching between police text and images, and calculates the cosine similarity score between the global features of the text and the global features of the image. The similarity score serves as the preliminary matching result for the image-text correlation, as shown in the following formula:

[0187]

[0188] in, This represents the L2 magnitude of the vector. The similarity calculation formula ensures that the output score falls within the range of [-1, 1], effectively avoiding the influence of vector magnitude on the similarity calculation. The closer the score is to 1, the more relevant the text-image pair is; the closer it is to -1, the less relevant it is.

[0189] The generator subheading is primarily used for generative tasks, aiming to autonomously generate coherent text sequences and preliminary inference results based on multimodal input information. Examples include automatically generating case reasoning chains, producing case summary reports from text and image inputs, and answering natural language questions about the situation at a crime scene. The generator subheading employs a Transformer-based decoder to generate preliminary inference text sequences using a regression approach; during decoding, a cross-attention mechanism is used to query the cross-modal joint representation. Proceed to output word sequences As a preliminary result of reasoning, among which Indicates the length of the word sequence;

[0190] The decoding process begins with a special start marker. <sos>The decoder operates in an autoregressive manner, generating one word at each step. At time t, the decoder receives the word embeddings from the previous step and its own hidden state from the previous step. The currently generated sequence is processed using a masked self-attention mechanism. Then, the previously fused multimodal information Z is queried using an encoder-decoder cross-attention mechanism. After passing through the feedforward network, a probability distribution across the entire vocabulary is output. The word with the highest probability is selected as the output of the current time step and used as the input for the next step. This process is repeated until an end marker is generated. <eos>This ultimately yields a complete word sequence. This serves as a preliminary result of reasoning.

[0191] Based on the preliminary classification results output by the classification subheader, the similarity score output by the matching subheader, and the preliminary inference text output by the generated subheader, a heterogeneous evidence graph is constructed. This graph structure not only integrates multi-source evidence but also introduces a police knowledge graph as a domain prior knowledge source to enhance the evidence information. This heterogeneous evidence graph can be represented as follows:

[0192]

[0193] in, It is a set of nodes, representing various types of evidence and knowledge entities in the reasoning process; The set of edges defines the semantic and evidentiary relationships between these entities;

[0194] Node set It is a heterogeneous set, the set required to generate the reasoning chain:

[0195]

[0196] Among them, text node features Source word-level visual context vector The core words in the text that represent key entities, actions, and attributes, such as "man in black," "robbery," and "escape," are corresponding to... Instantiate nodes in the graph. Image node features. Source image region features Each image node in the diagram represents a visual entity, such as the facial region of a suspect captured in surveillance footage, or the features of an unlicensed motorcycle. It provides visual verification for textual descriptions. Knowledge Node Features Knowledge entity embeddings retrieved from knowledge graphs The system extracts entities, concepts, and rules related to the current case as nodes from a pre-built police knowledge graph. For example, when a "motorcycle" image node exists in the graph, the system can automatically retrieve related nodes from the knowledge graph. For example, "unlicensed motorcycle" → (drive-by robbery), "drive-by robbery" → (two-person crime, escape direction: alley).

[0197] For knowledge nodes The core functions of a knowledge graph include: **Enhancement:** Imbuing ambiguous evidence with clear meaning. For example, an image node of a "long, black object" becomes semantically enhanced by associating it with the knowledge point "weapon: controlled knife." **Association:** Connecting isolated pieces of evidence. Relationships in the knowledge graph, such as is-a, part-of, and used-for, act as edges, linking text, image nodes, and knowledge nodes. For example, associating "shoe print at the scene" with the knowledge node "brand: a certain brand of sports shoes," and further associating it with "this brand of shoes → popular among: young people," provides support for the inference that the suspect might be a young person. **Correction:** Preventing the generation of conclusions that contradict common sense. For example, if the encoder initially tends to generate a hit-and-run conclusion, but there is an incompatible relationship between the knowledge node "crime location: narrow alley" and the "car" node.

[0198] edge set It defines the path of information flow between evidence and knowledge, forming the framework of logical reasoning:

[0199]

[0200] Among them, knowledge edge It serves as a bridge connecting evidence and knowledge. These edges, defined by relations in the knowledge graph, are the primary channels for injecting prior knowledge. The construction method involves establishing connections between text / image nodes and corresponding nodes in the knowledge graph through entity linking and relation matching techniques.

[0201] A heterogeneous evidence graph integrating multimodal evidence and domain knowledge enhancement was constructed. Next, a joint reasoning module will be implemented to simulate the thought process of investigators. By allowing information to iteratively propagate and interact along the relational edges in the evidence graph, mutual verification, contradiction resolution, and logical chain splicing between pieces of evidence will be achieved, ultimately generating a reliable case reasoning conclusion. This module uses a graph neural network as its core engine and is an end-to-end learnable reasoning system containing a three-layer structure: a reasoning input layer, a reasoning propagation layer, and a reasoning output layer.

[0202] The inference input layer is responsible for preparing initial feature representations for each node in the enhanced heterogeneous evidence graph. Since the nodes originate from three heterogeneous sources—text, images, and knowledge—and have different feature distributions, this layer first maps them into an aligned, unified feature space through a lightweight projection network, laying the foundation for subsequent information interaction computation.

[0203] Text node initialization: For text nodes Its initial features are directly derived from the output of the previous cross-modal alignment module, i.e., the corresponding word-level mapping features. For nodes representing global text semantics, then use... , is represented as:

[0204]

[0205] Image node initialization: For image nodes Its initial features are projected, text-highly correlated region mapping features. or global features of the image , is represented as:

[0206]

[0207] Knowledge node initialization: For image nodes Knowledge entity embedding Derived from pre-training, these embeddings have already captured the semantics between entities and relations, represented as:

[0208]

[0209] The reasoning propagation layer is the core of joint reasoning. It uses a multi-layer graph neural network to perform message passing on the evidence graph, simulating the mutual influence between evidence and the gradual progression of the reasoning process. Each layer (or each round of reasoning) allows nodes to aggregate information from their neighbors and update their own representations, thereby capturing broader context and more complex logical relationships. Its node update rules follow an improved graph convolution operation, specifically formalized as follows:

[0210]

[0211] in, This represents the feature representation of node v at layer l. Denotes the set of neighbors of node v. For edge weights, and For learnable parameters, It is a non-linear activation function;

[0212] pass Through iterative propagation of layers, the final representation of each node... They are no longer their original information, but rather a fusion of their original information. An enhanced representation rich in contextual semantics that incorporates all relevant evidence and knowledge within the jump-neighbor range. For example, after multiple layers of propagation, the representation of a "suspect" node may integrate multi-source information such as "eyewitness descriptions," "surveillance screenshots," and "prior criminal records," forming a comprehensive representation.

[0213] After completing multiple rounds of information propagation, the inference output layer is responsible for integrating all the enhanced node representations in the graph to generate a global case inference representation vector. .

[0214] Node representation enhancement: after Layered reasoning propagation yields the final state of each node. .

[0215] Global aggregation: using an aggregate function Compress the information from all nodes into a single global case reasoning representation vector. This vector represents the overall reasoning state of the entire case. This invention employs attention-weighted pooling, calculated as follows:

[0216]

[0217] Will The joint inference result used for the output layer is output through a three-task output head; the three-task output head includes: classification output, matching output, and generation output.

[0218] The classification output is used for case characterization; it will The input is fed into a fully connected layer, which projects the dimensions of the vector to the number of case categories. The logits scores are obtained. These scores are then converted into a probability distribution using the softmax function. Finally, the category with the highest probability is selected as the output. The calculation formula is as follows:

[0219]

[0220]

[0221]

[0222] in, and For the learnable parameters of the classification layer, This is the final case type classification label output. For example, the system inputs text describing "the suspect pried open the window with tools" and a "close-up" image of the window. After the aforementioned layers of reasoning, the output layer ultimately generates a probability distribution. The final output will be the category label. Theft.

[0223] The matching output is used for image-text correlation verification, specifically to verify whether a given police text description is highly correlated with a scene image, thus supporting the construction of the chain of evidence. It will... The score is compressed into a scalar score through a linear transformation and then mapped to the (0,1) interval using the sigmoid function. This score serves as the confidence probability for text-image relevance, calculated as follows:

[0224]

[0225]

[0226] in, and The output is the learnable parameters of the matching layer. This represents the probability of a text-image correlation; the closer to 1, the stronger the correlation. For example, in case file management, the system needs to verify whether a text record about "a red car at an intersection" matches a certain surveillance screenshot. The matching output subheader outputs a score. This indicates a significant mismatch between the text and the image, suggesting that the police may need to re-verify the physical evidence to prevent misattribution and ensure the accuracy of the chain of evidence.

[0227] The output is used to generate interpretable reasoning results, which is the highest manifestation of system intelligence. This sub-header is centered around a Transformer-based decoder, which... As the initial state and ongoing contextual information for the decoder, the decoder operates in an autoregressive manner, progressively generating a sequence of words to form a structurally complete and logically clear summary of the case's reasoning. For example, an image shows a messy indoor environment with personal belongings scattered on the floor (image evidence). Combined with the victim's description of 'lost phone and wallet' (textual evidence) and the rule in the knowledge base that 'scattered indoor items are often associated with theft' (knowledge evidence), it is highly probable that this case is a burglary. It is recommended to focus on investigating traces of illegal intrusion and tracing the whereabouts of the stolen goods.

[0228] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / eos> < / sos>

Claims

1. A method for police file and image association reasoning of a multi-modal large model, characterized in that, Includes the following steps: Step S1: Collect text and image files related to police work and perform data preprocessing; Step S2: Encode the preprocessed text and image files using a multimodal large model; Step S3: Align text files and image files across modalities and construct a heterogeneous evidence map; Step S4: Perform joint reasoning and output the reasoning results and evidence chain.

2. The method for multimodal large-scale model-based police document and image association reasoning according to claim 1, characterized in that, The data preprocessing involves performing layout analysis, text correction, and structured extraction on the files, quality assessment, noise reduction, and key region extraction on the image files, and desensitization of sensitive personal information.

3. The method for multimodal large-scale model-based police document and image association reasoning according to claim 1, characterized in that, The aforementioned multimodal large model comprises: an input layer, a processing layer, an encoding layer, a cross-modal fusion layer, a policing adaptive reasoning layer, and an output layer; The input layer is used to receive text files and image files; The processing layer is used to process the input data; The encoding layer includes a text encoder and an image encoder. The text encoder uses a pre-trained model based on the BERT architecture and extracts word-level and sentence-level feature representations from the text file through a 12-layer Transformer structure. The image encoder uses ResNet and utilizes a deep residual network structure to extract visual features from the image, including local region features and global semantic features. The cross-modal fusion layer is used to perform cross-modal alignment and fusion of text files and image files; The police adaptive reasoning layer is used to receive the output of the cross-modal fusion layer, generate preliminary reasoning results through the police task output header, and construct a heterogeneous evidence graph based on the preliminary reasoning results; the heterogeneous evidence graph is enhanced by injecting domain prior knowledge through a police knowledge graph, and then joint reasoning is performed based on the enhanced heterogeneous evidence graph. The output layer is used to output the joint inference results.

4. The method for multimodal large-scale model-based police document and image association reasoning according to claim 3, characterized in that, The word-level feature representation maps the processed police text sequence into a set of related word vectors. The specific steps are as follows: The input text file was processed to obtain the following text sequence: Where L represents the length of the text sequence, This represents each word in the text sequence; Then each word Mapping to word vectors The formula is as follows: in, For word embedding matrix, For position matrix, For paragraph matrix, Paragraph markers indicating words; The input is processed by the text encoder to obtain the final word-level representation matrix. : in, express Corresponding word-level feature representation; The sentence-level feature representation is obtained by using an aggregation mechanism to obtain a global semantic vector based on the word-level feature representation. The aggregation mechanism is a weighted summation of the word-level feature representation using an attention-weighted approach, expressed by the following formula: in, It is a sentence-level feature representation; These are queryable vectors used to measure the importance of words. The projection matrix projects the word-level feature representations onto the attention space. It is the importance weight of the i-th word. It is a non-linear activation function. This indicates the matrix transpose.

5. The method for multimodal large-scale model-based police document and image association reasoning according to claim 3, characterized in that, The local region features are obtained by passing the input image file I through ResNet layer-by-layer convolution, pooling, and residual concatenation to obtain feature map F, and then flattening the feature map F into a set of region features: in, This represents the feature of the nth local region in the image file. This represents the total number of local regions into which the image is divided; The global semantic features are based on local region features, and are weighted and aggregated through an attention mechanism: in, These are the global semantic features of the image; It is a queryable vector used to measure the importance of local features; U is the projection matrix that projects the local features into the attention space. It is the first Attention weights for local region features.

6. The method for multimodal large-scale model-based police document and image association reasoning according to claim 3, characterized in that, The cross-modal alignment and fusion include: Text modality includes word-level and sentence-level feature representations of text files, while image modality includes local region features and global semantic features of image files. The text modality and the image modality are mapped into a unified cross-modal semantic space, and cross-modal alignment and fusion are achieved through an attention mechanism; No. The word-level features are represented as follows: Sentence-level features are represented as , No. The characteristics of each local region are The global semantic features are The text modality and the image modality are mapped to a cross-modal semantic space through linear projection, as shown in the following formula: in, , , and For learnable projection matrices, , , and For bias terms; Represents global features of the text. Represents global features of an image; Represents word-level mapping features; Indicates the characteristics of the region mapping; Then, alignment is achieved through an attention mechanism, as shown in the following formula: in, This represents the relationship between the i-th word in the word-level mapping feature and the i-th region mapping feature. Weights between regions; The fusion representation is calculated using the following formula: in, This represents the visual context vector of the i-th word. Represents the visual context matrix; Then, the global features of the text are concatenated with the global features of the image, as shown in the following formula: in, This represents the concatenated global union representation. It is a fusion function; The final cross-modal joint representation is obtained. This is used to generate preliminary reasoning results in the police adaptive reasoning layer, and to construct a heterogeneous evidence graph and perform joint reasoning.

7. The method for multimodal large-scale model-based police document and image association reasoning according to claim 6, characterized in that, The police task output head is used to receive cross-modal joint representations and generate corresponding inference results according to different police task types; the task output head includes a classification subheader, a matching subheader, and a generation subheader; The classifier subheader is used to classify target objects, and outputs a class probability distribution based on a fully connected layer and a softmax function. The specific form of the probability distribution is a vector. Where C represents the total number of case categories distinguished by the model. This represents the predicted probability that the input belongs to c categories, and satisfies the following conditions: The category corresponding to the maximum value in the probability distribution. As a preliminary case type classification result; The matching subheader is used to achieve correlation matching between police text and images, and to calculate the cosine similarity score between the global features of the text and the global features of the image, as shown in the following formula: in, This represents the L2 magnitude of the vector. The formula for calculating similarity is given; the similarity score serves as a preliminary matching result for the correlation between images and text. The generated subheader uses a Transformer-based decoder to generate an initial inference text sequence in a regression manner; during the decoding process, a cross-modal joint representation is queried through a cross-attention mechanism. Proceed to output word sequences As a preliminary result of reasoning, among which Indicates the length of the word sequence; The preliminary classification results output by the classification subheader, the similarity scores output by the matching subheader, and the preliminary reasoning text output by the generated subheader are used to construct a heterogeneous evidence graph.

8. The method for multimodal large-scale model-based police document and image association reasoning according to claim 7, characterized in that, The heterogeneous evidence graph is generated from the cross-modal joint representation based on the preliminary classification results of the classification sub-head output, the similarity score of the matching sub-head output, and the preliminary inference text of the generated sub-head output. The heterogeneous evidence graph is constructed by selectively extracting information from a police knowledge graph; the heterogeneous evidence graph includes two parts: a set of nodes and a set of edges. The node set is composed of evidence information of different modalities, including text nodes, image nodes, and knowledge nodes; The edge set is used to characterize the semantic and evidence relationships between nodes, including same-modal edges, cross-modal edges, and knowledge edges.

9. The method for multimodal large-scale model-based police document and image association reasoning according to claim 8, characterized in that, The joint reasoning takes a heterogeneous evidence graph as input, uses a graph neural network to realize information interaction and evidence fusion between nodes, and finally outputs a case reasoning text; the joint reasoning includes a reasoning input layer, a reasoning propagation layer and a reasoning output layer. The inference input layer receives a heterogeneous evidence graph composed of text nodes, image nodes, and knowledge nodes. ,in, For a set of nodes, The set is an edge set; the initial feature representations of each node are derived from the following sources: Text node features originate from word-level visual context vectors Image node features originate from image region features. Knowledge node features are derived from the embedding of knowledge entities retrieved from the knowledge graph. ; The inference propagation layer, through a graph neural network architecture, transmits and updates features among text nodes, image nodes, and knowledge nodes in the heterogeneous evidence graph. Its update formula is as follows: in, This represents the feature representation of node v at layer l. Denotes the set of neighbors of node v. For edge weights, and For learnable parameters, It is a non-linear activation function; The inference output layer receives the passed... The enhanced node representation is obtained after layered inference propagation. A global case reasoning representation vector is generated through aggregation functions. The calculation formula is: in, Represents aggregate functions, Indicates the total number of inference propagation layers; Used for joint inference results in the output layer.

10. The method for multimodal large-scale model-based police document and image association reasoning according to claim 9, characterized in that, The joint reasoning result is a global case reasoning representation vector. The combined result is output through the three task output heads; The three task output heads include: classification output, matching output, and generation output; The classification output is... Input a fully connected layer and a softmax function, output case type classification labels; The matching output will The image-text matching score output is converted into a probability value using the sigmoid function. The generated output will As the initial state input decoder, autoregression generates structured case reasoning chain text.

Images