Multi-modal large model-based footprint image human body attribute automatic inference method

Abstract

The invention discloses a footprint image human body attribute automatic inference method based on a multi-modal large model, and relates to the technical field of mode recognition. The inference method comprises the following steps: constructing a CLIP-based footprint image and text analysis multi-modal large model: the model adopts a double-flow network structure, a graphic encoder processes image features, a text encoder processes text features, and cross-modal feature fusion is realized through a feature fusion layer; and constructing a Grad-CAM interpretable visualization model, wherein Grad-CAM is a gradient-based feature visualization technology in deep learning. The gradient of the feature map is output through the calculation model, an attention map is generated, and key areas concerned by the convolutional neural network model during decision making are displayed; the correlation between the footprint image and the human body attribute is deeply discussed, a new multi-modal feature fusion method and model architecture are provided, and the algorithm, the model and the application method thereof formed in the research process also promote the industrial application of the technology.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition technology, specifically relating to an automatic inference method for human attributes from footprint images based on a multimodal large model. Background Technology

[0002] Footprint examination technology is an important component of modern criminal investigation technology, widely applied in police investigations and subsequent court trials. Footprints are a common and crucial piece of physical evidence at crime scenes, providing abundant clues and information. By examining the characteristics of footprint images, individual suspect features can be inferred and physical comparisons made, playing a key role in criminal case investigation and suspect identification. Currently, traditional footprint examination techniques still rely heavily on expert experience for analysis and judgment, presenting several problems. First, they lack scientific rigor, heavily depending on subjective human judgment. Second, they are time-consuming and labor-intensive, resulting in low detection efficiency. Third, they require highly experienced personnel, hindering widespread adoption within the police system. These issues significantly limit the effectiveness of footprint analysis and even impact the extraction of footprints from crime scenes. After 2000, the China Criminal Police University attempted to automate footprint analysis using computer technology, leveraging its expertise in footprint analysis. However, due to the complexity of footprint images, the developed software system still requires intervention from experienced technicians, exhibiting low robustness and a high error rate. Summary of the Invention

[0003] To address the problems mentioned in the background section, the present invention aims to provide an automatic method for inferring human attributes from footprint images based on a multimodal large model.

[0004] The present invention provides an automatic inference method for human attributes from footprint images based on a multimodal large model. The inference method is as follows:

[0005] Step 1: Construct a large multimodal model for footprint image and text analysis based on CLIP: The model adopts a two-stream network structure, with a graph encoder processing image features and a text encoder processing text features, and cross-modal feature fusion is achieved through a feature fusion layer;

[0006] Step 2: Construct a Grad-CAM interpretable and visual model:

[0007] Grad-CAM is a gradient-based feature visualization technique in deep learning. By calculating the gradient of the model output with respect to the feature map, it generates an attention map, which shows the key regions that the convolutional neural network model focuses on during decision-making.

[0008] As a preferred approach, the CLIP model maps images and text to the same feature space using a contrastive learning method. By maximizing the similarity of matching image-text pairs and minimizing the similarity of mismatched pairs, it achieves cross-modal feature alignment. For a training dataset containing N text-image pairs, the CLIP model combines N text features and N image features pairwise. The CLIP model predicts the similarity of N² possible text-image pairs. The similarity is directly calculated by the cosine similarity between the text features and the image features, including N positive samples (i.e., text and image belonging to a pair) and the remaining text-image pairs as negative samples. The training objective of CLIP is to maximize the similarity of the N positive samples while minimizing the similarity of the negative samples. The loss function is as follows:

[0009]

[0010] Where sim() represents the similarity function, This represents the temperature parameter, and N represents the number of samples.

[0011] As a preferred embodiment, Grad-CAM is a weighted sum of the forward activation values ​​of a CNN, passed through a modified linear unit.

[0012] As a preferred embodiment: the modified linear unit (MRU) is a commonly used activation function in artificial neural networks, referring to the ramp function in mathematics. Given a footprint image and the class of interest as input, the image is forward-propagated through the CNN part of the model. Then, the raw score for that class is obtained through task-specific computation. The gradients of all classes are set to 0, except for the desired class, which is set to 1. This signal is then backpropagated to the modified convolutional feature map of interest, and the combined features are stacked to compute Grad-CAM localization, representing the location the model must find. Where:

[0013]

[0014] in, Represents gradient weights. Representing the feature map, Grad-CAM generates the final attention map by weighted summation of gradient weights and feature maps.

[0015] Compared with existing technologies, the beneficial effects of this invention are: it will explore the relationship between footprint images and human attributes in depth, propose new multimodal feature fusion methods and model architectures, and the algorithms, models and application methods formed in the research process will also promote the industrial application of the technology. Attached Figure Description

[0016] For ease of explanation, the present invention will be described in detail below with reference to specific embodiments and accompanying drawings.

[0017] Figure 1 This is a schematic diagram of CLIP comparison pre-training in this invention;

[0018] Figure 2 This is a schematic diagram of the Grad-CAM interpretable and visualized model in this invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is described below with reference to specific embodiments shown in the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. The structures, proportions, sizes, etc., illustrated in the accompanying drawings are only for illustrative purposes and to enable those skilled in the art to understand and read the invention, and are not intended to limit the conditions under which the invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to the size, without affecting the effects and objectives achieved by the invention, should still fall within the scope of the technical content disclosed in this invention. Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concept of the invention.

[0020] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0021] Combination Figure 1 , Figure 2 The following is an explanation of this specific embodiment, which adopts the following technical solution: Its deduction method is as follows:

[0022] Step 1: Construct a multimodal large model for footprint image and text analysis based on CLIP: The CLIP (Contrastive Language-Image Pre-Training) model is a pre-trained neural network model released by OpenAI in early 2021 for matching images and text. The model employs a two-stream network structure. An image encoder processes image features, and a text encoder processes text features. Cross-modal feature fusion is achieved through a feature fusion layer. The CLIP model uses a contrastive learning method to map images and text into the same feature space. By maximizing the similarity of matching image-text pairs and minimizing the similarity of mismatched pairs, cross-modal feature alignment is achieved. For a training dataset containing N text-image pairs, the CLIP model combines N text features and N image features pairwise. The CLIP model predicts the similarity of N² possible text-image pairs. This similarity is directly calculated using the cosine similarity between text and image features. It includes N positive samples (text and image pairs, represented by diagonal elements in the matrix) and the remaining text-image pairs are negative samples. The training objective of CLIP is to maximize the similarity of the N positive samples while minimizing the similarity of the negative samples. The loss function is as follows:

[0023] .

[0024] Where sim() represents the similarity function, This represents the temperature parameter, and N represents the number of samples.

[0025] Step 2: Construct a Grad-CAM interpretable and visual model:

[0026] Grad-CAM (Gradient-weighted Class Activation Mapping) is a gradient-based feature visualization technique in deep learning. It generates an attention map by calculating the gradient of the model's output onto the feature map, showing the key regions that the Convolutional Neural Network (CNN) model focuses on during decision-making. Grad-CAM is achieved by weighted summation of the CNN's forward activation values ​​and then passing them through a rectified linear unit (CLU). The CLU is a commonly used activation function in artificial neural networks, representing a ramp function in mathematics. Given a footprint image and the class of interest as input, the image is propagated forward through the CNN part of the model. The raw score for that class is then obtained through task-specific computation. The gradients of all classes are set to 0, except for the desired class, which is set to 1. This signal is then backpropagated to the rectified convolutional feature map of interest, and the combined signals are stacked to calculate the Grad-CAM localization, representing the location the model must find.

[0027]

[0028] in, Represents gradient weights. The feature map is represented. Grad-CAM generates the final attention map by weighted summation of gradient weights and feature maps.

[0029] This specific implementation method constructs a multimodal large-scale model for footprint image and text analysis. First, preprocessing and feature extraction of footprint image data and expert text data are required. The collected footprint image data undergoes preprocessing steps such as image enhancement, denoising, and standardization using pre-trained models (e.g., ResNet, Inception), automatically extracting multi-level feature representations. Expert text data is processed using deep learning-based language models (e.g., BERT, GPT) to extract semantic feature representations. The extracted footprint image features and text features are used as input. A multimodal model capable of processing image and text features is designed based on the CLIP model, achieving efficient feature fusion and accurate inference of human attributes. The CLIP model contains two independent encoders. The image encoder converts footprint images into image feature vectors, and the text encoder converts text descriptions into text feature vectors. Through contrastive learning, images and text are mapped to the same feature space. The two encoders are optimized to minimize the distance between matching image-text pairs and maximize the distance between mismatched pairs, achieving cross-modal feature alignment. To further improve the performance of multimodal large models, an attention mechanism can be introduced to enable information interaction within image features and text features, thereby achieving more refined feature fusion.

[0030] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.

[0031] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multi-modal large model-based footprint image human attribute automatic inference method, characterized in that: Its inference method is as follows: Step one: construct a multi-modal large model based on CLIP foot image and text analysis: the model adopts a double-flow network structure, a graph encoder processes image features, a text encoder processes text features, and cross-modal feature fusion is realized through a feature fusion layer; Step two: build a Grad-CAM interpretable visualization model: Grad-CAM is a gradient-based feature visualization technology in deep learning; By calculating the gradient of the model output on the feature map, a focus map is generated, which shows the key areas that the convolutional neural network model focuses on when making decisions.

2. The method of claim 1, wherein the method is based on a multi-modal large model for automatic human attribute inference of a footprint image. The CLIP model maps images and texts into the same feature space through a contrast learning method, maximizes the similarity of matching image-text pairs, and minimizes the similarity of non-matching pairs, aligns cross-modal features, and for a training containing N text-image pairs, the N text features and N image features are combined two by two, the CLIP model will predict the similarity of N2 possible text-image pairs, the similarity is the cosine similarity of the text feature and the image feature, contains N positive samples, that is, the text and image belonging to a pair, the remaining text-image pairs are negative samples, the training goal of CLIP is to maximize the similarity of N positive samples, while minimizing the similarity of negative samples, the loss function is as follows: wherein sim() denotes a similarity function, denotes a temperature parameter, and N denotes a number of samples.

3. The method of claim 1, wherein the method is based on a multi-modal large model. The Grad-CAM is the weighted sum of the forward activation values of CNN, and then passes through a rectified linear unit.

4. The method of claim 3, wherein the method is based on a multi-modal large model. The rectified linear unit is a commonly used activation function in artificial neural networks, which refers to the ramp function in mathematics, taking a given footprint image and a class of interest as input, propagating the image forward through the CNN part of the model, then obtaining the original score of the class through task-specific calculation, except for the expected class which is set to 1, all class gradients are set to 0, then the signal is back-propagated to the correction convolution feature map of interest, combined and superimposed to calculate the Grad-CAM positioning, which represents the location that the model must find, wherein: wherein, denotes the gradient weight, denotes the feature map, Grad-CAM generates the final attention map through the weighted sum of the gradient weight and the feature map.

Images