Method and system for intelligent classification and verification of evidence photos based on deep learning

Abstract

The present invention provides a method and system for intelligent classification and consolidation of evidence photos based on deep learning. The method comprises the following steps: first, obtaining evidence photos and preprocessing them according to the case type, and then inputting them into a deep learning model comprising a multi-level classification framework and a feature enhancement module. The feature enhancement module extracts key information features in the photos, and the multi-level classification framework performs coarse-grained and fine-grained classification based on these features to form an initial photo classification set. Subsequently, the logical associations between different photos are evaluated through an evidence correlation analysis mechanism, and the classification results are optimized to form a target photo classification set. Finally, the integrity of the classification set is tested according to the preset evidence specification requirements of the case type, and the consolidation data set is output after verification. The present invention realizes the intelligent and automatic classification of evidence photos, significantly improving the classification efficiency and accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a method and system for intelligent classification and consolidation of evidence photos based on deep learning. Background Art

[0002] In the field of judicial evidence management and processing, traditional methods face significant challenges in the inefficiency and inaccuracy of photo evidence classification. With the surge in caseloads and the diversification of evidence forms, legal personnel are faced with the time and effort required to manually classify and organize various types of evidence photos. This is not only inefficient but also prone to human error. Especially in complex cases, where the number of evidence photos is large and diverse, traditional manual classification methods are unable to meet the high efficiency and accuracy requirements of modern judicial work. Summary of the Invention

[0003] The present invention provides a method and system for intelligent classification and consolidation of evidence photos based on deep learning, so as to solve the problems of low efficiency and proneness to errors in the manual evidence consolidation process.

[0004] In a first aspect, the present invention provides a method for intelligent classification and consolidation of evidence photos based on deep learning, the method comprising the following steps:

[0005] Obtain evidence photos to be classified and pre-process them according to the case type;

[0006] Build an evidence photo classification model based on a deep learning model, which includes a multi-level classification framework and feature enhancement modules;

[0007] The pre-processed evidence photos are input into the evidence photo classification model, and the key information features in the evidence photos are extracted through the feature enhancement module;

[0008] A multi-level classification framework is used to perform coarse-grained and fine-grained classification of evidence photos based on key information features to obtain an initial photo classification set.

[0009] Analyze the correlation between different evidence photos in the initial photo classification set through the evidence correlation analysis mechanism, and optimize the initial photo classification set into the target photo classification set based on the correlation analysis results;

[0010] The integrity of the target photo classification set is detected according to the evidence specifications preset by the case type. If the integrity verification passes, the target photo classification set is output as the solid evidence dataset.

[0011] Optionally, the preprocessing step includes an image scaling step, an illumination equalization step, and a noise removal step.

[0012] Optionally, building an evidence photo classification model based on a deep learning model that includes a multi-level classification framework and a feature enhancement module includes the following steps:

[0013] The pre-trained YOLOn8 model is used as the base model, and a feature enhancement module is embedded in the base model. The feature enhancement module includes a text recognition unit, an object detection unit, and a feature fusion unit.

[0014] Construct a multi-level classification framework in the basic model, which includes a coarse-grained classification network and a fine-grained classification network;

[0015] Establish a residual connection between the coarse-grained classification network and the fine-grained classification network, and introduce an attention mechanism into the fine-grained classification network to construct an initial evidence photo classification model;

[0016] Initialize the model parameters of the initial evidence photo classification model, use the labeled evidence photo dataset and transfer learning method to complete the training and optimization of the initial evidence photo classification model, and obtain the evidence photo classification model.

[0017] Optionally, extracting key information features from the evidence photo using a feature enhancement module includes the following steps:

[0018] Extract text information from evidence photos using optical character recognition technology in the text recognition unit;

[0019] Utilize the preset target detection algorithm in the target detection unit to identify and extract key area features in the evidence photo;

[0020] The text information and key area features are fused into enhanced feature representation through the feature fusion unit;

[0021] The enhanced feature representation is fused with the original evidence photo to form key information features.

[0022] Optionally, using a multi-level classification framework to perform coarse-grained and fine-grained classification of evidence photos based on key information features to obtain an initial photo classification set includes the following steps:

[0023] The key information features are coarsely classified through the coarse-grained classification network to obtain the first-level classification results;

[0024] The first-level classification results are fused with the original key information features through residual connections to form fusion features;

[0025] The fused features are input into the fine-grained classification network for fine-grained classification. During the fine-grained classification process, the attention mechanism is used to enhance the perception ability of the fine-grained classification network on fine-grained category features, and the second-level classification results are obtained;

[0026] The second-level classification results of the evidence photos and the corresponding confidence levels are integrated to obtain the initial photo classification set.

[0027] Optionally, analyzing the correlation between different evidence photos in the initial photo classification set through an evidence correlation analysis mechanism, and optimizing the initial photo classification set into a target photo classification set based on the correlation analysis results includes the following steps:

[0028] Extract metadata information from the evidence photos in the initial photo classification set, the metadata information including the timestamp and geographic location;

[0029] Combining metadata information and key information features, the feature correlation between different evidence photos in the initial photo classification set is calculated;

[0030] Constructing an evidence photo association graph of the initial photo classification set based on feature association;

[0031] The initial photo classification set is optimized into the target photo classification set using the evidence photo association graph and the preset evidence logic relationship rule library.

[0032] Optionally, checking the integrity of the target photo classification set according to the evidence specification requirements preset for the case type includes the following steps:

[0033] Define the necessary evidence types based on the evidence standards set by the case type;

[0034] Perform statistical analysis on the classified evidence photos in the target photo classification set to identify the types of evidence collected in the current case;

[0035] Compare the collected evidence types with all necessary evidence types to detect the completeness of the target photo classification set.

[0036] Optionally, the method further includes the following steps:

[0037] Analyze the computational complexity and storage requirements of evidence photo classification models;

[0038] Based on computational complexity, the model knowledge of the evidence photo classification model is transferred to a preset lightweight classification model through knowledge distillation technology;

[0039] The model pruning method is used to remove network connections in the lightweight classification model whose contribution to the classification results is less than a preset threshold;

[0040] Use quantization technology to reduce the accuracy requirements of model parameters in lightweight classification models, and use model compression technology to reduce the model size until the storage requirements of lightweight classification models meet the deployment requirements of mobile devices;

[0041] Deploy lightweight models on mobile devices.

[0042] In the second aspect, the present invention also provides an evidence photo intelligent classification and verification system based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the evidence photo intelligent classification and verification method based on deep learning as described in the first aspect.

[0043] In a third aspect, the present invention also provides a computer-readable storage medium having instructions stored thereon, which, when executed by a processor, configures the processor to execute the method for intelligent classification and consolidation of evidence photos based on deep learning in the first aspect.

[0044] The beneficial effects of the present invention are:

[0045] Compared with the existing technology, the present invention has significant beneficial effects, which are mainly reflected in three aspects: efficiency improvement, accuracy enhancement and system improvement. By introducing deep learning technology and a multi-level classification framework, the present invention realizes the intelligent automatic classification of evidence photos, which greatly reduces manual intervention. The application of the feature enhancement module enables the system to extract richer and more accurate information features from evidence photos, and can achieve high-accuracy classification even in the case of poor photo quality or incomplete information. The original evidence correlation analysis mechanism of the present invention can automatically identify the logical relationship and spatiotemporal connection between different evidence photos, construct a complete chain of evidence, and avoid the problem of evidence fragmentation in traditional methods. At the same time, the evidence specification requirement detection function based on the case type ensures the integrity and compliance of the evidence set, reducing procedural defects caused by insufficient or non-standard evidence. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of a method for intelligent classification and verification of evidence photos based on deep learning in one embodiment of the present application.

[0047] Figure 2 This is a schematic diagram of the secondary classification of evidence photos of drunk driving cases in one embodiment of this application. DETAILED DESCRIPTION

[0048] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0049] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0050] Figure 1 The figure is a flowchart of a method for intelligent classification and verification of evidence photos based on deep learning in one embodiment. Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps in the above process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps. Figure 1 As shown, the method for intelligent classification and consolidation of evidence photos based on deep learning disclosed in the present invention specifically includes the following steps:

[0051] S101. Obtain evidence photos to be classified and pre-process the evidence photos according to the case type.

[0052] In the intelligent classification and consolidation of evidence photos, first, data acquisition equipment, such as professional forensic cameras, smartphones, or other digital imaging devices, captures the evidence photos to be classified. After acquiring the photos, appropriate preprocessing strategies are selected based on the case type (e.g., traffic accident, labor dispute, contract dispute, etc.). Preprocessing involves three key steps: image scaling, illumination equalization, and noise removal. Image scaling resizes all photos to a standard resolution (e.g., 1024×768 pixels) to ensure consistency and computational efficiency in subsequent processing. Illumination equalization utilizes histogram equalization technology to redistribute the grayscale values of image pixels, enhancing image contrast and ensuring a consistent brightness distribution across evidence photos taken under different lighting conditions. For example, for underexposed nighttime accident scene photos, contrast in dark areas can be enhanced to make key details more visible. Noise removal uses algorithms such as Gaussian filtering or bilateral filtering to improve image quality while preserving edge details and removing random noise. Preprocessing parameters vary depending on the case type. For example, traffic accident cases may prioritize preserving vehicle damage details, while contract dispute cases may prioritize text clarity.

[0053] S102. Construct an evidence photo classification model based on a deep learning model, which includes a multi-level classification framework and a feature enhancement module.

[0054] The evidence photo classification model is based on a pre-trained YOLOv8 model, which has strong feature extraction capabilities on large-scale image datasets. This model is then embedded in a feature enhancement module, which comprises three core units: a text recognition unit employs OCR technology based on a CRNN (convolutional recurrent neural network) to identify textual information in evidence photos, such as contract numbers and license plate numbers; an object detection unit, based on the Faster R-CNN algorithm, is specifically trained to identify key objects in different case types, such as vehicle damage in traffic accidents and scars in personal injuries; and a feature fusion unit uses an attention mechanism to weightedly fuse textual information and visual features. The multi-level classification framework consists of two levels: coarse-grained and fine-grained classification networks. The coarse-grained network, using the ResNet-50 architecture, is responsible for classifying evidence photos into broad categories (e.g., contracts, bills, and scene photos); the fine-grained network, using the DenseNet-121 architecture, further subdivides broad categories into specific categories (e.g., contracts into labor contracts and lease contracts). Residual connections are used to transfer information between the two-level networks, preventing the vanishing gradient problem. A squeeze-and-excitation (SE) attention mechanism is introduced in the fine-grained network to enhance the perception of subtle features by learning the interdependencies between channels. The model is trained using transfer learning, first freezing the base layers and training only the newly added layers. The entire network is then fine-tuned. The cross-entropy loss function and the Adam optimizer are used, with a learning rate set to 0.001 and decayed by 10% every 50 epochs. This effectively improves the model's ability to recognize features of legal evidence.

[0055] S103. Input the pre-processed evidence photos into the evidence photo classification model, and extract key information features in the evidence photos through the feature enhancement module.

[0056] The preprocessed evidence photos are first input into the evidence photo classification model, where key information features are extracted using the feature enhancement module. The text recognition unit uses optical character recognition (OCR) to identify text within the photos. This technology first locates text regions using the MSER (Maximally Stable Extremal Region) algorithm, and then uses a CRNN (Convolutional Recurrent Neural Network) model to recognize the text content. For example, in contract photos, key text information such as "Contract Number" and "Signing Date" can be identified with over 95% accuracy. The object detection unit uses a pre-defined object detection algorithm (such as Faster R-CNN) to identify and extract key features within the photos. This unit has pre-trained specialized detectors for different case types. For example, in traffic accident cases, it can accurately locate key areas such as vehicle damage and license plate numbers. A confidence threshold of 0.75 is used to ensure the reliability of detection results. The feature fusion unit uses a BiLSTM (Bidirectional Long Short-Term Memory) network architecture to fuse text information vectors with visual feature vectors. Finally, the fused enhanced feature representation is combined with the deep features of the original evidence photo through skip connections to form key information features containing rich semantic and visual information, greatly improving the accuracy of subsequent classification.

[0057] S104. Use a multi-level classification framework to perform coarse-grained and fine-grained classification on the evidence photos based on key information features to obtain an initial photo classification set.

[0058] The multi-level classification framework first processes key information features through a coarse-grained classification network. This network uses a ResNet-50 architecture and consists of five residual blocks, each of which includes multiple convolutional layers, batch normalization layers, and ReLU activation functions. The coarse-grained classification classifies evidence photos into basic categories, such as "contract documents," "bills and receipts," and "on-site photos," outputting first-level classification results and corresponding confidence scores. During the classification process, a softmax function is used to calculate the probability of each category. Subsequently, the first-level classification results are fused with the original key information features through residual connections. This fusion preserves the original feature information while incorporating semantic information from the coarse-grained classification. The fused features are then fed into a fine-grained classification network using a DenseNet-121 architecture for enhanced feature extraction. During the fine-grained classification process, the SE attention mechanism is introduced to compress the feature map into channel descriptors through global average pooling. Subsequently, a two-layer fully connected network learns the importance weights between channels and recalibrates the weights of each channel in the feature map. This enables the network to focus on feature channels that are more important for fine-grained classification, such as the signature area in a contract document and the amount in a bill. Ultimately, the fine-grained classification network outputs second-level classification results, such as further subdividing "contract document" into specific categories such as "labor contract," "lease contract," and "sales contract." The second-level classification results and confidence levels for each evidence photo are integrated to form an initial photo classification set, providing a foundation for subsequent correlation analysis.

[0059] S105. Analyze the correlation between different evidence photos in the initial photo classification set through the evidence correlation analysis mechanism, and optimize the initial photo classification set into the target photo classification set based on the correlation analysis results.

[0060] The evidence relevance analysis mechanism first extracts metadata information for each evidence photo from the initial photo classification set, including the shooting timestamp (accurate to the second) and geographic coordinates (latitude and longitude). For photos lacking GPS information, the location is inferred based on the scene features in the photo. Combining the metadata information with the previously extracted key information features, the feature relevance between different evidence photos is calculated. The relevance is calculated using the weighted cosine similarity formula:

[0061] ;

[0062] in is the temporal similarity (exponential decay function based on the shooting time interval), is the geographic location similarity (Gaussian function based on coordinate distance), is the content similarity (cosine similarity based on key information features), 、 、 The corresponding weight coefficients are dynamically adjusted based on the case type. Based on the calculated feature correlation, an evidence photo association graph is constructed, where nodes represent evidence photos and edges represent the strength of associations between photos. Once the association graph is constructed, a community detection algorithm (such as the Louvain algorithm) is used to identify clusters of highly correlated evidence photos. This is then optimized using a pre-set rule base for logical evidence relationships. The rule base contains the logical relationships between evidence under different case types. For example, in a traffic accident case, photos of vehicle damage and photos of the accident scene should be consistent in time and location. Through rule-based reasoning, the system can identify and correct errors in initial classification, such as reclassifying a "payment voucher" that was incorrectly classified as a "contract attachment" or grouping photos with high temporal and location correlations into the same event sequence. The resulting target photo classification set achieves higher classification accuracy and logical consistency.

[0063] S106. Check the integrity of the target photo classification set according to the evidence specification requirements preset for the case type. If the integrity verification passes, output the target photo classification set as the solid evidence data set.

[0064] Among them, the integrity detection step first calls the preset evidence specification requirement library according to the case type, which defines the necessary evidence types required for different case types. For example, for labor dispute cases, the necessary evidence types include labor contracts, wage payment vouchers, attendance records, etc.; for traffic accident cases, they include accident scene photos, vehicle damage photos, injured photos, responsibility determination letters, etc. By querying the specification requirement library, the necessary evidence list and its importance weight under the current case type are obtained. Subsequently, a statistical analysis is performed on the classified evidence photos in the target photo classification set to identify the types of evidence collected for the current case and its quantity and quality. The statistical analysis adopts a weighted scoring mechanism, and the calculation formula is: ,in is the importance weight of the i-th category of evidence, is the average quality score of this type of evidence (based on factors such as image clarity and information completeness), is the effective number of evidence of this type. The collected evidence types are compared with the required evidence types to calculate the evidence coverage. The system sets a coverage threshold (usually 0.8 or higher). When the coverage exceeds the threshold, the integrity verification passes. For cases that fail the verification, the system will generate a list of missing evidence prompts, sorted by importance, to guide the user to collect additional evidence. When the integrity verification passes, the system outputs the target photo classification set as a fixed evidence dataset, and simultaneously generates a metadata file containing the evidence classification structure, association relationships, and integrity assessment report.

[0065] In one embodiment, building an evidence photo classification model based on a deep learning model that includes a multi-level classification framework and a feature enhancement module includes the following steps:

[0066] The pre-trained YOLOn8 model is used as the base model, and a feature enhancement module is embedded in the base model. The feature enhancement module includes a text recognition unit, an object detection unit, and a feature fusion unit.

[0067] Construct a multi-level classification framework in the basic model, which includes a coarse-grained classification network and a fine-grained classification network;

[0068] Establish a residual connection between the coarse-grained classification network and the fine-grained classification network, and introduce an attention mechanism into the fine-grained classification network to construct an initial evidence photo classification model;

[0069] Initialize the model parameters of the initial evidence photo classification model, use the labeled evidence photo dataset and transfer learning method to complete the training and optimization of the initial evidence photo classification model, and obtain the evidence photo classification model.

[0070] In this implementation, YOLOv8 was chosen as the base model due to its excellent performance and efficiency in object detection. This model utilizes the CSPDarknet53 backbone network, which offers powerful feature extraction capabilities. Furthermore, a feature enhancement module is embedded to enhance the model's ability to identify key information in evidence photos. This feature enhancement module first includes a text recognition unit, based on a CRNN (Convolutional Recurrent Neural Network) architecture. It consists of three components: a convolutional layer to extract text image features, a bidirectional LSTM layer to capture text sequence features, and a CTC (Connected Temporal Classification) decoding layer to output the final text content. The text recognition unit is capable of extracting key textual information from evidence photos, such as contract numbers, dates, and amounts. The object detection unit, based on the Faster R-CNN architecture, comprises a region proposal network (RPN) and an object classification network. This unit detects specific objects in evidence photos of different case types, such as damaged vehicle parts in traffic accidents and scars in personal injuries. The object detection unit uses the non-maximum suppression (NMS) algorithm to filter overlapping detection boxes, with an IoU threshold of 0.5 to ensure accurate detection results. The feature fusion unit uses an attention mechanism to fuse textual and visual features, employing a bilinear pooling approach. This fusion method effectively captures high-order correlations between textual and visual features, significantly improving the model's ability to understand complex evidence photos.

[0071] The design concept of the multi-level classification framework is to gradually refine the categories of evidence photos through a hierarchical classification strategy, improving classification accuracy and efficiency. The coarse-grained classification network uses the ResNet-50 architecture, which consists of five residual blocks. Each residual block consists of multiple convolutional layers, batch normalization layers, and ReLU activation functions. The core formula of the residual block is: H(x) = F(x) + x, where F(x) represents the residual map and x represents the input features. This design effectively alleviates the vanishing gradient problem of deep networks, enabling the network to learn deeper feature representations. The coarse-grained classification network inputs the feature map extracted by the YOLOv8 base model. After global average pooling, it is connected to a fully connected layer and a softmax layer. The output is a probability distribution of the broad categories of the evidence photos, such as "contract document category," "bill voucher category," and "scene photo category." The coarse-grained classification uses the cross-entropy loss function. The fine-grained classification network uses the DenseNet-121 architecture, characterized by each layer being directly connected to all previous layers, forming a densely connected structure. The connection is: x_l = H_l([x_0,x_1,...,x_{l-1}]), where [x_0,x_1,...,x_{l-1}] represents the concatenation of features from all previous layers, and H_l represents a composite function, including batch normalization, ReLU activation, and convolution. This densely connected structure maximizes feature reuse, reduces the number of parameters, and improves the network's ability to perceive subtle features. The fine-grained classification network is responsible for further subdividing coarse-grained categories, such as "contract documents" into specific categories such as "labor contracts," "lease contracts," and "sales contracts." This multi-level classification framework allows the model to quickly determine the general category of evidence photos and then precisely identify their specific type, significantly improving both classification accuracy and computational efficiency.

[0072] The residual connections between the coarse-grained and fine-grained classification networks are implemented using skip connections. This connection effectively transfers semantic information from the coarse-grained classification to the fine-grained classification network while preserving the original feature information. This is achieved by fusing the feature map of the coarse-grained classification network with its output class probability vector. This residual connection design enables the fine-grained classification network to fully leverage the results of the coarse-grained classification, improving classification accuracy and robustness. The attention mechanism introduced in the fine-grained classification network uses a SE module, which adaptively adjusts the importance weights of feature channels. The SE module first performs a "squeeze" operation on the feature map using global average pooling, compressing the spatial information of each channel into a single value to form a channel descriptor. It then performs an "excitation" operation through two fully connected layers to learn the interdependencies between channels. Finally, the learned channel weights are multiplied with the original feature map to obtain a recalibrated feature map. This attention mechanism enables the network to focus on feature channels that are more important for fine-grained classification, such as the signature area in a contract document or the amount in a bill. By combining residual connections and attention mechanisms, the initial evidence photo classification model significantly improves its ability to perceive subtle features while maintaining computational efficiency, especially for evidence photo categories that are visually similar but have different details, the classification accuracy is significantly improved.

[0073] The initialization process uses a layered strategy. For the YOLOv8 base model, weight parameters pre-trained on a large-scale object detection dataset are directly loaded, resulting in robust feature extraction capabilities. For the newly added feature enhancement module and multi-level classification framework, the He initialization method is used for parameter initialization. This method adjusts the initialization variance based on the input dimension of each layer, calculated as: Var(W) = 2 / n_in, where n_in is the number of input neurons in that layer. This initialization method effectively prevents vanishing or exploding gradients in deep networks. Training utilizes transfer learning in three phases: In the first phase, the parameters of the YOLOv8 base model are frozen, and only the newly added modules are trained with a learning rate of 0.001, the Adam optimizer, a batch size of 32, and 50 epochs. In the second phase, the high-level parameters of the base model are unfrozen, the learning rate is reduced to 0.0001, and training continues for 30 epochs. In the third phase, all parameters are unfrozen, the learning rate is further reduced to 0.00001, and the entire network is fine-tuned for 20 epochs. The loss function adopts a multi-task learning framework, which includes the weighted sum of coarse-grained classification loss, fine-grained classification loss and feature enhancement module loss.

[0074] In one embodiment, extracting key information features from the evidence photo using the feature enhancement module includes the following steps:

[0075] Extract text information from evidence photos using optical character recognition technology in the text recognition unit;

[0076] Utilize the preset target detection algorithm in the target detection unit to identify and extract key area features in the evidence photo;

[0077] The text information and key area features are fused into enhanced feature representation through the feature fusion unit;

[0078] The enhanced feature representation is fused with the original evidence photo to form key information features.

[0079] In this implementation, the text recognition unit utilizes an improved CRNN (Convolutional Recurrent Neural Network) architecture, combined with a Transformer attention mechanism, to achieve high-precision extraction of text from evidence photos. The unit first preprocesses the input image, including grayscale conversion, binarization, and adaptive thresholding to enhance the contrast between text and background. The preprocessed image then passes through a feature extraction network consisting of six convolutional layers, each followed by batch normalization and a ReLU activation function. The convolution kernel size is 3×3, the stride is 1, and the padding is 1. The output of the feature extraction network is downsampled by a max pooling layer to form a feature sequence F_text. This feature sequence is fed into a bidirectional LSTM network with a hidden layer dimension of 256 to capture the context of the text. The LSTM output is then weighted using a self-attention mechanism. Finally, the weighted features are converted into a text sequence through a fully connected layer and a CTC (Connection Temporal Classification) decoder. For Chinese text recognition, character-level recognition is employed, with a character set consisting of approximately 7,000 common Chinese characters, numbers, and punctuation. In actual application, this unit can effectively identify key text information in evidence photos, such as contract number, date, amount, name of the parties, etc.

[0080] The object detection unit, based on a modified Cascade R-CNN architecture, specifically targets key regions in evidence photos for precise localization and feature extraction. This unit utilizes the ResNeXt-101 backbone network, combined with the Feature Pyramid Network (FPN) for multi-scale feature extraction. Through a top-down approach and lateral connections, the FPN constructs feature maps at five scales, from P2 to P6, corresponding to resolutions of 1 / 4 to 1 / 64 of the original image. Based on this, a three-stage cascaded detection head is designed, with the IoU thresholds set to 0.5, 0.6, and 0.7 at each stage, respectively, to gradually improve detection accuracy. The detection head consists of two branches: a bounding box regression branch and a class prediction branch. The bounding box regression uses a smooth L1 loss function, while the class prediction uses a cross-entropy loss function. Different key region categories are predefined for different types of evidence photos, such as the "signature area," "seal area," and "contract title" in contract documents; the "amount area," "date area," and "number area" in bills and vouchers; and the "damage area" and "accident scene" in scene photos. Overlapping detection boxes are filtered using the non-maximum suppression (NMS) algorithm, with an IoU threshold of 0.3, retaining the detections with the highest confidence. For each detected key region, a fixed-size (7×7) feature map is extracted using the RoIAlign layer, and then a 256-dimensional region feature vector is generated using two fully connected layers.

[0081] The feature fusion unit uses a multimodal attention fusion mechanism to deeply fuse the text features extracted by the text recognition unit and the key region features extracted by the object detection unit. First, the text features and region features are transformed to have the same dimensionality (256). Then, a bidirectional cross-attention mechanism is used to calculate the correlation between the two features, and cross-attention features are generated based on the attention weights. Next, the original features and cross-attention features are adaptively fused using a gating mechanism. Finally, a multi-layer perceptron and residual connections are used to generate the final enhanced feature representation. This fusion approach fully captures the complementarity and correlation between textual information and visual features, resulting in an enhanced feature representation that is richer and more robust than single-modality features.

[0082] The process of fusing the enhanced feature representation with the original evidence photo utilizes feature map reconstruction and injection techniques to achieve seamless information integration. First, a pre-trained ResNet-50 network is used to extract a multi-level feature map from the original evidence photo. These features include low-level (conv2_x), mid-level (conv3_x), and high-level (conv4_x) features, capturing edge, texture, and semantic information, respectively. The enhanced feature representation is then reconstructed into a feature map with the same spatial dimensions as the original feature map using a deconvolutional network. The deconvolutional network consists of three transposed convolutional layers, each followed by batch normalization and a ReLU activation function, gradually expanding the feature dimension from 256 to match the original feature map. Next, a feature injection module is designed to inject the reconstructed feature map into the original feature map using spatially adaptive instance normalization (SPADE). To prevent excessive modification of the original image information, a residual connection is introduced. Finally, a feature pyramid network (FPN) is used to integrate the fused features from different levels to generate the final key information features. This multi-level, adaptive feature fusion method can highlight the key information areas in the evidence photo while retaining the visual information of the original image, allowing subsequent classification and analysis tasks to focus more on the core content of the evidence.

[0083] In one embodiment, using a multi-level classification framework to perform coarse-grained and fine-grained classification of evidence photos based on key information features to obtain an initial photo classification set includes the following steps:

[0084] The key information features are coarsely classified through the coarse-grained classification network to obtain the first-level classification results;

[0085] The first-level classification results are fused with the original key information features through residual connections to form fusion features;

[0086] The fused features are input into the fine-grained classification network for fine-grained classification. During the fine-grained classification process, the attention mechanism is used to enhance the perception ability of the fine-grained classification network on fine-grained category features, and the second-level classification results are obtained;

[0087] The second-level classification results of the evidence photos and the corresponding confidence levels are integrated to obtain the initial photo classification set.

[0088] In this embodiment, the coarse-grained classification network adopts an improved EfficientNet-B3 architecture, which is specially optimized for the first-level classification task of evidence photos. The network first receives key information features of size 256×256×3 as input, and extracts features through 8 consecutive MBConv modules. Each MBConv module contains depthwise separable convolution, squeeze-and-excitation modules and jump connections, which effectively reduce the number of parameters while maintaining a high feature extraction capability. The expansion ratio of the MBConv modules in the network ranges from 1 to 6, the convolution kernel sizes are 3×3 and 5×5, and the number of channels is gradually expanded from 32 to 384. In order to adapt to the characteristics of evidence photos, a spatial pyramid pooling (SPP) module is added to the standard EfficientNet, using three pooling scales of 1×1, 2×2, and 4×4 to enhance the network's perception of features of different scales. Finally, the features are compressed into a 1536-dimensional vector through a global average pooling layer, and then mapped to the coarse-grained category space (such as ) through a fully connected layer containing Dropout (ratio of 0.3). Figure 2 The classification loss function uses cross entropy with label smoothing.

[0089] The residual connection fusion module combines feature enhancement and category embedding to deeply fuse coarse-grained classification results with the original key information features. First, the category probability vector p∈R^6 (corresponding to six coarse-grained categories) output by the coarse-grained classification network is mapped into a high-dimensional feature representation through a category embedding layer. Category information is then injected into the original key information features through a feature modulation mechanism. Specifically, spatially adaptive instance normalization is performed on the original key information features F_key∈R^(H×W×C). To maintain the integrity of the original features, a gated residual connection is introduced. Finally, the fused features are channel-adjusted and normalized through a 1×1 convolutional layer and a batch normalization layer to obtain the final fused features. This fusion approach effectively incorporates coarse-grained classification information into the feature representation while preserving the original key information features, enhancing the discriminability of the features and providing richer contextual information for subsequent fine-grained classification.

[0090] The fine-grained classification network is based on a dual-path attention enhancement architecture and is specifically designed to distinguish fine-grained subcategories under the same coarse-grained category. The network consists of two parts: a backbone feature extraction path and an attention enhancement path. The backbone path adopts the ResNet-50 architecture, but replaces the standard convolution with a deformable convolution to enhance adaptability to fine-grained features. Deformable convolution enables the convolution kernel to dynamically adjust the sampling position according to the input features by learning additional offset parameters. The attention enhancement path contains three parallel attention modules: a channel attention module (SE module), a spatial attention module (non-local module), and a category activation map module (CAM). The three attention maps are adaptively fused to obtain a comprehensive attention map:. Finally, the attention-enhanced features are fused with the backbone features in a residual manner. A two-layer classifier structure is used at the end of the network. The first-layer classifier selects the corresponding second-layer classifier according to the coarse-grained category, such as Figure 2 As shown in the figure, for the "ID Photo" category, the second-layer classifier further distinguishes fine-grained categories such as "ID Card," "Driver's License," and "Vehicle Registration Certificate." This hierarchical classification strategy, combined with multiple attention mechanisms, enables the network to focus on key regional features that distinguish fine-grained categories.

[0091] The classification results are integrated using a hierarchical probability fusion approach combined with confidence calibration to construct a structured initial set of photo classifications. First, the second-level classification results for each evidence photo are post-processed, including softmax temperature scaling to calibrate the confidence. Then, hierarchical classification probabilities are calculated. The final probability of a fine-grained category is the product of the coarse-grained probability and the conditional probability. For example, the final probability of "ID card" is equal to the probability of "ID card photo" multiplied by the probability of "ID card" conditional on "ID card photo." Next, a structured classification result representation is generated for each photo, consisting of the following fields: photo ID, coarse-grained category, coarse-grained confidence, fine-grained category, fine-grained confidence, and a hierarchical path (e.g., "ID card photo / ID card"). To address classification uncertainty, when the fine-grained confidence falls below a threshold τ, the top k (k = 3) candidate categories and their confidences are retained. Finally, based on the semantic connections between photos, a relationship graph for the photo classification set is constructed. This graph structure represents the categorical relationships between photos, such as the strong association between "breath photos" and "breath alcohol content test records" in the same case. This structured classification set representation not only includes the classification information for each photo, but also preserves the uncertainty of the classification and the semantic connections between photos.

[0092] In one embodiment, analyzing the correlation between different evidence photos in the initial photo classification set by using an evidence correlation analysis mechanism, and optimizing the initial photo classification set into a target photo classification set based on the correlation analysis results includes the following steps:

[0093] Extract metadata information from the evidence photos in the initial photo classification set, the metadata information including the timestamp and geographic location;

[0094] Combining metadata information and key information features, the feature correlation between different evidence photos in the initial photo classification set is calculated;

[0095] Constructing an evidence photo association graph of the initial photo classification set based on feature association;

[0096] The initial photo classification set is optimized into the target photo classification set using the evidence photo association graph and the preset evidence logic relationship rule library.

[0097] In this implementation, raw metadata is first obtained by parsing the photo's EXIF data, including capture time (DateTime tag), GPS coordinates (GPSLatitude and GPSLongitude tags), and device information (Make and Model tags). Geographical location information is then acquired through a combination of scene recognition and landmark detection. The scene recognition network, based on the DenseNet-161 architecture and pre-trained on the Places365 dataset, can identify indoor / outdoor, urban / rural scene types. The landmark detection network specifically detects and recognizes geographically significant elements such as traffic signs, street signs, and buildings. The reliability of the time and location information extracted from the image content is assessed using a confidence assessment mechanism. Information with a confidence score below a threshold is marked as "uncertain." Finally, the metadata is stored in a structured format, including a timestamp (accurate to the second), geographic coordinates (latitude and longitude), a location description, a time reliability score, and a location reliability score.

[0098] Next, a multi-dimensional fusion approach is used to comprehensively consider metadata similarity and content semantic relevance to quantify the degree of association between different evidence photos. First, metadata similarity is calculated. Temporal similarity uses a Gaussian decay function based on time intervals; geographic location similarity is calculated based on spherical distance. Content semantic relevance is calculated at three levels: visual feature similarity uses the cosine similarity of key information features; category relevance is based on a predefined category association matrix. For example, the category relevance between "breath photo" and "breath alcohol test record" is 0.9; and text content relevance is calculated by comparing the semantic similarity of text extracted from the photos and calculating the cosine similarity between text embedding vectors using the BERT model. The final feature relevance is obtained through weighted fusion. For example, in a drunk driving case, a "breath photo" will have a high correlation with "scene photos" taken at the same time and location, but a low correlation with "ID photos" taken at different times and locations. This module accurately captures the multi-dimensional associations between evidence photos, laying a solid foundation for constructing an evidence photo association graph.

[0099] Next, the feature correlation is converted into a structured graph representation to reveal the inherent connections between evidence photos. First, each evidence photo is represented as a node in the graph, with node attributes including photo ID, classification result, metadata, and key information features. Next, edges are connected between nodes based on feature correlation, and an adaptive threshold strategy is used to determine the existence of edges: when the feature correlation R(i, j) between two photos exceeds a threshold τ, an edge is established between the corresponding nodes, with the edge weight equal to the correlation value. The threshold τ is dynamically determined through cluster analysis. Specifically, the DBSCAN algorithm is used to cluster the correlations of all photo pairs, with the lower bound of the highest-density cluster being used as the threshold. To enhance the graph's expressiveness, edge types are labeled based on dominant correlation factors, such as "temporal correlation," "geographic correlation," and "content correlation." Furthermore, a community detection algorithm (Louvain method) is introduced to partition the correlation graph into communities, identifying subsets of closely related photos. Each community may correspond to a chain of evidence or event in the case. For example, in a traffic accident case, relevant evidence such as scene photos, vehicle damage photos, and accident liability determination documents form a closely connected subgraph. The association graph also supports multi-scale representation, allowing you to adjust the displayed association strength level as needed. The evidence photo association graph constructed by this module can intuitively display the logical relationship between evidence.

[0100] Next, the initial classification results are corrected and improved by combining graph reasoning and rule-based reasoning. First, a rule base for evidence logic relationships is constructed, consisting of three types of rules: category constraint rules (e.g., "A case should include at least one on-site photo"), temporal logic rules (e.g., "Blood photos should be taken later than breath photos"), and association consistency rules (e.g., "ID photos within the same community should belong to the same person"). These rules are expressed in predicate logic, supporting reasoning under uncertainty. A graph convolutional network (GCN) is then used to perform message passing on the evidence photo association graph to update the node category representation. After three layers of GCN, the node representation incorporates information from neighboring nodes, helping to correct isolated misclassifications. Next, a Markov logic network (MLN) is applied to the rule base for reasoning, calculating the posterior probability of each possible classification. Finally, the optimized classification results are obtained by combining the GCN output with the MLN inference results. For example, in a drunk driving case, if a blurred photo is initially classified as a "breathalyzer photo" with low confidence, but the association graph and rule-based reasoning indicate it's more likely a "breathalyzer test record," its classification will be adjusted to the latter. This optimization process significantly improves classification accuracy and ensures that the classification results align with the logical relationships of the evidence, making the target photo classification set more reliable and consistent, providing a high-quality evidence organization structure for subsequent case processing.

[0101] In one embodiment, checking the integrity of the target photo classification set according to the evidence specification requirements preset for the case type includes the following steps:

[0102] Define the necessary evidence types based on the evidence standards set by the case type;

[0103] Perform statistical analysis on the classified evidence photos in the target photo classification set to identify the types of evidence collected in the current case;

[0104] Compare the collected evidence types with all necessary evidence types to detect the completeness of the target photo classification set.

[0105] In this implementation, a mapping relationship between case types and required evidence types is first constructed based on a multi-level knowledge graph. This knowledge graph comprises three levels: the top level represents broad case categories (e.g., traffic violations, criminal cases), the middle level represents specific case types (e.g., drunk driving, speeding, theft), and the bottom level represents evidence type nodes. Each case type node is connected to the corresponding evidence type node via three types of edges: "required evidence," "supporting evidence," and "optional evidence." For example, for a "drunk driving" case, required evidence includes "breath photograph," "breath alcohol content test record," "ID card," and "driver's license"; supporting evidence includes "scene photos" and "body camera video"; and optional evidence includes "blood sample photos" and "blood sample test report." Each evidence type node in the knowledge graph also contains attribute information such as evidence description, legal basis, and collection requirements.

[0106] Next, a comprehensive analysis of the target photo classification set is conducted using a method combining multi-dimensional aggregation and quality assessment. First, based on the fine-grained classification results, the photos are clustered by category, and the number, temporal distribution, and quality distribution of photos for each evidence type are counted. The aggregation process takes classification confidence into account. If a photo's classification confidence falls below a threshold, it is marked as "pending" and counted separately. Next, a quality assessment is performed on each evidence type, including a clarity score (based on Laplace gradient variance), an exposure score (based on brightness histogram analysis), and a completeness score (based on edge detection and content analysis). Quality scores are standardized on a scale of 0-100, with a passing score of 60. Next, duplicate photos of the same type are analyzed, and photo similarity is calculated using a perceptual hashing algorithm. Photos with a similarity exceeding 95% are marked as duplicates. Finally, a structured evidence statistics report is generated, including a list of collected evidence types, a count of each type's quantity, quality distribution, temporal distribution, and anomalies (such as substandard quality or unusual quantity). Taking drunk driving cases as an example, the statistical report may show: 3 "breath photos" (average quality 85 points), 1 "ID card" (quality 76 points), 1 "driver's license" (quality 92 points), 5 "scene photos" (average quality 79 points), etc. have been collected.

[0107] Next, a comprehensive assessment of the completeness of evidence collection is conducted through a combination of semantic matching and rule-based reasoning. First, a semantic matching matrix for evidence types is constructed to address inconsistent evidence naming. This matrix, based on word vector similarity and a predefined synonym table, calculates the degree of match between collected and required evidence types. Evidence types exceeding a threshold (typically 0.85) are considered equivalent. Next, a three-level completeness check is performed: the first level checks the coverage of required evidence, calculated as "number of required evidence collected / total number of required evidence"; the second level checks the evidence quality acceptance rate, calculated as "number of evidence of acceptable quality / number of evidence collected"; and the third level checks whether the evidence combination logic is met, such as complex rules like "choose one of two" and "at least N pieces." A structured completeness check report is then generated, including an overall completeness score (0-100), a list of missing evidence, a list of evidence of unacceptable quality, and specific improvement recommendations. Finally, based on the completeness check results, intelligent evidence collection guidance is provided, such as "recommendation for supplementary photos of the Breath Alcohol Content Record Form" and "poor quality of on-site photos, recommendation for re-shooting." Taking drunk driving cases as an example, if the Breath Alcohol Content Test Record Form is missing, the system will mark it as seriously missing and give the highest priority supplementary suggestions.

[0108] In one embodiment, the method further comprises the steps of:

[0109] Analyze the computational complexity and storage requirements of evidence photo classification models;

[0110] Based on computational complexity, the model knowledge of the evidence photo classification model is transferred to a preset lightweight classification model through knowledge distillation technology;

[0111] The model pruning method is used to remove network connections in the lightweight classification model whose contribution to the classification results is less than a preset threshold;

[0112] Use quantization technology to reduce the accuracy requirements of model parameters in lightweight classification models, and use model compression technology to reduce the model size until the storage requirements of lightweight classification models meet the deployment requirements of mobile devices;

[0113] Deploy lightweight models on mobile devices.

[0114] In this implementation, we first calculated theoretical complexity, including the number of floating-point operations (FLOPs) and parameter count. Taking the currently deployed ResNet-101 model as an example, it contains approximately 43 million parameters and requires approximately 7.8 billion floating-point operations per inference. Next, we conducted real-world runtime tests, measuring the model's average inference time and memory usage on different hardware platforms (server GPUs, PC CPUs, and mobile processors). Regarding storage requirements, the original model file size was 174MB, and the peak memory usage during loading reached 512MB, exceeding the ideal range for mobile devices. Furthermore, energy consumption analysis revealed that running this model on mobile devices results in rapid battery drain; processing 100 photos continuously consumes approximately 5% of the battery. Based on this analysis, we established lightweighting targets: inference time must be reduced to less than 200 milliseconds, model size must be compressed to less than 20MB, and memory usage must be kept below 100MB, while maintaining a maximum reduction in classification accuracy of 2%.

[0115] MobileNetV3 was chosen as the student model architecture. It has only 2.7 million parameters and a theoretical computational load of 220 million FLOPs, approximately 16 times smaller than the original ResNet-101 model. A distillation loss function was then constructed, simultaneously considering both soft label matching and hard label accuracy. The soft labels were adjusted using a temperature parameter T (set to 4) to adjust the probability distribution of the teacher model's output, enhancing similarity between categories. The hard labels were directly annotated with the ground-truth categories. The weight ratio of the soft label loss to the hard label loss in the loss function was set to 0.7:0.3 to balance knowledge transfer and classification accuracy. A two-stage training strategy was employed: the first stage used only hard labels for pre-training, while the second stage introduced soft labels for knowledge distillation. To enhance the distillation effect, feature-level distillation was also introduced, matching corresponding intermediate-layer features from the teacher and student models. Experiments on the evidence photo dataset showed that the distilled MobileNetV3 model achieved a classification accuracy of 91.8%, only 1.2% lower than the original model, while increasing inference speed by approximately 7 times, initially meeting the performance requirements for mobile deployment.

[0116] An importance assessment mechanism is introduced, calculating the L1 norm of each convolutional filter as its importance metric. Combined with statistical analysis of feature map activations, this comprehensively evaluates the filter's contribution to the final classification result. An adaptive pruning threshold is then set, with varying pruning percentages for each layer based on sensitivity analysis results. Shallow networks retain more parameters (pruning rate of approximately 10-15%), while deeper networks prune at a higher rate (pruning rate of approximately 30-40%). An iterative pruning process is then performed: each pruning step is followed by a short-term fine-tuning (5 epochs), followed by further evaluation and pruning, and repeated for 3-5 rounds until the target model size is reached. To maintain model structural consistency, channel-level pruning is employed, completely removing entire convolutional filters and their corresponding batch normalization layer parameters. On the evidence photo classification task, pruning reduced the number of parameters in the MobileNetV3 model from 2.7 million to 1.8 million, reducing computational overhead by approximately 35%, while only decreasing classification accuracy by 0.5%. Especially when processing simple evidence types (such as standardized documents such as ID cards and driver's licenses), the pruned model performs almost as well as the original model.

[0117] Model parameters are converted from 32-bit floating-point (FP32) to 8-bit integer (INT8) representation. The quantization process uses a calibration-assisted dynamic range determination method. By analyzing the distribution characteristics of parameters and activation values at each layer, optimal quantization parameters are set for each layer. For critical layers sensitive to classification accuracy (such as the last few convolutional layers), 16-bit floating-point (FP16) precision is retained to minimize precision loss. Next, Huffman coding is applied to losslessly compress the quantized weights. Leveraging the statistical characteristics of the weight distribution, high-frequency values are assigned short codes, further reducing storage space. Model structure optimizations are then performed, including operator fusion (combining consecutive convolutions, batch normalization, and activation functions into a single operation) and memory layout optimization (reducing the storage of intermediate results). Finally, the final model is packaged in a mobile-optimized model format, such as TFLite or ONNX. After quantization and compression, the model size was reduced from the original 45MB (after pruning) to 12MB, fully meeting mobile storage requirements. Memory usage was reduced to 85MB, and single-shot inference time was reduced to 120 milliseconds on mid-range phones (such as the Snapdragon 765 processor), while classification accuracy only decreased by 0.3% (compared to the pruned model). The quantized model also significantly improved energy consumption, consuming approximately 60% less power to process the same number of photos, which is particularly important for mobile devices with limited battery capacity.

[0118] The present invention also discloses a system for intelligent classification and consolidation of evidence photos based on deep learning, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the method for intelligent classification and consolidation of evidence photos based on deep learning as described in any of the above embodiments.

[0119] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0120] Among them, the memory can be an internal storage unit of a computer device, such as a hard disk or memory of a computer device, or an external storage device of a computer device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD) or flash memory card (FC) equipped on the computer device. In addition, the memory can also be a combination of an internal storage unit and an external storage device of a computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or is to be output. This application does not impose any restrictions on this.

[0121] The present invention also discloses a computer-readable storage medium, which stores instructions. When the instructions are executed by a processor, the processor is configured to execute the deep learning-based evidence photo intelligent classification and verification method described in any of the above embodiments.

[0122] Among them, the computer program can be stored in a machine-readable medium, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or certain middleware, etc. The machine-readable medium includes any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the machine-readable medium includes but is not limited to the above-mentioned components.

[0123] Among them, through this computer-readable storage medium, the deep learning-based evidence photo intelligent classification and evidence consolidation method in the above embodiment is stored in the computer-readable storage medium, and is loaded and executed on the processor to facilitate the storage and application of the above method.

[0124] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as above, which are not provided in detail for the sake of simplicity.

[0125] The one or more embodiments of this application are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this application should be included in the scope of protection of this application.

Claims

1. A method for intelligent classification and consolidation of evidence photos based on deep learning, characterized by: The steps include: Obtain evidence photos to be classified and pre-process them according to the case type. Pre-processing parameters will be adjusted for different case types. Build an evidence photo classification model based on a deep learning model, which includes a multi-level classification framework and feature enhancement modules; The pre-processed evidence photos are input into the evidence photo classification model, and the key information features in the evidence photos are extracted through the feature enhancement module; A multi-level classification framework is used to perform coarse-grained and fine-grained classification of evidence photos based on key information features to obtain an initial photo classification set. Analyze the correlation between different evidence photos in the initial photo classification set through the evidence correlation analysis mechanism, and optimize the initial photo classification set into the target photo classification set based on the correlation analysis results; The integrity of the target photo classification set is checked according to the evidence specifications preset by the case type. If the integrity verification passes, the target photo classification set is output as the solid evidence dataset; The construction of an evidence photo classification model based on a deep learning model, which includes a multi-level classification framework and a feature enhancement module, includes the following steps: The pre-trained YOLOn8 model is used as the base model, and a feature enhancement module is embedded in the base model. The feature enhancement module includes a text recognition unit, an object detection unit, and a feature fusion unit. Construct a multi-level classification framework in the basic model, which includes a coarse-grained classification network and a fine-grained classification network; Establish a residual connection between the coarse-grained classification network and the fine-grained classification network, and introduce an attention mechanism into the fine-grained classification network to construct an initial evidence photo classification model; Initialize the model parameters of the initial evidence photo classification model, use the labeled evidence photo dataset and complete the training and optimization of the initial evidence photo classification model through transfer learning method to obtain the evidence photo classification model; Extracting key information features from evidence photos using the feature enhancement module includes the following steps: Extract text information from evidence photos using optical character recognition technology in the text recognition unit; Utilize the preset target detection algorithm in the target detection unit to identify and extract key area features in the evidence photo; The text information and key area features are fused into enhanced feature representation through the feature fusion unit; Fuse the enhanced feature representation with the original evidence photo to form key information features; The text recognition unit uses an improved convolutional recurrent neural network architecture combined with the Transformer attention mechanism to extract text information from evidence photos. The object detection unit is based on an improved Cascade R-CNN architecture. It accurately locates and extracts key areas in evidence photos, identifies key areas in evidence photos, and extracts key features from evidence photos. It uses ResNeXt-101 as the backbone network and combines it with FPN to achieve multi-scale feature extraction. The feature fusion unit adopts a multimodal attention fusion mechanism to deeply fuse the text features extracted by the text recognition unit and the key area features extracted by the object detection unit; Among them, the feature fusion unit adopts a multimodal attention fusion mechanism to perform feature transformation on text features and distinguishing features respectively so that they have the same dimension; a bidirectional cross-attention mechanism is used to calculate the correlation between the two features, and a cross-attention feature is generated based on the attention weight; Adaptively fuse original features and cross-attention features through a gating mechanism; generate the final enhanced feature representation through multi-layer perceptrons and residual connections; The process of fusing the enhanced feature representation with the original evidence photo adopts feature map reconstruction and injection technology. The multi-level feature map of the original evidence photo is extracted through a pre-trained ResNet-50 network, including low-level, mid-level and high-level features, which capture edge, texture and semantic information respectively. The enhanced feature representation is reconstructed into a feature map with the same spatial dimension as the original feature map through a deconvolution network. A feature injection module is designed, and the spatial adaptive instance normalization technology is used to inject the reconstructed feature map into the multi-level feature map of the original evidence photo. The fusion features at different levels are integrated through FPN to generate the final key information features.

2. The method for intelligent classification and consolidation of evidence photos based on deep learning according to claim 1 is characterized in that: The preprocessing steps include image scaling, illumination equalization and noise removal.

3. The method for intelligent classification and consolidation of evidence photos based on deep learning according to claim 1 is characterized in that: Using a multi-level classification framework to perform coarse-grained and fine-grained classification of evidence photos based on key information features, the initial photo classification set is obtained, which includes the following steps: The key information features are coarsely classified through the coarse-grained classification network to obtain the first-level classification results; The first-level classification results are fused with the original key information features through residual connections to form fusion features; The fused features are input into the fine-grained classification network for fine-grained classification. During the fine-grained classification process, the attention mechanism is used to enhance the perception ability of the fine-grained classification network on fine-grained category features, and the second-level classification results are obtained; The second-level classification results of the evidence photos and the corresponding confidence levels are integrated to obtain the initial photo classification set.

4. The method for intelligent classification and consolidation of evidence photos based on deep learning according to claim 3 is characterized in that: Analyzing the correlation between different evidence photos in the initial photo classification set through the evidence correlation analysis mechanism, and optimizing the initial photo classification set into the target photo classification set based on the correlation analysis results includes the following steps: Extract metadata information from the evidence photos in the initial photo classification set, the metadata information including the timestamp and geographic location; Combining metadata information and key information features, the feature correlation between different evidence photos in the initial photo classification set is calculated; Constructing an evidence photo association graph of the initial photo classification set based on feature association; The initial photo classification set is optimized into the target photo classification set using the evidence photo association graph and the preset evidence logic relationship rule library.

5. The method for intelligent classification and consolidation of evidence photos based on deep learning according to claim 1 is characterized in that: Testing the integrity of the target photo classification set based on the evidence standards preset for the case type includes the following steps: Define the necessary evidence types based on the evidence standards set by the case type; Perform statistical analysis on the classified evidence photos in the target photo classification set to identify the types of evidence collected in the current case; Compare the collected evidence types with all necessary evidence types to detect the completeness of the target photo classification set.

6. The method for intelligent classification and consolidation of evidence photos based on deep learning according to claim 1 is characterized in that: The method further comprises the steps of: Analyze the computational complexity and storage requirements of evidence photo classification models; Based on computational complexity, the model knowledge of the evidence photo classification model is transferred to a preset lightweight classification model through knowledge distillation technology; The model pruning method is used to remove network connections in the lightweight classification model whose contribution to the classification results is less than a preset threshold; Use quantization technology to reduce the accuracy requirements of model parameters in lightweight classification models, and use model compression technology to reduce the model size until the storage requirements of lightweight classification models meet the deployment requirements of mobile devices; Deploy lightweight models on mobile devices.

7. A system for intelligent classification and consolidation of evidence photos based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the method for intelligent classification and consolidation of evidence photos based on deep learning as described in any one of claims 1 to 6.

8. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by the processor, the processor is configured to execute the method for intelligent classification and verification of evidence photos based on deep learning according to any one of claims 1 to 6.

Images