A multi-modal information intelligent analysis method, device, equipment and storage medium

Abstract

The present application relates to a kind of multimodal information intelligent analysis method, device, equipment and storage medium.The method comprises: based on YOLOv11 architecture construction target detection model, target detection model is integrated with light weight modal classification head, for simultaneously executing target detection and modal identification, output target detection result and corresponding modal label.Infrared image data is executed physical enhancement processing, and enhanced infrared image data and other modal data are input to target detection model, the multi-layer feature of target detection model is extracted, and auxiliary feature is extracted from original multimodal data, feature fusion is generated by attention mechanism, and fusion feature vector is generated.Input to large language model, in combination with preset analysis prompt word, generate structured analysis result.According to the analysis result, according to preset report template, information analysis report is automatically generated.The method can improve multimodal information processing efficiency and infrared modal adaptability.

Description

Technical Field

[0001] This invention relates to the field of multimodal information processing technology, and in particular to a method, apparatus, device, and storage medium for intelligent analysis of multimodal information. Background Technology

[0002] In the field of smart security analysis, especially in urban security scenarios, perceived data exists in multimodal forms such as images, videos, and infrared images. It includes information on various key targets such as crowds, non-motorized vehicles, open flames, and smoke. Rapid and accurate target identification, correlation analysis, and situational assessment are required to provide technical support for security patrols. Current data analysis methods mainly rely on manual processing, which has the following technical shortcomings: 1. Inefficient: Manual processing of multimodal intelligence data takes a lot of time. On average, target recognition for a single image takes about 15-20 seconds, and target analysis for a single 3-minute video segment takes about 30 minutes. This makes it difficult to meet the real-time requirements in scenarios such as peak passenger flow and to respond quickly to decision-making needs. 2. Insufficient multimodal fusion: Existing technologies mostly analyze single types of data, such as only performing target recognition on images or only performing time-series analysis on videos. They do not fully utilize the feature information of multimodal data such as images, videos, and infrared data, resulting in insufficient accuracy and one-sided analysis results. 3. Low level of intelligence: There is a lack of effective intelligent analysis models. Target detection, situation assessment and other processes rely on human experience, which is prone to misjudgment and omission. The misjudgment rate is about 12-15%. Moreover, it cannot automatically generate structured inspection reports, and the results need to be manually sorted and analyzed, which increases the cost of manual sorting. 4. Poor generalization: Existing target detection models are not adaptable to complex scenarios (such as severe weather, target occlusion, low light, and infrared imaging). The detection accuracy for dense targets is difficult to meet actual needs. For example, in rainy or foggy weather, the target detection accuracy will drop by more than 30%. 5. Engineering defects: Existing systems are mostly designed with a single interface and lack cross-platform support. In Windows environment, they are prone to problems such as DLL loading conflicts, multi-threading instability, and Python version compatibility, making it difficult to deploy applications in practice.

[0003] 6. Weak modality adaptation: Existing target detection models use uniform detection parameters for images of different modalities (visible light, infrared), without considering the differences in characteristics of images of different modalities, resulting in a significant decrease in detection accuracy under special modalities such as infrared.

[0004] YOLOv11, a commonly used target detection model in engineering applications and academia, boasts higher detection accuracy, faster inference speed, and stronger small target detection capabilities compared to existing models, effectively adapting to target recognition needs in complex scenarios. Larger models possess powerful semantic understanding and deep analysis capabilities, enabling the fusion and intelligent judgment of multimodal features. Based on this, by integrating innovative technologies such as infrared enhancement, staged training, and a dual-frontend architecture, a multimodal intelligence intelligent analysis method and system are designed to address the shortcomings of existing technologies, meeting an urgent need in the current field of multimodal information analysis. Summary of the Invention

[0005] Therefore, it is necessary to address the aforementioned technical problems by providing a multimodal information intelligent analysis method, device, equipment, and storage medium that can solve the problems of low efficiency in multimodal information processing, poor infrared modal adaptability, and insufficient intelligence, and achieve efficient, accurate, modally adaptive end-to-end intelligent analysis and automatic report generation.

[0006] A multimodal information intelligent analysis method, the method comprising: Acquire multimodal information data, which includes image data, video data, and infrared image data.

[0007] An object detection model is built based on the YOLOv11 architecture. The object detection model integrates a lightweight modality classification head to perform object detection and modality recognition simultaneously, and outputs object detection results and corresponding modality labels.

[0008] Physical enhancement processing is performed on the infrared image data. The enhanced infrared image data and other modal data are input into the target detection model. Multi-layer features of the target detection model are extracted and combined with auxiliary features extracted from the original multi-modal data. Feature fusion is performed through an attention mechanism to generate a fused feature vector.

[0009] The fused feature vectors are input into a large language model, and combined with preset analysis prompts, structured analysis results are generated.

[0010] Based on the structured analysis results, an information analysis report is automatically generated according to a preset report template.

[0011] A multimodal information intelligent analysis device, the device comprising: The data acquisition module is used to acquire multimodal information data, including image data, video data, and infrared image data.

[0012] The detection model building module is used to build an object detection model based on the YOLOv11 architecture. The object detection model integrates a lightweight modality classification head to perform object detection and modality recognition simultaneously, and outputs the object detection results and the corresponding modality labels.

[0013] The feature fusion module is used to perform physical enhancement processing on infrared image data. The enhanced infrared image data and other modal data are input into the target detection model to extract multi-layer features of the target detection model. Combined with auxiliary features extracted from the original multi-modal data, feature fusion is performed through an attention mechanism to generate a fused feature vector.

[0014] The intelligent analysis module is used to input the fused feature vector into the large language model and generate structured analysis results by combining it with preset analysis prompts.

[0015] The report generation module is used to automatically generate information analysis reports based on structured analysis results and according to preset report templates.

[0016] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Acquire multimodal information data, which includes image data, video data, and infrared image data.

[0017] An object detection model is built based on the YOLOv11 architecture. The object detection model integrates a lightweight modality classification head to perform object detection and modality recognition simultaneously, and outputs object detection results and corresponding modality labels.

[0018] Physical enhancement processing is performed on the infrared image data. The enhanced infrared image data and other modal data are input into the target detection model. Multi-layer features of the target detection model are extracted and combined with auxiliary features extracted from the original multi-modal data. Feature fusion is performed through an attention mechanism to generate a fused feature vector.

[0019] The fused feature vectors are input into a large language model, and combined with preset analysis prompts, structured analysis results are generated.

[0020] Based on the structured analysis results, an information analysis report is automatically generated according to a preset report template.

[0021] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire multimodal information data, which includes image data, video data, and infrared image data.

[0022] An object detection model is built based on the YOLOv11 architecture. The object detection model integrates a lightweight modality classification head to perform object detection and modality recognition simultaneously, and outputs object detection results and corresponding modality labels.

[0023] Physical enhancement processing is performed on the infrared image data. The enhanced infrared image data and other modal data are input into the target detection model. Multi-layer features of the target detection model are extracted and combined with auxiliary features extracted from the original multi-modal data. Feature fusion is performed through an attention mechanism to generate a fused feature vector.

[0024] The fused feature vectors are input into a large language model, and combined with preset analysis prompts, structured analysis results are generated.

[0025] Based on the structured analysis results, an information analysis report is automatically generated according to a preset report template.

[0026] The aforementioned multimodal information intelligent analysis method, device, equipment, and storage medium significantly improve information processing efficiency and real-time performance by integrating multimodal data acquisition, infrared physical enhancement, YOLOv11 joint detection and recognition with an integrated lightweight modality classification head, attention fusion of multi-layer features and auxiliary features, structured analysis of a large language model, and report generation into an end-to-end automated process. This enables millisecond-level response for single-image recognition and allows for the completion of long video analysis in a short time. The fourth-order physical enhancement (normalization, nonlinear compression, filtering and denoising, and edge enhancement) designed for infrared images effectively improves the problems of low contrast and blurred contours. Combined with the modality labels output by the model, modality adaptive detection is achieved, resulting in a significant improvement in target detection accuracy under infrared modalities. The overall average detection accuracy is high, and the false positive rate is significantly reduced. By extracting auxiliary features such as image HSV color and video temporal sequence and performing attention-weighted fusion with deep learning features, the information deficiency of single features in complex environments is compensated for. The accuracy decrease is minimal under adverse conditions such as rain and low light, demonstrating strong generalization and robustness. Meanwhile, this method utilizes preset analysis prompts to guide the large language model to output structured results such as target association and situation assessment, which are then automatically populated into the report template. This achieves full-process intelligence and automation from raw data to analysis reports, significantly reducing labor costs. The lightweight modal classification head has extremely low computational increment, ensuring efficient system deployment and engineering practicality, and can be flexibly adapted to various scenarios such as security monitoring, community patrols, and public safety. Attached Figure Description

[0027] Figure 1 This is a diagram illustrating the application environment of a multimodal information intelligent analysis method in one embodiment. Figure 2 This is a flowchart illustrating a multimodal information intelligent analysis method in one embodiment; Figure 3 This is a schematic diagram of a YOLOv11 training and detection process in one embodiment; Figure 4 This is a schematic diagram of a large model invocation process in one embodiment; Figure 5 This is a structural block diagram of a multimodal information intelligent analysis device in one embodiment; Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0029] This invention provides a multimodal information intelligent analysis method that can be applied to, for example... Figure 1 The application environment shown is a multimodal information intelligent analysis system framework, which adopts a five-stage pipeline architecture: data preprocessing, target detection, feature fusion, intelligent analysis, and report generation. First, the data preprocessing module receives image, video, and infrared image data from different acquisition terminals. For infrared images, this module incorporates a fourth-order physical enhancement algorithm (normalization, nonlinear compression, filtering and denoising, edge enhancement) to improve the quality of low-contrast infrared images; for visible light images and videos, resolution normalization and frame rate unification are performed. The preprocessed data is then sent to the target detection module in batches.

[0030] The object detection module is based on the YOLOv11 architecture and incorporates a lightweight modality classification head after the high-level features of its backbone network. This module simultaneously performs object bounding box detection and modality recognition (visible light / infrared) on each frame / image, outputting the object's category, location, confidence score, and corresponding modality label. The detected object feature vectors (e.g., 1024-dimensional) are extracted for subsequent fusion.

[0031] Simultaneously, the feature fusion module extracts auxiliary features from the original multimodal data: HSV color histogram features are extracted from the original image data; temporal motion features are extracted from continuous video frames using a 3D convolutional neural network; and thermal radiation intensity distribution features are extracted from the enhanced infrared image. These auxiliary features, after being normalized and aligned with the aforementioned deep target features, are then weighted and fused using an attention mechanism to generate a unified fusion feature vector. This fusion vector integrates multi-dimensional information such as semantics, color, motion, and thermal radiation.

[0032] The intelligent analysis module receives the fused feature vector and inputs it into a locally deployed large language model (such as Deepseek-r1:8b). This module has pre-set professional prompt word templates for information analysis (covering dimensions such as target association analysis, situation assessment, and risk assessment). Based on these, the large language model outputs structured analysis results, such as target association graphs, overall situation descriptions, and lists of high-attention targets.

[0033] Finally, the report generation module automatically populates the structured analysis results into the corresponding sections (including information summaries, correlation analysis, situation conclusions, risk assessments, etc.) according to a preset standard report template, and can export them as PDF, Word, or JSON formats, completing end-to-end automated processing from raw multimodal data to the final analysis report. The modules are connected through clear data interfaces, forming a unidirectional data flow, ensuring efficient system collaboration and the feasibility of engineering deployment.

[0034] In one embodiment, such as Figure 2 As shown, a multimodal information intelligent analysis method is provided, which is applied to... Figure 1 Taking the system architecture in [the document] as an example, the following steps are included: Step 202: Obtain multimodal information data.

[0035] Multimodal information data includes image data, video data, and infrared image data.

[0036] Step 204: Construct an object detection model based on the YOLOv11 architecture. The object detection model integrates a lightweight modality classification head, which is used to perform object detection and modality recognition simultaneously, and output the object detection results and the corresponding modality labels.

[0037] Step 206: Perform physical enhancement processing on the infrared image data, input the enhanced infrared image data and other modal data into the target detection model, extract multi-layer features of the target detection model, and combine them with auxiliary features extracted from the original multi-modal data. Then, perform feature fusion through an attention mechanism to generate a fused feature vector.

[0038] Step 208: Input the fused feature vector into the large language model, and combine it with the preset analysis prompts to generate structured analysis results.

[0039] Step 210: Based on the structured analysis results, automatically generate an information analysis report according to the preset report template.

[0040] The aforementioned intelligent multimodal information analysis method integrates multimodal data acquisition, infrared physical enhancement, YOLOv11 joint detection and recognition with a lightweight modality classification head, attention fusion of multi-layer features and auxiliary features, structured analysis of a large language model, and report generation into an end-to-end automated process. This significantly improves information processing efficiency and real-time performance, enabling millisecond-level response for single image recognition and allowing long video analysis to be completed in a short time. The fourth-order physical enhancement (normalization, nonlinear compression, filtering and denoising, and edge enhancement) designed for infrared images effectively improves low contrast and blurry contours. Combined with the modality labels output by the model, modality adaptive detection is achieved, resulting in a significant improvement in target detection accuracy under infrared modalities. The overall average detection accuracy is high, and the false positive rate is significantly reduced. By extracting auxiliary features such as image HSV color and video temporal sequence and performing attention-weighted fusion with deep learning features, the method compensates for the lack of information from single features in complex environments. Accuracy degradation is minimal under adverse conditions such as rain and low light, demonstrating strong generalization and robustness. Meanwhile, this method utilizes preset analysis prompts to guide the large language model to output structured results such as target association and situation assessment, which are then automatically populated into the report template. This achieves full-process intelligence and automation from raw data to analysis reports, significantly reducing labor costs. The lightweight modal classification head has extremely low computational increment, ensuring efficient system deployment and engineering practicality, and can be flexibly adapted to various scenarios such as security monitoring, community patrols, and public safety.

[0041] In one embodiment, a linear normalization method is used to map the pixel values ​​of the infrared image data to a preset numerical range to obtain normalized image data, thereby eliminating brightness differences. ; in, For input pixel values, and These are the minimum and maximum pixel values, respectively. This is a very small constant. A nonlinear transformation using the hyperbolic tangent function is applied to the normalized image data to obtain compressed image data, thereby enhancing details in low-brightness areas. ; in, To adaptively adjust the compression intensity parameters, guided filtering or bilateral filtering is applied to the compressed image data to preserve edge features and remove noise. Gradients are calculated on the filtered image data, and edge enhancement is performed using a preset convolution kernel to highlight the thermal radiation characteristics of the target.

[0042] In one embodiment, a lightweight modality classification head is connected after the high-level features of the YOLOv11 backbone network. This lightweight modality classification head includes a globally average pooling layer, a first fully connected layer, an activation function layer, a Dropout layer, and a second fully connected layer connected in sequence, used to output labels for visible light or infrared modalities. A three-stage progressive training strategy is employed to train the object detection model, performing scene-specific pre-adaptation training for different data scenarios. Different training parameters and data augmentation strategies are used for each scenario to obtain scene pre-adaptation weights. These scene pre-adaptation weights are then loaded and connected to the lightweight modality classification head for joint training of object detection and modality classification tasks, resulting in a trained object detection model. Finally, a global fusion fine-tuning is performed on the object detection model using a full dataset containing all scenarios and all modalities.

[0043] In one embodiment, the total loss function of the object detection model is: ; in, For the total loss function, For target detection loss, For modal classification loss weights, Cross-entropy loss for modality classification.

[0044] In one embodiment, a target feature vector is extracted from the target detection model. HSV color space histogram features are extracted from the original image data as color feature vectors. Temporal motion features based on a 3D convolutional neural network are extracted from the original video data as temporal feature vectors. Thermal radiation intensity distribution features are extracted from the enhanced infrared image data as thermal radiation feature vectors. All extracted color feature vectors, temporal feature vectors, and thermal radiation feature vectors are normalized and aligned. An attention fusion mechanism is used to weight and fuse the normalized and aligned target feature vectors, color feature vectors, temporal feature vectors, and thermal radiation feature vectors to generate a unified fused feature vector.

[0045] In one embodiment, an attention fusion mechanism is used to weight and fuse the normalized and aligned target feature vector, the color feature vector, the temporal feature vector, and the thermal radiation feature vector to generate a unified fused feature vector. ; in, To fuse feature vectors, These are the feature vectors corresponding to the target, color, time series, and thermal radiation, respectively. For the corresponding preset weights, and satisfying .

[0046] In one embodiment, the target detection model is further configured with an adaptive modality-aware detector. Based on the modality labels output by the lightweight modality classification head, the adaptive modality-aware detector dynamically adjusts the detection parameters of the target detection model when performing target detection.

[0047] In one embodiment, the detection parameters include image size and confidence threshold. When the modal label is infrared, the adaptive modal perception detector adjusts the image size to a first preset size and the confidence threshold to a first preset threshold. When the modal label is visible light, the adaptive modal perception detector adjusts the image size to a second preset size and the confidence threshold to a second preset threshold.

[0048] In one embodiment, such as Figure 3 As shown, a YOLOv11 training and detection workflow is provided, with the following specific steps: Step 1: Multimodal intelligence data preprocessing (1) Data collection and standardization: Collect multimodal intelligence data in the form of images, videos, and infrared images. The resolution of image data is uniformly adjusted to 640×640 or 1024×1024 pixels, and the frame rate of video data is uniformly adjusted to 25fps.

[0049] (2) Fourth-order infrared physical enhancement technology: To address the issues of low contrast, blurred target outlines, and significant noise in high-altitude infrared images, a fourth-order physical enhancement algorithm is designed: First-order normalization: A linear normalization method is used to eliminate image brightness differences and map pixel values ​​to the [0, 255] interval, as shown in the following formula: ; in Input features , and This indicates taking the minimum and maximum pixel values ​​from the features. It is a minimal constant, set as .

[0050] Implementation code: img_norm = (img.astype(np.float32) - np.min(img)) / (np.max(img) -np.min(img) + 1e-6).

[0051] Second-order nonlinear compression: The hyperbolic tangent function (tanh) is used to perform nonlinear compression on the normalized image to enhance details in low-brightness areas. The formula is as follows: ; in, Used to adaptively adjust compressive strength.

[0052] Implementation code: alpha = np.percentile(img, 95) * 0.8; compressed = np.tanh(img_norm / (alpha / 255.0)) * 255.0.

[0053] Third-order filtering for noise reduction: Guided filtering is preferred to preserve edge features; if the OpenCV version does not support it, bilateral filtering is used as a fallback. Implementation code: filtered=cv2.ximgproc.guidedFilter(compressed.astype(np.uint8),compressed.astype(np.uint8), 15, 1.0).

[0054] Downgrade plan: filtered = cv2.bilateralFilter(compressed.astype(np.uint8), 9, 75,75).

[0055] Fourth-order edge enhancement: The gradient is calculated using the Sobel operator, and a specific convolution kernel is used to enhance the edges of hot spots, highlighting the thermal radiation characteristics of infrared targets.

[0056] Implementation code: sobel_x = cv2.Sobel(filtered, cv2.CV_64F, 1, 0, ksize=3) sobel_y = cv2.Sobel(filtered, cv2.CV_64F, 0, 1, ksize=3) edge = np.sqrt(sobel_x ** 2 + sobel_y ** 2) edge_enhance = cv2.filter2D(edge, -1, np.array([[0.1, 0.2, 0.1],[0.2, 0.8, 0.2], [0.1, 0.2, 0.1]])) enhanced = filtered + edge_enhance * 0.3.

[0057] (3) Data labeling and partitioning: The Labelme annotation tool was used to annotate the data, with annotation targets including five categories: road vehicles, suspicious persons, dangerous knives, open flames, and smoke. A standardized intelligence dataset was constructed and divided into training, validation, and test sets in an 8:1:1 ratio. The system has an automatic validation set partitioning function; when the validation set has fewer than 10 samples and the training set has more than 50 samples, 10% of the data is automatically and randomly allocated to the validation set.

[0058] Step 2: Object detection based on the YOLOv11 model (1) Model initialization: We directly load the official original yolo11s.pt model as the base to ensure compatibility and stability.

[0059] (2) Lightweight modal classification head design: To enable the model to perceive both visible and infrared modes, a lightweight modality classification head was designed. Based on high-level features of the YOLOv11 backbone network, modality classification was achieved with a computational increase of less than 3%, without affecting the detection speed. Architecture Design: Global average pooling layer: nn.AdaptiveAvgPool2d((1, 1)), compressing the high-level features of the backbone network (dimension [batch, 512, H, W]) to obtain a feature vector of dimension [batch, 512]. First fully connected layer: nn.Linear(512, 128), feature dimensionality reduction. Activation function: nn.ReLU(inplace=True), enhances nonlinearity. Dropout regularization: nn.Dropout(0.1) prevents overfitting. The second fully connected layer: nn.Linear(128, 2), outputs modality classification logits (0=visible light, 1=infrared). Modal label mapping: scene1_high_altitude_visible (high-altitude visible light) → 0 (visible light) scene2_close_mid_low_altitude (mid-to-low altitude visible light) → 0 (visible light) scene3_high_altitude_infrared (high-altitude infrared) → 1 (infrared) Core code: class LightweightModalClassifier(nn.Module): def __init__(self, in_channels=512, num_classes=2): super().__init__() self.avg_pool = nn.AdaptiveAvgPool2d((1, 1)) self.fc = nn.Sequential( nn.Linear(in_channels, 128), nn.ReLU(inplace=True), nn.Dropout(0.1), nn.Linear(128, num_classes) ) def forward(self, x): x = self.avg_pool(x) x = torch.flatten(x, 1) return self.fc(x) (3) Three-stage progressive training strategy [Phase 1: Scene-Specific Pre-Adaptation Training] For each scenario, a separate model is trained to quickly adapt to different target scales, background features, and data distributions. Scenario 1 configuration (high-altitude visible light, small target, complex background): Image size: imgsz=1280 Batch size: batch=8 Number of training epochs: 50 Initial learning rate: lr0 = 0.001 Number of freeze layers: freeze=8 Multi-scale training: multi_scale=True, scale=0.5 Class weights: class_weights=[1.0, 1.2, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 3.0] Data augmentation: hsv_h=0.0, hsv_s=0.0, hsv_v=0.4, degrees=0.0, mosaic=1.0, mixup=0.3, copy_paste=0.5 Scenario 2 configuration (low-to-mid altitude visible light, target size moderate, rich detail): Image size: imgsz=800 Batch size: batch=16 Number of training epochs: 40 Initial learning rate: lr0 = 0.001 Number of freeze layers: freeze=8 Multi-scale training: multi_scale=True, scale=0.2 Class weights: class_weights=[1.0, 1.2, 1.2, 1.0, 0.7, 2.0, 1.2, 1.5, 1.0, 1.5, 1.2, 2.5, 1.0, 1.0, 2.5, 1.0] Data augmentation: hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=15.0, translate=0.2, mosaic=1.0, mixup=0.15, copy_paste=0.2 Scene 3 configuration (high-altitude infrared, low contrast, blurred target outline): Image size: imgsz=1024 Batch size: batch=12 Training epochs: 60 (more iterations are needed due to poor infrared image quality) Initial learning rate: lr0 = 0.001 Freeze layer: freeze=10 Multi-scale training: multi_scale=True, scale=0.3 Class weights: class_weights=[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0] Data augmentation: hsv_h=0.0, hsv_s=0.0, hsv_v=0.6 (enhance brightness and contrast), degrees=10.0, translate=0.2, mosaic=0.5, mixup=0.1, copy_paste=0.3 Key points of training strategy: Initialize the model separately for each scenario to avoid the decline in model generalization ability caused by differences in data distribution across different scenarios. A cosine annealing learning rate (cos_lr=True) is used to gradually reduce the learning rate and improve the model's convergence stability. The optimal weights for low-to-mid-altitude visible light scenes are selected as the initial weights for the next stage.

[0060] [Phase Two: Joint Modal Pre-training] Based on the scene pre-adaptation weights from the first stage, a modality classification head is added to perform joint training for the dual tasks of "detection / segmentation + modality classification": Model loading: YOLOModalAware(stage1_weight) Number of training epochs: 20 Image size: imgsz=800 Batch size: batch=16 Initial learning rate: lr0 = 0.0005 (lower than the first stage) Freeze layer number: freeze=0 (do not freeze, train all layers together) Data augmentation: mosaic=0.5, mixup=0.1, copy_paste=0.2 Total loss function: ; in, For detection / segmentation loss ( ), For modal classification loss weights, Cross-entropy loss for modality classification: ; [Phase 3: Full Integration and Fine-tuning] Fine-tuning was performed using the full dataset (all scenes, all modalities) to allow the model to incorporate features from different scenes and modalities: Loading the model: YOLO(modal_joint_weight) Number of training epochs: 30 Image size: imgsz=800 Batch size: batch=16 Initial learning rate: lr0 = 0.0001 (significantly lower than the previous two stages) Disable mixup and copy_paste enhancements to reduce mosaic intensity (0.3). Cancel cosine annealing (cos_lr=False) and use a fixed learning rate decay.

[0061] Step 3: Multimodal feature fusion (1) Feature extraction: YOLOv11 target features: Extract the 1024-dimensional target feature vector from the model output; Image HSV color features: 256-dimensional HSV color histogram features were extracted using OpenCV; Video 3D-CNN Temporal Features: 512-dimensional temporal motion features are extracted from 16 consecutive video frames using a 3D convolutional neural network; Infrared thermal radiation characteristics: Extract 256-dimensional thermal radiation intensity distribution characteristics from enhanced infrared images.

[0062] (2) Feature alignment and fusion: The Min-Max normalization algorithm is used to normalize all feature vectors to the [0,1] interval, eliminating the difference in feature dimensions; Color feature 0.2, temporal feature 0.3, generating a unified 1024-dimensional multimodal feature vector; After fusion, the feature recognition accuracy is improved by more than 20%, and feature visualization is supported.

[0063] Step 4: Intelligent analysis based on large models. The large model calling process is as follows: Figure 4 As shown, the specific content is as follows: (1) Large-scale model deployment architecture: Base model: The Deepseek-r1:8b large model is selected as the default base model; Multi-model support: Simultaneously supports switching between models such as ChatGLM3 and Llama3, and provides three loading methods: llama-cpp-python (GGUF format), transformers (HuggingFace format), and local rule analysis (downgrade solution). Deployment method: Local deployment based on the PyTorch framework, with reserved API interfaces to support large cloud models such as OpenAIGPT, Baidu Wenxin, and Alibaba Tongyi.

[0064] (2) Design of prompt words for professional multimodal information analysis: Target correlation analysis: Analyzes "the relationship between personnel and accompanying personnel", "the correlation of social vehicle driving trajectories", and "multi-target peer aggregation and linkage patterns"; Situation assessment of key urban areas: assessing "degree of population gathering", "prediction of abnormal behavior trends", "assessment of potential event risks", and "identification of group marching formations"; Risk assessment: Perform "risk target identification", "safety risk level classification (high / medium / low)" and "incident handling priority ranking".

[0065] (3) Analysis of triggering mechanism: Real-time single-frame analysis is triggered once every 30 frames, while also supporting batch statistical analysis and video scene summary generation.

[0066] (4) Output format: Output structured analysis results in JSON format, including fields such as target ID, category, location, confidence level, correlation, and risk level.

[0067] Step 5: Analysis report automatically generated (1) Report template design: The pre-set standardized analysis report template includes four core modules: Summary of target information: target category, quantity, location coordinates, confidence level, key attributes (such as item type, vehicle model); Association analysis results: Target-to-target companion relationship map, movement trajectory logic analysis, and group flow and aggregation patterns; Situation analysis conclusions: overall description of regional security situation, prediction of target behavior trends, and early warning of potential threats from abnormal events; Risk assessment recommendations: Focus on the target list (suspicious persons, vehicles violating traffic rules, suspicious items left behind), risk event response strategies, and key areas for security control.

[0068] (2) Multi-format export: Automatically generates PDF / Word format reports, and also supports the export of structured data in TXT / CSV / JSON formats for easy secondary analysis. (3) Interactive functions: Supports manual editing of report content, classified archiving management, and query and comparison of historical reports.

[0069] It is worth noting that compared with existing technologies, efficiency is significantly improved: through the collaborative work of the YOLOv11 model and large models, the automated processing and intelligent analysis of multimodal intelligence data are realized. The time for target recognition of a single image is reduced to less than 0.02 seconds, and the time for target analysis of a single 3-minute video segment is reduced to less than 5 minutes, reducing intelligence analysis time by more than 80%, meeting real-time requirements, and greatly improving decision-making response efficiency.

[0070] Multimodal fusion is comprehensive: By adopting feature alignment and attention fusion mechanisms, feature information from multimodal data such as images and videos is effectively fused, which solves the problem of one-sided analysis of single data types in existing technologies, improves the comprehensiveness of analysis results, and enables intelligence analysis to not only identify targets, but also judge the target's action trends and mission intentions; High analysis accuracy: The optimized YOLOv11 model improves the target detection accuracy in small targets and complex scenes, with a mean average precision (mAP) of over 92.3% and a false positive rate of less than 3%. Combined with the semantic understanding capabilities of the large model, it further reduces the probability of false positives and false negatives, resulting in more accurate analysis results that can meet the high-precision requirements of patrol and security. High level of intelligence: It realizes full automation from data preprocessing and target detection to intelligence report generation, eliminating the need for manual sorting and analysis of results, reducing labor costs, improving the standardization of intelligence analysis, avoiding errors caused by manual sorting, and supporting multi-terminal access and data encryption, thereby improving the system's usability and security. High generalization ability: Through model optimization and feature fusion, it can adapt to multimodal intelligence data in different scenarios. It can flexibly adjust the target category and analysis needs. In complex environments such as rain, fog, and low light, the detection accuracy drops by less than 5%. It is suitable for various scenarios such as land patrol, security monitoring, and community prevention and control, and has good practicality and promotion value.

[0071] Outstanding modal adaptation capability (core innovative effect): Modal perception achieves a computational increase of less than 3% through a lightweight modal classification head. Combined with a dynamic modal perception detector, it can automatically adjust detection parameters according to the modality (visible light / infrared) of the input image, solving the problem of accuracy degradation caused by adapting a single parameter to all modalities. The detection accuracy in the infrared modality is improved by more than 15%. It should be understood that, although Figures 2-4 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2-4 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0072] In one embodiment, such as Figure 5As shown, a multimodal information intelligent analysis device is provided, including: a data acquisition module 502, a detection model construction module 504, a feature fusion module 506, an intelligent analysis module 508, and a report generation module 510, wherein: The data acquisition module 502 is used to acquire multimodal information data, which includes image data, video data, and infrared image data.

[0073] The detection model building module 504 is used to build an object detection model based on the YOLOv11 architecture. The object detection model integrates a lightweight modality classification head to perform object detection and modality recognition simultaneously, and outputs the object detection results and the corresponding modality labels.

[0074] The feature fusion module 506 is used to perform physical enhancement processing on infrared image data. It inputs the enhanced infrared image data and other modal data into the target detection model, extracts multi-layer features of the target detection model, and combines them with auxiliary features extracted from the original multi-modal data. It then performs feature fusion through an attention mechanism to generate a fused feature vector.

[0075] The intelligent analysis module 508 is used to input the fused feature vector into the large language model and generate structured analysis results by combining it with preset analysis prompts.

[0076] The report generation module 510 is used to automatically generate an information analysis report based on the structured analysis results and according to a preset report template.

[0077] For specific limitations regarding the multimodal information intelligent analysis device, please refer to the limitations of the multimodal information intelligent analysis method described above, which will not be repeated here. Each module in the aforementioned multimodal information intelligent analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0078] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal information intelligent analysis method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0079] Those skilled in the art will understand that Figures 5-6 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0080] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps: Acquire multimodal information data, which includes image data, video data, and infrared image data.

[0081] An object detection model is built based on the YOLOv11 architecture. The object detection model integrates a lightweight modality classification head to perform object detection and modality recognition simultaneously, and outputs object detection results and corresponding modality labels.

[0082] Physical enhancement processing is performed on the infrared image data. The enhanced infrared image data and other modal data are input into the target detection model. Multi-layer features of the target detection model are extracted and combined with auxiliary features extracted from the original multi-modal data. Feature fusion is performed through an attention mechanism to generate a fused feature vector.

[0083] The fused feature vectors are input into a large language model, and combined with preset analysis prompts, structured analysis results are generated.

[0084] Based on the structured analysis results, an information analysis report is automatically generated according to a preset report template.

[0085] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire multimodal information data, which includes image data, video data, and infrared image data.

[0086] An object detection model is built based on the YOLOv11 architecture. The object detection model integrates a lightweight modality classification head to perform object detection and modality recognition simultaneously, and outputs object detection results and corresponding modality labels.

[0087] Physical enhancement processing is performed on the infrared image data. The enhanced infrared image data and other modal data are input into the target detection model. Multi-layer features of the target detection model are extracted and combined with auxiliary features extracted from the original multi-modal data. Feature fusion is performed through an attention mechanism to generate a fused feature vector.

[0088] The fused feature vectors are input into a large language model, and combined with preset analysis prompts, structured analysis results are generated.

[0089] Based on the structured analysis results, an information analysis report is automatically generated according to a preset report template.

[0090] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink, DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0091] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A multimodal information intelligent analysis method, characterized in that, The method includes: Acquire multimodal information data, which includes image data, video data, and infrared image data; A target detection model is built based on the YOLOv11 architecture. The target detection model integrates a lightweight modality classification head, which is used to perform target detection and modality recognition simultaneously, and output the target detection results and the corresponding modality labels. Physical enhancement processing is performed on the infrared image data. A linear normalization method is used to map the pixel values ​​of the infrared image data to a preset numerical range to obtain normalized image data, thereby eliminating brightness differences. in, For input pixel values, and These are the minimum and maximum pixel values, respectively. It is a very small constant; The normalized image data is subjected to a nonlinear transformation using the hyperbolic tangent function to obtain compressed image data, thereby enhancing details in low-brightness areas. in, The parameters are adaptively determined based on the pixel distribution of infrared image data and used to adjust the compression intensity; Guided filtering or bilateral filtering is applied to the compressed image data to preserve edge features and remove noise; The gradient is calculated on the filtered image data, and edge enhancement is performed using a preset convolution kernel to highlight the thermal radiation characteristics of the target. The enhanced infrared image data and other modal data are input into the target detection model, multi-layer features of the target detection model are extracted, and auxiliary features extracted from the original multimodal data are combined with the features through an attention mechanism to generate a fused feature vector. The fused feature vector is processed into a text serialization process to obtain a structured text feature description. The structured text feature description is then combined with a professional prompt word template for information analysis and input into a locally deployed large language model to generate structured analysis results. Based on the structured analysis results, an information analysis report is automatically generated according to a preset report template.

2. The method according to claim 1, characterized in that, A target detection model based on the YOLOv11 architecture is built, including: A lightweight modality classification head is connected after the high-level features of the YOLOv11 backbone network. The lightweight modality classification head includes a global average pooling layer, a first fully connected layer, an activation function layer, a Dropout layer and a second fully connected layer connected in sequence, which are used to output labels for visible light or infrared modalities. The target detection model is trained using a three-stage progressive training strategy. Scene-specific pre-adaptation training is performed for different data scenarios. Different training parameters and data augmentation strategies are used for each scenario to obtain scene pre-adaptation weights. Load the scene pre-adapted weights, connect the lightweight modal classification head, and perform joint training of the target detection task and the modal classification task to obtain the trained target detection model; The object detection model is globally fused and fine-tuned using a full dataset containing all scenes and all modalities.

3. The method according to any one of claims 1 to 2, characterized in that, The total loss function of the target detection model is: in, For the total loss function, For target detection loss, For modal classification loss weights, Cross-entropy loss for modality classification; The target detection model is also equipped with an adaptive modal perception detector; Based on the modal labels output by the lightweight modal classification head, the detection parameters of the target detection model are dynamically adjusted by the adaptive modal perception detector when performing target detection.

4. The method according to claim 3, characterized in that, Multi-layer features of the target detection model are extracted and combined with auxiliary features extracted from the original multimodal data. Feature fusion is performed through an attention mechanism to generate a fused feature vector, including: Extract the target feature vector from the target detection model; Extract HSV color space histogram features from the original image data as color feature vectors; Temporal motion features based on 3D convolutional neural networks are extracted from the raw video data and used as temporal feature vectors. The thermal radiation intensity distribution features are extracted from the enhanced infrared image data and used as thermal radiation feature vectors. All extracted color feature vectors, temporal feature vectors, and thermal radiation feature vectors are normalized and aligned. The normalized and aligned target feature vector, color feature vector, temporal feature vector, and thermal radiation feature vector are weighted and fused through an attention fusion mechanism to generate a unified fused feature vector.

5. The method according to claim 4, characterized in that, Through an attention fusion mechanism, the normalized and aligned target feature vector, color feature vector, temporal feature vector, and thermal radiation feature vector are weighted and fused to generate a unified fused feature vector, including: Through an attention fusion mechanism, the normalized and aligned target feature vector, color feature vector, temporal feature vector, and thermal radiation feature vector are weighted and fused to generate a unified fused feature vector. ， in, To fuse feature vectors, These are the feature vectors corresponding to the target, color, time series, and thermal radiation, respectively. For the corresponding preset weights, and satisfying .

6. The method according to claim 1, characterized in that, The detection parameters include image size and confidence threshold; When the modal label is infrared, the adaptive modal perception detector adjusts the image size to a first preset size and the confidence threshold to a first preset threshold; When the modal label is visible light, the adaptive modal perception detector adjusts the image size to a second preset size and the confidence threshold to a second preset threshold.

7. A multimodal information intelligent analysis device, characterized in that, The apparatus for implementing the method according to any one of claims 1 to 6, the apparatus comprising: The data acquisition module is used to acquire multimodal information data, which includes image data, video data, and infrared image data. The detection model building module is used to build an object detection model based on the YOLOv11 architecture. The object detection model integrates a lightweight modality classification head, which is used to perform object detection and modality recognition simultaneously, and output the object detection results and the corresponding modality labels. The feature fusion module is used to perform physical enhancement processing on the infrared image data, input the enhanced infrared image data and other modal data into the target detection model, extract multi-layer features of the target detection model, and combine them with auxiliary features extracted from the original multimodal data. The feature fusion is performed through an attention mechanism to generate a fused feature vector. The intelligent analysis module is used to perform text serialization processing on the fused feature vector to obtain a structured text feature description. The structured text feature description is then combined with a professional prompt word template for information analysis and input into a locally deployed large language model to generate structured analysis results. The report generation module is used to automatically generate an information analysis report based on the structured analysis results and according to a preset report template.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images