Case handling method assisted by monitoring camera based on multi-modal visual large model

Abstract

The application discloses a kind of based on multimodal vision big model's monitoring camera auxiliary case handling method, the method is through camera integration visible light, infrared thermal imaging, audio acquisition module, multi-dimensional data are synchronously collected, and complete noise reduction, time alignment, feature extraction and other standardization pretreatment.Rely on built-in multimodal vision big model, carry out joint inference by fusing multimodal features, realize target accurate identification, behavior intelligent analysis and key evidence extraction.Analysis result is standardized, automatically generates the evidence file and case handling report with watermark, supports one-key export to case handling system.Meanwhile, a multi-level early warning mechanism is constructed, and when an anomaly is detected, an early warning is promptly pushed, and a full-process intervention record is completed by a linked device.The application solves the drawbacks of traditional monitoring case handling, such as high dependence on manpower, low efficiency, poor recognition accuracy and non-standard evidence integration, effectively improving the intelligent and standardized level of judicial case handling and the reliability of evidence collection.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring and judicial case handling assistance technology, and more specifically to a case handling assistance method based on a multimodal visual large model surveillance camera. Background Technology

[0002] With the increasing standardization of judicial case handling, surveillance cameras have become an indispensable basic device in case handling scenarios. However, existing technologies have obvious limitations: First, traditional surveillance can only record and play back videos, relying on manual review and analysis, which is inefficient and prone to missed or false alarms due to fatigue. Second, single visual recognition technology cannot handle complex case handling scenarios (such as violations in the interrogation room, multi-target correlation at the crime scene, and evidence collection from blurry images). Third, it lacks the ability to integrate multi-dimensional evidence, making it difficult to quickly extract key case information (such as suspect characteristics, tools used in the crime, and behavioral trajectories).

[0003] Therefore, how to achieve comprehensive perception and intelligent analysis of target, behavior, and environmental information in case-handling scenarios through the in-depth application of multimodal visual large models, thereby improving case-handling efficiency and evidence reliability, is the core challenge currently facing the technology. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a case-handling auxiliary monitoring camera system equipped with a multimodal visual large-scale model. By integrating multimodal perception modules (high-definition vision, infrared, and audio) with a self-developed multimodal visual large-scale model, a comprehensive case-handling assistance solution is formed, encompassing "data acquisition - multimodal fusion - intelligent analysis - evidence generation - early warning feedback." This system synchronously collects multi-source data, including video, audio, and infrared images, from the case-handling scene through multiple sensors built into the camera. Real-time analysis by the multimodal visual large-scale model enables precise target identification, early warning of violations, extraction of key evidence, and generation of standardized reports, providing comprehensive intelligent support for case-handling work.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for assisting in case handling using a surveillance camera based on a multimodal visual large model, comprising the following steps: Step 1: The camera integrates a high-definition visible light module, an infrared thermal imaging module, and a high-fidelity audio acquisition module to simultaneously acquire multi-dimensional data, including visible light data, infrared data, and audio data. Then, Gaussian filtering is used to remove image noise, and multi-modal data time synchronization is achieved through timestamp alignment. Noise reduction and feature extraction are performed on the audio signal to provide standardized data for subsequent model input. Step two: By using the built-in multimodal visual big model, visual, infrared and audio features are combined for joint reasoning to achieve accurate target recognition, intelligent behavior analysis and intelligent evidence extraction. Step 3: Standardize the identification and analysis results from Step 2: Sort key evidence fragments by timeline, automatically label evidence types, and generate watermarked, tamper-proof evidence files; at the same time, generate case-handling auxiliary reports based on the analysis results, including a summary of target information, behavioral analysis conclusions, and a list of evidence, and support one-click export to the case-handling system; Step four: Upon detecting violations or abnormal situations, immediately trigger a multi-level early warning mechanism, including issuing local audible and visual alarms, pushing early warning information to the investigators' mobile apps, and simultaneously uploading the information to the case supervision platform; it also supports linkage with access control and recording equipment in the case handling location to achieve immediate intervention and full-process recording of violations.

[0006] As a further improvement of the present invention, the multimodal visual large model in step two adopts a "visual-infrared-audio" multimodal feature fusion architecture, which is as follows: Let the feature vectors of the three modalities be visual features. Infrared features Audio features The features are weighted and fused into a unified feature vector through an attention mechanism. :

[0007]

[0008]

[0009]

[0010] in, For the attention weights of each modality, This is the attention weight matrix, obtained through model training and optimization, ensuring that important modal features receive higher weights. Subsequently, based on fusion features The probability distribution of the target category or behavior category is calculated using a fully connected layer and a softmax function:

[0011]

[0012] in, This is the output layer weight matrix. For bias terms, The total number of categories, This refers to the predicted probability of a certain category after fusing multimodal information.

[0013] As a further improvement to the present invention, in step two, the multimodal visual large model, for the specific needs of the case-handling scenario, is based on the YOLOv11 lightweight architecture and constructs a three-level recognition model of "object detection + key point recognition + behavior temporal analysis", as follows: Using the improved YOLOv11 bounding box regression formula, the target location and size are predicted:

[0014]

[0015]

[0016]

[0017] in, The coordinates of the center of the predicted bounding box. To predict the width and height of the bounding box; Predict the offset for the model; The width and height of the anchor frame; The Sigmoid function maps the coordinate offset to the [0,1] interval; The spatiotemporal features of behavior are extracted using a 3D convolutional neural network (3D-CNN), as shown in the following formula:

[0018] in, The video sequence in spatial coordinates Time frame Pixel value at; 3D convolution kernel; Let be the spatiotemporal feature vector at time t, capturing the dynamic changes in behavior.

[0019] As a further improvement of this invention, the multimodal visual large model proposes an evidence enhancement model based on generative adversarial networks for blurry images commonly encountered in case-handling scenarios. It also incorporates an attention mechanism to automatically extract key information, as detailed below: Loss function for GAN evidence enhancement models: Generator With discriminator The losses from the confrontation are:

[0020] in, For clear evidence images, It is random noise. The enhanced image output by the generator. For the true data distribution, This represents the noise distribution.

[0021] Introducing perceptual loss enhances the realism of image details:

[0022] in, For pre-trained CNNs (such as VGG16) Feature mapping of layers, For the first Total number of elements in the layer feature map The number of feature extraction layers; Total loss function: , For perceived loss weights.

[0023] (2) Key information attention extraction: The spatial attention mechanism is used to locate key regions (such as faces or tools used in crimes) in images, as shown in the following formula:

[0024]

[0025] in, For the input feature map, , These are the results of global max pooling and average pooling, respectively. For feature splicing, This is a two-dimensional convolution operation. For the Sigmoid function, This is a spatial attention map. The key feature map after attention weighting.

[0026] As a further improvement of the present invention, step four also includes adaptive learning of case-handling scenarios and dynamic adjustment of early warning thresholds. Through the built-in adaptive learning module, the model parameters and early warning thresholds are optimized based on the characteristics of different case-handling scenarios and historical data to improve scenario adaptability, as detailed below: A scenario adaptation strategy based on reinforcement learning is used to define the state. Given the current scene type and environmental parameters, the action... Rewards for adjusting model parameters To improve accuracy, the Q function is updated using the following formula:

[0027] in, For learning rate, Discount factor; Warning threshold Based on scenario complexity Historical false alarm rate Dynamic adjustment:

[0028] in, Basic threshold; The scene complexity is represented by a value between 0 and 1, indicating higher population density and more complex environment. The larger; Historical false alarm rate; , This is for adjusting the coefficient.

[0029] Beneficial effects: This invention, through a comprehensive solution encompassing multimodal data synchronous acquisition, intelligent analysis, standardized evidence generation, and multi-level early warning linkage, completely resolves the industry pain points of traditional monitoring and case handling, which rely on manual review, are inefficient, and prone to errors. By integrating multi-dimensional data from visible light, infrared, and audio, it overcomes the technical limitations of single-vision recognition, adapting to various complex case-handling scenarios such as interrogation monitoring and crime scene evidence collection, effectively avoiding identification omissions caused by the lack of single-modal information. Simultaneously, it achieves the automatic extraction, organization, and solidification of key case evidence, generating tamper-proof standardized evidence documents and case reports. This solves the problems of traditional equipment lacking systematic evidence integration capabilities and cumbersome key information extraction, significantly improving the intelligence, standardization, and reliability of judicial case handling.

[0030] Furthermore, this invention further improves the accuracy of target and behavior recognition, the ability to repair fuzzy evidence, and the adaptability to different scenarios through a multimodal attention fusion architecture, a lightweight three-level recognition model, GAN image enhancement technology, and reinforcement learning adaptive warning optimization. It effectively reduces the probability of false alarms and missed alarms in complex case-handling scenarios, further optimizes the overall case-handling assistance effect, and adapts to the usage needs of various differentiated case-handling scenarios. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to the given embodiments.

[0032] This embodiment of a surveillance camera-assisted case-handling method based on a multimodal visual large model includes the following steps: Step 1: The camera integrates a high-definition visible light module, an infrared thermal imaging module, and a high-fidelity audio acquisition module to simultaneously acquire multi-dimensional data, including visible light data, infrared data, and audio data. Then, Gaussian filtering is used to remove image noise, and multi-modal data time synchronization is achieved through timestamp alignment. Noise reduction and feature extraction are performed on the audio signal to provide standardized data for subsequent model input. Step two: By using the built-in multimodal visual big model, visual, infrared and audio features are combined for joint reasoning to achieve accurate target recognition, intelligent behavior analysis and intelligent evidence extraction. Step 3: Standardize the identification and analysis results from Step 2: Sort key evidence fragments by timeline, automatically label evidence types, and generate watermarked, tamper-proof evidence files; at the same time, generate case-handling auxiliary reports based on the analysis results, including a summary of target information, behavioral analysis conclusions, and a list of evidence, and support one-click export to the case-handling system; Step four: Upon detecting violations or abnormal situations, immediately trigger a multi-level early warning mechanism, including issuing local audible and visual alarms, pushing early warning information to the investigators' mobile apps, and simultaneously uploading the information to the case supervision platform; it also supports linkage with access control and recording equipment in the case handling location to achieve immediate intervention and full-process recording of violations.

[0033] In practical application, this method synchronously collects multi-dimensional raw data from multiple sensors and performs standardized preprocessing such as noise reduction and time-series alignment. This solves the problems of traditional surveillance, which only collects single video data, resulting in disorganized raw data that cannot be directly used for intelligent analysis. Relying on a multimodal large model, it achieves intelligent analysis, replacing manual review and screening. This completely improves the shortcomings of manual analysis, such as low efficiency and susceptibility to fatigue and errors. Furthermore, through standardized evidence generation and multi-level early warning linkage, it achieves intelligent processing of the entire process, including case handling, evidence collection, risk warning, and workflow recording. This effectively improves case handling efficiency and evidence standardization, realizing the core beneficial effects of this invention.

[0034] Furthermore, the multimodal visual large model in step two adopts a "visual-infrared-audio" multimodal feature fusion architecture, which is as follows: Let the feature vectors of the three modalities be visual features. Infrared features Audio features The features are weighted and fused into a unified feature vector through an attention mechanism. :

[0035]

[0036]

[0037]

[0038] in, For the attention weights of each modality, This is the attention weight matrix, obtained through model training and optimization, ensuring that important modal features receive higher weights. Subsequently, based on fusion features The probability distribution of the target category or behavior category is calculated using a fully connected layer and a softmax function:

[0039]

[0040] in, This is the output layer weight matrix. For bias terms, The total number of categories, This refers to the predicted probability of a certain category after fusing multimodal information.

[0041] This architecture dynamically allocates the weights of each modality's features through an attention mechanism, automatically emphasizing effective case-handling information and weakening invalid interference information. This enables deep fusion analysis of multimodal data, solving the problems of fixed weights, poor adaptability, and low recognition accuracy in complex scenarios inherent in traditional multimodal fusion methods. Furthermore, it achieves precise linkage analysis of multi-dimensional information, significantly improving the accuracy of target recognition and behavior judgment, and adapting to complex case-handling environments such as low light and noise interference.

[0042] Furthermore, in step two, the multimodal visual large model, tailored to the specific needs of case-handling scenarios, is built based on the YOLOv11 lightweight architecture, constructing a three-level recognition model of "object detection + key point recognition + behavior temporal analysis," as detailed below: Using the improved YOLOv11 bounding box regression formula, the target location and size are predicted:

[0043]

[0044]

[0045]

[0046] in, The coordinates of the center of the predicted bounding box. To predict the width and height of the bounding box; Predict the offset for the model; The width and height of the anchor frame; The Sigmoid function maps the coordinate offset to the [0,1] interval; The spatiotemporal features of behavior are extracted using a 3D convolutional neural network (3D-CNN), as shown in the following formula:

[0047] in, The video sequence in spatial coordinates Time frame Pixel value at; 3D convolution kernel; Let be the spatiotemporal feature vector at time t, capturing the dynamic changes in behavior.

[0048] This three-level recognition model is based on a lightweight YOLOv11 architecture, balancing recognition accuracy and operating speed. Through layered processing of target detection, key point recognition, and temporal behavior analysis, it accurately captures target information and dynamic behaviors in case-handling scenarios, solving the problems of traditional recognition models that can only perform static detection, cannot analyze continuous behavior, and suffer from chaotic multi-target recognition. It also achieves accurate capture of dynamic case-handling information such as suspect trajectories and violations, adapting to the intelligent analysis needs of multi-target and dynamic behaviors in complex scenarios.

[0049] Furthermore, for the common blurred images in case-handling scenarios, the multimodal visual big data model proposes an evidence enhancement model based on generative adversarial networks, while combining an attention mechanism to achieve automatic extraction of key information, as detailed below: Loss function for GAN evidence enhancement models: Generator With discriminator The losses from the confrontation are:

[0050] in, For clear evidence images, It is random noise. The enhanced image output by the generator. For the true data distribution, This represents the noise distribution.

[0051] Introducing perceptual loss enhances the realism of image details:

[0052] in, For pre-trained CNNs (such as VGG16) Feature mapping of layers, For the first Total number of elements in the layer feature map The number of feature extraction layers; Total loss function: , For perceived loss weights.

[0053] (2) Key information attention extraction: The spatial attention mechanism is used to locate key regions (such as faces or tools used in crimes) in images, as shown in the following formula:

[0054]

[0055] in, For the input feature map, , These are the results of global max pooling and average pooling, respectively. For feature splicing, This is a two-dimensional convolution operation. For the Sigmoid function, This is a spatial attention map. The key feature map after attention weighting.

[0056] This module uses a GAN model to intelligently enhance blurry evidence images. Combined with a spatial attention mechanism, it accurately locks onto key case information, effectively extracting incomplete and blurry evidence. This solves the problems of traditional surveillance's inability to repair blurry images, the easy loss of key evidence, and invalid information interfering with evidence collection. Additionally, it optimizes and precisely filters flawed evidence, significantly improving the integrity and effectiveness of evidence in complex evidence collection scenarios.

[0057] Furthermore, step four involves adaptive learning for case-handling scenarios and dynamic adjustment of early warning thresholds. Through the built-in adaptive learning module, model parameters and early warning thresholds are optimized based on the characteristics of different case-handling scenarios and historical data to improve scenario adaptability, as detailed below: A scenario adaptation strategy based on reinforcement learning is used to define the state. Given the current scene type and environment parameters, the action... Rewards for adjusting model parameters To improve accuracy, the Q function is updated using the following formula:

[0058] in, For learning rate, Discount factor; Warning threshold Based on scenario complexity Historical false alarm rate Dynamic adjustment:

[0059] in, Basic threshold; The scene complexity is represented by a value between 0 and 1, indicating higher population density and more complex environment. The larger; Historical false alarm rate; , This is for adjusting the coefficient.

[0060] This adaptive learning module dynamically optimizes model parameters and warning thresholds through reinforcement learning, adapting the warning mechanism to different case-handling scenarios. This solves the problems of fixed warning thresholds, high false alarm and false negative rates in complex scenarios, and poor scenario adaptability inherent in traditional methods. Furthermore, it implements intelligent iterative optimization of the warning strategy, adapting to the differentiated warning needs of different scenarios such as interrogation rooms and crime scenes, significantly improving warning accuracy and system versatility.

[0061] In summary, this invention constructs a surveillance camera-assisted case-handling solution that integrates multimodal perception, intelligent fusion analysis, standardized evidence generation, and adaptive early warning linkage. By synchronously collecting multi-dimensional case-handling data through multiple sensors and combining it with a self-developed multimodal visual large model to complete precise intelligent analysis, it effectively solves the technical problems of high reliance on manual labor, low recognition accuracy, weak evidence integration capabilities, and poor scene adaptability in traditional case-handling surveillance. It realizes intelligent assistance in judicial case handling, standardized evidence retention, and real-time risk early warning, significantly improving case-handling efficiency and the reliability and standardization of judicial evidence collection.

[0062] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for assisting in case handling using surveillance cameras based on a multimodal visual large model, characterized in that: Includes the following steps: Step 1: The camera integrates a high-definition visible light module, an infrared thermal imaging module, and a high-fidelity audio acquisition module to simultaneously acquire multi-dimensional data, including visible light data, infrared data, and audio data. Then, Gaussian filtering is used to remove image noise, and multi-modal data time synchronization is achieved through timestamp alignment. Noise reduction and feature extraction are performed on the audio signal to provide standardized data for subsequent model input. Step two: By using the built-in multimodal visual big model, visual, infrared and audio features are combined for joint reasoning to achieve accurate target recognition, intelligent behavior analysis and intelligent evidence extraction. Step 3: Standardize the identification and analysis results from Step 2: Sort key evidence fragments by timeline, automatically label evidence types, and generate watermarked, tamper-proof evidence files; at the same time, generate case-handling auxiliary reports based on the analysis results, including a summary of target information, behavioral analysis conclusions, and a list of evidence, and support one-click export to the case-handling system; Step four: Upon detecting violations or abnormal situations, immediately trigger a multi-level early warning mechanism, including issuing local audible and visual alarms, pushing early warning information to the investigators' mobile apps, and simultaneously uploading the information to the case supervision platform; it also supports linkage with access control and recording equipment in the case handling location to achieve immediate intervention and full-process recording of violations.

2. The method for assisting in case handling based on a multimodal visual large model using a surveillance camera according to claim 1, characterized in that: The multimodal visual large model in step two adopts a "visual-infrared-audio" multimodal feature fusion architecture, which is as follows: Let the feature vectors of the three modalities be visual features. Infrared features Audio features The features are weighted and fused into a unified feature vector through an attention mechanism. : in, For the attention weights of each modality, This is the attention weight matrix, obtained through model training and optimization, ensuring that important modal features receive higher weights. Subsequently, based on fusion features The probability distribution of the target category or behavior category is calculated using a fully connected layer and a softmax function: in, This is the output layer weight matrix. For bias terms, The total number of categories, This refers to the predicted probability of a certain category after fusing multimodal information.

3. The method for assisting in case handling based on a multimodal visual large model using a surveillance camera according to claim 2, characterized in that: In step two, the multimodal visual large model, tailored to the specific needs of case-handling scenarios, is based on the YOLOv11 lightweight architecture and constructs a three-level recognition model consisting of "object detection + key point recognition + behavior temporal analysis," as detailed below: Using the improved YOLOv11 bounding box regression formula, the target location and size are predicted: in, The coordinates of the center of the prediction box. To predict the width and height of the bounding box; Predict the offset for the model; The width and height of the anchor frame; The Sigmoid function maps the coordinate offset to the [0,1] interval; The spatiotemporal features of behavior are extracted using a 3D convolutional neural network, as shown in the following formula: in, The video sequence in spatial coordinates Time frame Pixel value at; 3D convolution kernel; Let be the spatiotemporal feature vector at time t, capturing the dynamic changes in behavior.

4. The method for assisting in case handling based on a multimodal visual large model using a surveillance camera according to claim 3, characterized in that: The proposed multimodal visual big data model addresses the common problem of blurred images in case-handling scenarios by proposing an evidence enhancement model based on generative adversarial networks. It also incorporates an attention mechanism to automatically extract key information, as detailed below: Loss function for GAN evidence enhancement models: generator With discriminator The losses from the confrontation are: in, For clear evidence images, It is random noise. The enhanced image output by the generator. For the true data distribution, Noise distribution; Introducing perceptual loss enhances the realism of image details: in, For pre-training CNN Feature mapping of layers, For the first Total number of elements in the layer feature map The number of feature extraction layers; Total loss function: , Weights for perceived loss; (2) Key information attention extraction: The spatial attention mechanism is used to locate key regions (such as faces or tools used in crimes) in images, as shown in the following formula: in, For the input feature map, , These are the results of global max pooling and average pooling, respectively. For feature splicing, This is a two-dimensional convolution operation. For the Sigmoid function, This is a spatial attention map. The key feature map after attention weighting.

5. The method for assisting in case handling based on a multimodal visual large model using a surveillance camera according to claim 4, characterized in that: Step four also involves adaptive learning for case-handling scenarios and dynamic adjustment of early warning thresholds. Through the built-in adaptive learning module, the model parameters and early warning thresholds are optimized based on the characteristics of different case-handling scenarios and historical data to improve scenario adaptability, as detailed below: A scenario adaptation strategy based on reinforcement learning is used to define the state. Given the current scene type and environment parameters, the action... Rewards for adjusting model parameters To improve accuracy, the Q function is updated using the following formula: in, For learning rate, Discount factor; Warning threshold Based on scenario complexity Historical false alarm rate Dynamic adjustment: in, Basic threshold; The scene complexity is represented by a value between 0 and 1, indicating higher population density and more complex environment. The larger; Historical false alarm rate; , This is for adjusting the coefficient.

Images