Identity recognition monitoring method based on video image

Abstract

The invention relates to an identity recognition monitoring method based on video images, and relates to the technical field of monitoring systems. The method is executed by a server which deploys a TensorRT inference engine, and comprises the following steps: S1, carrying out depth separable convolution operation on an input video stream; s2, dynamically determining and generating a face-gait fusion weight through an illumination quality evaluation function; s3, constructing a cross-modal feature vector and comparing the cross-modal feature vector with a dynamic threshold library; s4, inserting a space-time attention unit behind a fourth residual block of the ResNet network; and S5, realizing three-pipeline parallel processing through double buffer queues: executing OpenCV image preprocessing accelerated by CUDA (Compute Unified Device Architecture) in the first queue, performing TensorRT reasoning in the second queue, and completing identification result encryption transmission in the third queue. According to the method, the accuracy and processing efficiency of identity recognition in complex dynamic scenes such as side faces, shielding, low illumination, different crowd densities and large monitoring view angles are effectively improved, and the requirement of a monitoring system for real-time and efficient processing of a large amount of video data is met.

Description

Technical Field

[0001] The present invention relates to the technical field of monitoring systems, and in particular to an identity recognition monitoring method based on video images. Background Art

[0002] In the field of surveillance systems, real-time identity recognition technology based on video images is of great significance. The current mainstream method is to compare static facial feature databases, such as the ResNet-50 network method disclosed in patent CN112233445A to extract facial features. However, this method significantly reduces recognition accuracy when the target subject's face is in profile, obscured, or in low light. Existing systems lack sufficient accuracy in capturing walking posture features, reportedly reaching less than 78%. Furthermore, they fail to effectively integrate gait and multimodal facial features, making it impossible to leverage multi-dimensional information to improve recognition accuracy and reliability. Furthermore, experiments have shown that when the monitoring angle of view exceeds 45 degrees, the error rate of traditional facial feature matching rises sharply to 32%, severely impacting the continuous tracking of moving targets. Current technical solutions commonly suffer from feature extraction redundancy. Laboratory tests at Harbin Institute of Technology show that existing methods take an average of 220 milliseconds to process a single frame, severely limiting the ability to process multiple video streams in parallel. This makes it difficult to meet the surveillance system's demand for real-time, efficient processing of large amounts of video data. New technical solutions are urgently needed to overcome these challenges. Summary of the Invention

[0003] In response to the above-mentioned shortcomings of the existing technology, the present invention provides an identity recognition and monitoring method based on video images, which improves the identity recognition accuracy and processing efficiency in dynamic scenes by establishing a multimodal biometric fusion model and a video stream parallel processing architecture.

[0004] The specific technical solutions are as follows: A video image-based identity recognition and monitoring method, executed by a server deploying a TensorRT inference engine, includes: S1. Perform a depth-wise separable convolution operation on the input video stream, where the convolution kernel size d=3, and the calculation formula is, ,in is a depthwise convolution kernel of size 3×3, is a point-by-point convolution kernel of size 1×1; S2. Dynamically determine the generated face-gait fusion weights through the lighting quality evaluation function: , where L is the scene illumination value collected in real time; S3. Constructing cross-modal feature vectors And compare with the dynamic threshold library; the threshold is adjusted dynamically ,in represents the basic threshold, S represents the population density index; S4. Insert the spatiotemporal attention unit after the fourth residual block of the ResNet network ,in is the three-dimensional convolution kernel parameter matrix, is the Sigmoid function; S5. Implement three-pipeline parallel processing through double-buffered queues: the first queue performs CUDA-accelerated OpenCV image preprocessing, the second queue performs TensorRT inference, and the third queue completes the encrypted transmission of recognition results.

[0005] The above solution is further improved by forcing the infrared fill light acquisition module to be enabled in S2 when L<25lux. The weight is increased to 0.85.

[0006] Furthermore, in the above solution, the switching condition of the double buffer queue in S5 is: the queue full load difference ≥ 30% or the time slice rotation period reaches 500ms.

[0007] Furthermore, in the above scheme, the depthwise separable convolution module in S1 is specifically implemented using the bneck structure of MobileNetV3, and the depthwise convolution layer is followed by a Hardswish activation function.

[0008] Furthermore, in the above solution, in S2 , the scene illumination value L is acquired in real time by a metadata extraction module of the CMOS image sensor, with a sampling frequency of 30 Hz.

[0009] Furthermore, in the above scheme, the size of the three-dimensional convolution kernel parameter matrix of S4 is set to time dimension × width × height = 3 × 5 × 5, and the number of output channels of the fourth residual block is kept consistent with 512 dimensions.

[0010] Furthermore, the three pipeline parallel channels of S5 are implemented through the CUDA inter-stream explicit synchronization mechanism, and the GPU memory allocation of each pipeline adopts the minimum contention strategy.

[0011] Furthermore, the above scheme dynamically adjusts the threshold in S4 based on the historical matching success rate as the reference value, and the adjustment amount is ,in The number of consecutive matching failures.

[0012] Compared with the prior art, the present invention has the following beneficial effects: The various steps of the present invention are closely linked, collectively forming an efficient and accurate system that comprehensively improves monitoring performance in complex dynamic scenes. In the video data preprocessing stage, depthwise separable convolutions are applied, along with the MobileNetV3 bneck architecture and Hardswish activation function, significantly reducing computational effort. In the feature processing phase, a light quality assessment function dynamically determines the fusion weights of facial and gait features based on the real-time scene illumination. This allows for the rational fusion of these two features according to varying lighting conditions, paving the way for the construction of more discriminative cross-modal feature vectors. Furthermore, the recognition threshold is dynamically adjusted based on crowd density, effectively improving recognition accuracy in various complex scenarios, such as profiles, facial occlusions, low light conditions, and varying crowd densities. In the feature enhancement step, a spatiotemporal attention unit is inserted after the fourth residual block of the ResNet network. Using convolutional kernels of specific sizes and a sigmoid function, it focuses on key spatiotemporal features in the video. This operation filters out interfering information and enhances feature validity. In particular, it mitigates the large errors in traditional facial feature matching caused by changes in the monitoring viewpoint, further improving identity recognition accuracy. Finally, at the system operation level, double-buffered queues are used to achieve three-pipeline parallel processing, covering CUDA-accelerated OpenCV image preprocessing, TensorRT reasoning, and encrypted transmission of recognition results; reasonable queue switching conditions ensure a smooth processing flow, and parallel processing fully taps the potential of hardware, greatly reducing processing latency, and further meeting the strict requirements of surveillance for real-time and efficient processing of large amounts of video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the implementation steps of the present invention; Figure 2 This is a flow chart of an identity recognition monitoring method based on video images. DETAILED DESCRIPTION

[0014] The following is a further detailed description of the embodiments of the invention in conjunction with the accompanying drawings to make the objectives, technical solutions and technical effects of the invention more clearly presented.

[0015] like Figure 1-Figure 2 As shown, the present invention discloses a video image-based identity recognition and monitoring method, which is executed by a server deploying a TensorRT inference engine. The specific steps include: S1. Perform a depth-wise separable convolution operation on the input video stream, where the convolution kernel size d=3, and the calculation formula is, , the computational complexity is reduced to 1 / 9 compared to the standard convolution. middle It is a depth convolution kernel of size 3×3, which is used to perform convolution operations on each channel of the input independently; A 1×1 point-by-point convolution kernel is used to combine the output channels of the depthwise convolution. This reduces the computational complexity of video stream processing and speeds up operations without significantly reducing feature extraction capabilities, enabling the system to process video image data more efficiently and improving overall processing efficiency.

[0016] S2. Dynamically determine the generated face-gait fusion weights through the lighting quality evaluation function: , where L is the real-time scene illumination value. This allows for dynamic adjustment of the fusion weight based on lighting conditions. The reliability of facial and gait features varies under different lighting conditions. The above formula rationally fuses these two features based on actual lighting conditions, improving the accuracy of identity recognition in complex and dynamic lighting scenarios.

[0017] S3. Constructing cross-modal feature vectors And compare with the dynamic threshold library; the threshold is adjusted dynamically ,in represents the base threshold, and S represents the crowd density index. Fusion of multimodal features is more discriminative than single-modal features, improving recognition accuracy. Furthermore, the threshold is dynamically adjusted based on crowd density, adapting to varying crowd density scenarios and further enhancing recognition accuracy and robustness.

[0018] S4. Insert the spatiotemporal attention unit after the fourth residual block of the ResNet network ,in is the three-dimensional convolution kernel parameter matrix, is the Sigmoid function. In dynamic scenes, videos contain a large amount of spatiotemporal information. The spatiotemporal attention unit can focus on key features and reduce the influence of interference information, thereby improving the accuracy of identity recognition.

[0019] S5. A dual-buffered queue enables three-pipeline parallel processing: the first queue performs CUDA-accelerated OpenCV image preprocessing, the second queue performs TensorRT inference, and the third queue completes encrypted transmission of recognition results. This three-way parallelism, achieved through dual-buffered queues and CUDA flow control, allows OpenCV preprocessing, TensorRT inference, and encrypted transmission to be executed simultaneously, with a single-frame processing latency of ≤68ms.

[0020] This video-based identity recognition and monitoring solution runs on the TensorRT inference engine. For feature extraction, it utilizes depthwise separable convolutions to decompose standard convolutions into depthwise and pointwise convolutions. For example, the bneck structure of MobileNetV3 is used to implement the depthwise separable convolution module, with the convolution layer followed by a Hardswish activation function, significantly reducing computational complexity. A lighting quality assessment function dynamically determines the face-gait fusion weight based on real-time scene illumination values to accommodate feature fusion requirements under varying lighting conditions. A spatiotemporal attention unit is inserted into the ResNet network, utilizing a three-dimensional convolution kernel parameter matrix and a Sigmoid function to highlight key spatiotemporal features. CUDA's explicit inter-stream synchronization mechanism and a minimally competitive GPU memory allocation strategy are employed to achieve three-pipeline parallel processing.

[0021] Here, in S2, when L<25lux, the infrared fill light acquisition module is forced to be enabled, and The weight has been increased to 0.85. In low-light environments, standard image acquisition methods can blur facial features. Enabling infrared fill light can improve image acquisition. The weight of facial features is increased because infrared fill light makes facial features more reliable, thereby improving recognition accuracy in low-light scenarios.

[0022] In S5, the switching conditions for the double-buffered queue are: a queue load difference of ≥ 30% or a time slice rotation period of 500ms. This ensures smooth queue data processing, avoids data backlogs or delayed processing, ensures efficient parallel processing of the three pipelines, and improves overall system performance.

[0023] The depthwise separable convolution module in S1 is implemented using the bneck architecture of MobileNetV3, with the deep convolution layer followed by a Hardswish activation function. MobileNetV3's bneck architecture further optimizes convolution operations, reducing computational effort and model size. The Hardswish activation function enhances the model's nonlinear expressiveness, improving feature extraction while reducing computational costs, thereby enhancing the efficiency and accuracy of identity recognition.

[0024] Here, in S2, the scene illumination value L is acquired in real time by the metadata extraction module of the CMOS image sensor, with a sampling frequency of 30Hz. This allows for accurate real-time lighting information, providing a basis for dynamically adjusting the face-gait fusion weights. This allows the system to quickly adapt to lighting changes and ensures good recognition performance under various lighting conditions.

[0025] The three-dimensional convolution kernel parameter matrix size of S4 is set to 3×5×5 (time dimension × width × height), consistent with the output channel count of the fourth residual block, which is 512. By properly setting the convolution kernel parameters, we can effectively extract spatiotemporal features from videos while ensuring compatibility with the network structure, improving the effectiveness and accuracy of feature extraction and, consequently, enhancing identity recognition accuracy.

[0026] Here, the three parallel pipelines in the S5 are implemented through explicit CUDA inter-stream synchronization, and the GPU memory allocation for each pipeline adopts a minimum contention strategy. This explicit synchronization mechanism ensures the correctness of data exchange between pipelines, and the minimum contention strategy avoids memory contention, ensuring optimal allocation of GPU resources and improving the efficiency and stability of parallel processing.

[0027] The dynamic adjustment threshold in S4 is based on the historical matching success rate as the reference value, and the adjustment amount is ,in The number of consecutive match failures. This allows the threshold to be dynamically adjusted based on the actual matching situation. When the number of consecutive match failures increases, it indicates that the current threshold may not be appropriate. By adjusting the threshold, we can adapt to different scenarios and improve the success rate and accuracy of identity recognition.

[0028] The specific operations include: performing a depth-separable convolution operation on the input video stream, with a convolution kernel size of 3, and the calculation formula is: ; Through the lighting quality evaluation function Dynamically determine the face-gait fusion weight, when the scene illumination value L <25 lux When the infrared fill light acquisition module is enabled, the facial features fface The weight is increased to 0.85; constructing a cross-modal feature vector and the dynamic threshold calculated based on the population density index and the basic threshold The threshold is adjusted according to the historical matching success rate and the number of consecutive matching failures; a spatiotemporal attention unit is inserted after the fourth residual block of the ResNet network. , and the size of the three-dimensional convolution kernel parameter matrix is set to 3×5×5 (time dimension × width × height), which is consistent with the number of output channels of the fourth residual block; three-pipeline parallel processing is achieved through double-buffered queues. The first queue performs CUDA-accelerated OpenCV image preprocessing, the second queue performs TensorRT inference, and the third queue completes the encrypted transmission of recognition results. The switching condition of the double-buffered queue is that the queue full load is ≥30% or the time slice rotation period reaches 500ms.

[0029] In the above-mentioned scheme, the present invention addresses the problem of low recognition accuracy when the target object is in profile, with facial occlusion, or in low light conditions. It dynamically determines the face-gait fusion weight through a lighting quality assessment function. In low light conditions, it also forcibly enables the infrared fill light acquisition module and increases the weight of facial features. It also constructs a cross-modal feature vector and combines it with dynamic threshold comparison to fully utilize multi-dimensional information, effectively improving recognition accuracy in complex situations. To address the issues of poor walking posture feature capture accuracy and ineffective multimodal feature fusion, it utilizes depthwise separable convolution operations for video stream processing to reduce computational redundancy. Based on this, it accurately extracts facial and gait features and fuses them according to dynamic weights, compensating for the shortcomings of single-modal features, improving recognition accuracy and reliability, and addressing the problems caused by low gait feature capture accuracy and poor modal fusion. To address the high error rate of traditional facial feature matching when the monitoring angle of view exceeds 45 degrees, which affects the continuous tracking of moving targets, a spatiotemporal attention unit is inserted after the fourth residual block of the ResNet network to focus on key spatiotemporal features, reduce interference, and minimize errors caused by viewpoint changes, thereby facilitating continuous tracking of moving targets. To address the existing issues of redundant feature extraction, time-consuming single-frame image processing, and limited parallel processing capabilities for multi-channel video, this method leverages the bneck architecture of MobileNetV3 to implement a depthwise separable convolutional module and incorporates the Hardswish activation function to reduce computational complexity and model size. Furthermore, a dual-buffered queue enables three-pipeline parallel processing, including CUDA-accelerated OpenCV image preprocessing, TensorRT inference, and encrypted transmission of recognition results. This significantly reduces processing time and improves the ability to efficiently and effectively process large amounts of video data in real time, meeting the needs of surveillance systems.

[0030] Taking a surveillance scene test as an example, in a low-light environment, after enabling infrared fill light, combined with the adjusted facial feature weights, the accuracy of personnel recognition is effectively improved; when processing multiple videos, the three-pipeline parallel processing greatly shortens the processing time and can meet real-time processing needs; for dynamic scenes with different lighting, crowd density and monitoring angles, by dynamically adjusting fusion weights, thresholds and other operations, compared with traditional methods, this technology significantly improves the accuracy and processing efficiency of identity recognition, and reduces recognition errors caused by problems such as redundant feature extraction and poor multimodal fusion.

[0031] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of the patent application of the present invention. Any equivalent changes, equivalent replacements or modified changes within the technical spirit and principles suggested by the present invention should be included in the scope of patent protection covered by the present invention.

Claims

1. A video image-based identity recognition and monitoring method, executed by a server deploying a TensorRT inference engine, characterized in that: include: S1. Perform a depth-wise separable convolution operation on the input video stream, where the convolution kernel size d = 3, and the calculation formula is, Among them DW 3×3 is a depth convolution kernel of size 3×3, PW 1×1 is a point-by-point convolution kernel of size 1×1; S2. Dynamically determine the generated face-gait fusion weights through the lighting quality evaluation function: Where L is the scene illumination value collected in real time; S3. Construct cross-modal feature vector F fusion =α·f face +(1-α)·f gait And compare with the dynamic threshold library; where the dynamic adjustment threshold T = T base -0.15·log(S), where T base represents the basic threshold, S represents the population density index; S4. Insert the spatiotemporal attention unit after the fourth residual block of the ResNet network Where W t is the three-dimensional convolution kernel parameter matrix, σ is the Sigmoid function; S5. Implement three-pipeline parallel processing through double-buffered queues: the first queue performs CUDA-accelerated OpenCV image preprocessing, the second queue performs TensorRT inference, and the third queue completes the encrypted transmission of recognition results.

2. The method for identity recognition and monitoring based on video images according to claim 1, characterized in that: In S2, when L<25lux, the infrared fill light acquisition module is forced to be enabled, and the f gait The weight is increased to 0.

85.

3. The method for identity recognition and monitoring based on video images according to claim 1, characterized in that: In S5, the switching condition of the double-buffered queue is: the queue full load difference ≥ 30% or the time slice rotation period reaches 500ms.

4. The method for identity recognition and monitoring based on video images according to claim 1, characterized in that: The depthwise separable convolution module in S1 is specifically implemented using the bneck structure of MobileNetV3, and the depthwise convolution layer is followed by a Hardswish activation function.

5. The method for identity recognition and monitoring based on video images according to claim 1, characterized in that: In S2, the scene illumination value L is obtained in real time by the metadata extraction module of the CMOS image sensor with a sampling frequency of 30 Hz.

6. The method for identity recognition and monitoring based on video images according to claim 1, characterized in that: The size of the three-dimensional convolution kernel parameter matrix of S4 is set to time dimension × width × height = 3 × 5 × 5, and the number of output channels of the fourth residual block is kept consistent with 512 dimensions.

7. The method for identity recognition and monitoring based on video images according to claim 1, characterized in that: The three-pipeline parallel channels in S5 are implemented through the CUDA inter-stream explicit synchronization mechanism, and the GPU memory allocation for each pipeline adopts the minimum contention strategy.

8. The method for identity recognition and monitoring based on video images according to claim 1, characterized in that: The dynamic adjustment threshold in S4 is based on the historical matching success rate, and the adjustment amount is ΔT = -0.02·log2(N fait ), where N fait The number of consecutive matching failures.

Images