Human face super-resolution reconstruction and recognition method and system for video surveillance

Abstract

The present invention discloses a method and system for super-resolution reconstruction and recognition of faces used in video surveillance, and belongs to the field of deep learning super-resolution reconstruction of faces. The method of the present invention first obtains the original video stream collected in real time and preprocesses it, then locates the face position of the preprocessed video stream in the target area, then inputs the low-resolution face image into the trained face super-resolution reconstruction network for reconstruction, outputs the reconstructed super-resolution face image, and finally performs face recognition on the reconstructed super-resolution face image, outputs the face recognition result and visualizes it, and stores the recognized personnel for later auditing. The present invention aims to efficiently and accurately restore high-quality face images from low-quality face images through the powerful learning ability of the face super-resolution reconstruction network, thereby providing a clearer and more reliable input for the face recognition system, and improving the overall recognition accuracy and robustness.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning face super-resolution reconstruction, and in particular to a face super-resolution reconstruction and recognition method and system applied in video surveillance. Background Art

[0002] Face recognition technology plays a key role in many fields such as intelligent monitoring and identity authentication. Its core goal is to accurately identify the identity information of faces in pictures and videos. However, practical applications often encounter challenges of low-quality face images, such as low resolution, high noise or insufficient lighting, which seriously hinder the performance of face recognition systems. Face Super-Resolution (FSR) technology came into being, aiming to restore high-quality (HQ) face images from low-quality (LQ) input images to achieve face restoration, which is crucial to improving the accuracy and robustness of face recognition systems.

[0003] The development of deep learning technology has provided new ideas for solving the above problems. Face restoration methods based on deep learning use different technologies to build the most advanced networks, mainly focusing on the following aspects: different deep learning architectures, different facial priors, different loss functions, different learning strategies, etc. Deep learning models can learn complex feature representations from large amounts of data, thereby achieving better results in face super-resolution reconstruction tasks.

[0004] Although traditional face restoration methods have promoted technological development to a certain extent, their reconstruction effects are often difficult to achieve ideal results when faced with complex and changeable practical application scenarios. The problems and shortcomings of existing technologies mainly come from the following aspects: insufficient adaptability to complex scenes. Existing methods are difficult to maintain stable reconstruction effects when faced with non-restricted conditions such as lighting changes, diverse postures, and the presence of occlusions, especially at night or when noise interference is severe; computational efficiency and resource consumption. Super-resolution reconstruction usually requires a large amount of computing resources. How to reduce computational complexity and resource consumption while ensuring reconstruction quality is an urgent problem to be solved in practical applications; insufficient model robustness. In practical applications, the quality of input images varies, and noise, blur, and other problems often occur. Existing models lack the robustness to deal with these unfavorable factors; real-time and interactive requirements. In real-time application scenarios, such as video surveillance and face recognition, super-resolution reconstruction is required to have a faster processing speed and good interactive performance, which puts forward high requirements for algorithm optimization and hardware support.

[0005] Given the limitations of existing technologies, it is particularly necessary to innovatively design a face super-resolution reconstruction and recognition method based on deep learning. Summary of the invention

[0006] The purpose of the present invention is to solve the problems existing in the prior art and to provide a method and system for super-resolution reconstruction and recognition of faces for use in video surveillance, aiming to overcome the limitations of traditional methods and to efficiently and accurately restore high-quality face images from low-quality face images through the powerful learning ability of deep learning models, thereby providing a clearer and more reliable input for the face recognition system and improving the overall recognition accuracy and robustness.

[0007] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for super-resolution reconstruction and recognition of faces in video surveillance, which comprises the following steps:

[0009] S1, obtaining the original video stream collected by the image acquisition device in real time, and preprocessing the original video stream;

[0010] S2. In the target area, the face detection algorithm locates the face position of the preprocessed video stream and outputs a low-resolution face image with a detection frame for super-resolution reconstruction;

[0011] S3, input the low-resolution face image into the trained face super-resolution reconstruction network, perform bilinear interpolation on the low-resolution face image to obtain an interpolated face image, process the low-resolution face image through the first point-by-point convolution layer to obtain an initial fused feature map, process the initial fused feature map through the backbone network and the super-resolution reconstruction module in turn to obtain a final fused feature map, perform feature stacking on the interpolated face image and the final fused feature map, and output a reconstructed super-resolution face image;

[0012] During the training process of the face super-resolution reconstruction network, an original data set consisting of face images is first obtained, and the original data set is preprocessed to obtain a preprocessed data set, and the face super-resolution reconstruction network is trained on the preprocessed data set; the backbone network is formed by sequentially cascading a plurality of key modules and an offset hole convolution-self-attention mechanism module, each of which is formed by sequentially cascading an offset hole convolution-transposed residual block and a dual-aggregation self-attention mechanism module, and the dual-aggregation self-attention mechanism module is formed by sequentially cascading a spatial aggregation module and a channel aggregation module, the spatial aggregation module includes a spatial self-attention mechanism, an activation function with adjustable thresholds, and a gating mechanism feedback network, and the channel aggregation module includes a channel self-attention mechanism, an activation function with adjustable thresholds, and a gating mechanism feedback network; in the super-resolution reconstruction module, the feature map output by the backbone network is sequentially processed by an offset hole convolution-self-attention mechanism module, a second point-by-point convolution layer, a pixel shuffling operation, and a third point-by-point convolution layer to obtain a final fused feature map;

[0013] S4. Perform face recognition on the reconstructed super-resolution face image using a face recognition algorithm, output the face recognition result, visualize the face recognition result and store the recognized person.

[0014] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0015] As a preferred embodiment of the above-mentioned first aspect, in step S3, the specific processing flow in the offset dilated convolution-transposed residual block is as follows: the feature map input to the offset dilated convolution-transposed residual block is passed through the first offset dilated point-by-point convolution layer, the feature channel of the input feature map is separated to obtain a first feature map, the first feature map is sequentially passed through the first depth convolution layer, the second offset dilated point-by-point convolution layer, the second depth convolution layer, the third offset dilated point-by-point convolution layer and the third depth convolution layer to obtain a second feature map, the first feature map and the second feature map are added to obtain a third feature map, the input feature map is spliced ​​with the third feature map and the first feature map to obtain a first spliced ​​feature map, and the first spliced ​​feature map is processed through the fourth offset dilated point-by-point convolution layer to obtain the output of the offset dilated convolution-transposed residual block.

[0016] As a preference for the first aspect, the convolution kernel sizes of the first depth convolution layer, the second depth convolution layer, and the third depth convolution layer are all 3×3.

[0017] As a preferred embodiment of the first aspect above, in step S3, the specific processing flow in the offset hole convolution-self-attention mechanism module is as follows: the feature map input to the offset hole convolution-self-attention mechanism module is passed through the fifth offset hole point-by-point convolution layer, the feature channel of the input feature map is separated to obtain a fourth feature map, the fourth feature map is input into the multi-head self-attention mechanism module to obtain a fifth feature map, the fourth feature map and the fifth feature map are added to obtain a sixth feature map, the sixth feature map is input into the feedforward neural network to obtain a seventh feature map, the sixth feature map and the seventh feature map are added and then spliced ​​with the fourth feature map to obtain a second spliced ​​feature map, the second spliced ​​feature map is processed by the sixth offset hole point-by-point convolution layer to obtain the output of the offset hole convolution-self-attention mechanism module.

[0018] As a preferred embodiment of the above-mentioned first aspect, in step S3, the specific processing flow in the spatial aggregation module is as follows: the feature map input to the spatial aggregation module is subjected to layer normalization in sequence to obtain a first layer of normalized feature maps, and the first layer of normalized feature maps is then subjected to a feature projection layer to obtain a first spatial feature reshaping matrix; after processing the first spatial feature reshaping matrix, a spatial query matrix, a spatial key matrix and a spatial value matrix are obtained, the spatial query matrix and the spatial key matrix are matrix multiplied and processed by a Softmax activation function using a spatial self-attention mechanism, and the first spatial feature map is output, the spatial value matrix and the first spatial feature map are matrix multiplied to obtain a second spatial feature reshaping matrix, and the second spatial feature reshaping matrix is ​​sequentially subjected to a first point-by-point convolution layer, a threshold adjustable activation function, a second point-by-point convolution layer, and a Sigmoid activation function to obtain a first spatial weight map; the first spatial feature reshaping matrix is ​​subjected to a first depth separable The second spatial feature map is processed by the convolution layer, and the second spatial feature map is output. The second spatial feature map is processed by the third point-by-point convolution layer, the threshold adjustable activation function, the fourth point-by-point convolution layer, and the Sigmoid activation function in sequence to obtain the second spatial weight map; the second spatial weight map is element-wise multiplied with the corresponding elements of the second spatial feature reshaping matrix to obtain the first spatial weighted feature, the first spatial weight map is element-wise multiplied with the corresponding elements of the second spatial feature map to obtain the second spatial weighted feature, the first spatial weighted feature and the second spatial weighted feature are added and passed through a feature projection layer to obtain the third spatial feature reshaping matrix, the first layer normalized feature map is added to the third spatial feature reshaping matrix to obtain the third spatial feature map, the third spatial feature map is processed by layer normalization and the gating mechanism feedback network in sequence to obtain the fourth spatial feature map, and the third spatial feature map and the fourth spatial feature map are added as the output of the spatial aggregation module.

[0019] As a preferred embodiment of the above-mentioned first aspect, in step S3, the specific processing flow in the channel aggregation module is as follows: the feature map input to the channel aggregation module is sequentially subjected to layer normalization to obtain a second layer of normalized feature maps, and the second layer of normalized feature maps is then subjected to a feature projection layer to obtain a first channel feature reshaping matrix; after processing the first channel feature reshaping matrix, a channel query matrix, a channel key matrix, and a channel value matrix are obtained, the channel query matrix and the channel key matrix are matrix multiplied and processed by a Softmax activation function using a channel self-attention mechanism, and the first channel feature map is output, the channel value matrix and the first channel feature map are matrix multiplied to obtain a second channel feature reshaping matrix, and the second channel feature reshaping matrix is ​​sequentially subjected to a fifth point-by-point convolution layer, a threshold adjustable activation function, a sixth point-by-point convolution layer, and a Sigmoid activation function to obtain a first channel weight map; the first channel feature reshaping matrix is ​​subjected to a second depth separable The convolution layer is processed to output the second channel feature map, and the second channel feature map is processed by the seventh point-by-point convolution layer, the threshold adjustable activation function, the eighth point-by-point convolution layer, and the Sigmoid activation function in sequence to obtain the second channel weight map; the second channel weight map is element-wise multiplied with the corresponding elements of the second channel feature reshaping matrix to obtain the first channel weighted feature, the first channel weight map is element-wise multiplied with the corresponding elements of the second channel feature map to obtain the second channel weighted feature, the first channel weighted feature and the second channel weighted feature are added and passed through a feature projection layer to obtain the third channel feature reshaping matrix, the second layer normalized feature map is added to the third channel feature reshaping matrix to obtain the third channel feature map, the third channel feature map is processed by layer normalization and the gating mechanism feedback network in sequence to obtain the fourth channel feature map, and the third channel feature map and the fourth channel feature map are added as the output of the channel aggregation module.

[0020] As a preferred embodiment of the above-mentioned first aspect, the specific processing flow in the gating mechanism feedback network is as follows: after the feature map input to the gating mechanism feedback network passes through a linear projection layer, a threshold-adjustable activation function, and a feature separation operation along the channel dimension in sequence, the first part obtained by feature separation is processed by a third depth-separable convolution layer to obtain an eighth feature map, the eighth feature map and the corresponding elements of the second part obtained by feature separation are element-wise multiplied to obtain a ninth feature map, and the ninth feature map is passed through a linear projection layer to obtain the output of the gating mechanism feedback network.

[0021] As a preferred embodiment of the first aspect, the activation function with adjustable threshold The functional form is:

[0022]

[0023] In the formula, Indicates an adjustable threshold; Represents the characteristic value of the input activation function; Indicates the maximum value operation.

[0024] As a preferred embodiment of the first aspect, the number of key modules is a positive integer and a multiple of 3.

[0025] In a second aspect, the present invention provides a face super-resolution reconstruction and recognition system for use in video surveillance, comprising:

[0026] The data acquisition module is used to acquire the original video stream acquired by the image acquisition device in real time and pre-process the original video stream;

[0027] The detection module is used to locate the face position of the preprocessed video stream in the target area using a face detection algorithm, and output a low-resolution face image with a detection frame for super-resolution reconstruction;

[0028] A super-resolution reconstruction module is used to input a low-resolution face image into a trained face super-resolution reconstruction network, perform bilinear interpolation on the low-resolution face image to obtain an interpolated face image, process the low-resolution face image through a first point-by-point convolutional layer to obtain an initial fused feature map, process the initial fused feature map through a backbone network and a super-resolution reconstruction module in sequence to obtain a final fused feature map, perform feature stacking on the interpolated face image and the final fused feature map, and output a reconstructed super-resolution face image;

[0029] During the training process of the face super-resolution reconstruction network, an original data set consisting of face images is first obtained, and the original data set is preprocessed to obtain a preprocessed data set, and the face super-resolution reconstruction network is trained on the preprocessed data set; the backbone network is formed by sequentially cascading a plurality of key modules and an offset hole convolution-self-attention mechanism module, each of which is formed by sequentially cascading an offset hole convolution-transposed residual block and a dual-aggregation self-attention mechanism module, and the dual-aggregation self-attention mechanism module is formed by sequentially cascading a spatial aggregation module and a channel aggregation module, the spatial aggregation module includes a spatial self-attention mechanism, an activation function with adjustable thresholds, and a gating mechanism feedback network, and the channel aggregation module includes a channel self-attention mechanism, an activation function with adjustable thresholds, and a gating mechanism feedback network; in the super-resolution reconstruction module, the feature map output by the backbone network is sequentially processed by an offset hole convolution-self-attention mechanism module, a second point-by-point convolution layer, a pixel shuffling operation, and a third point-by-point convolution layer to obtain a final fused feature map;

[0030] The recognition module is used to perform face recognition on the reconstructed super-resolution face image using a face recognition algorithm, output the face recognition result, visualize the face recognition result and store the recognized person.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The method of the present invention utilizes the existing fixed-point camera infrastructure and, through advanced algorithms, automatically completes the super-resolution reconstruction and face recognition of the face in the monitoring screen. Specifically, the present invention realizes intelligent early warning of personnel activities in the designated area through real-time analysis of the video stream. The super-resolution reconstruction and recognition method of the face of the present invention can significantly improve the clarity of the face image in the video, and can generate high-precision facial details even under low-resolution conditions. This not only enhances the visual effect of the image, but also provides high-quality basic data for subsequent face recognition. Through the application of deep learning technology, the present invention can recover more details from the low-resolution image, making the reconstructed face image more realistic. In terms of face recognition, the present invention adopts an advanced deep learning algorithm, which can accurately identify the identity of an individual in the reconstructed high-resolution image. By learning a large number of facial features, the present invention has a high recognition accuracy and robustness, and can work reliably under complex background and lighting conditions. The method not only provides a convenient, efficient and open information management platform for the security management of the perimeter of a designated place, but also significantly improves its intelligence level through the application of deep learning technology to ensure efficient operation.

[0033] In the super-resolution reconstruction of face images, the present invention designs a face super-resolution reconstruction network, the key points of which are:

[0034] (1) A threshold-adjustable activation function ReLU-T was constructed to weaken non-primary brightness information, which has a certain improvement effect on facial super-resolution reconstruction;

[0035] (2) We constructed an offset atrous convolution-transposed residual block, which enhances local feature aggregation through offset atrous convolution, introduces nonlinearity through the ReLU-T activation function, and solves the gradient vanishing problem through residual connection.

[0036] (3) An offset atrous convolution-self-attention mechanism module is constructed, aiming to improve efficiency by selectively calculating the self-attention of some features.

[0037] (4) Construct a dual-aggregation self-attention mechanism module, which aggregates spatial and channel features in a dual way, enhances the modeling ability of the self-attention mechanism, and realizes feature aggregation within the attention mechanism while maintaining low complexity and model size. The spatial attention mechanism can enhance the spatial expression of each feature map; the channel attention mechanism is used to better build the dependency between channels and expand the receptive field, thereby assisting in capturing spatial features. Spatial and channel information flow between continuous self-attention mechanisms to achieve inter-block feature aggregation. In addition, the dynamic modulation of the spatial dimension of the feature introduces channel information, while the dynamic modulation of the channel dimension of the feature introduces spatial information, realizing dynamic modulation of the spatial dimension and channel dimension of the feature, promoting better coupling between local (convolution) and global (attention), so that complementary channel information improves channel modeling capabilities, while spatial interaction enhances feature representation capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the steps of the method of the present invention;

[0039] Figure 2 A schematic diagram of a back-end implementation flow of the method of the present invention when applied;

[0040] Figure 3 A schematic diagram of the face super-resolution reconstruction network architecture of the present invention;

[0041] Figure 4 A schematic diagram of an activation function with adjustable threshold value according to the present invention;

[0042] Figure 5 Schematic diagram of the architecture of the offset dilated convolution-transposed residual block of the present invention;

[0043] Figure 6 Schematic diagram of the architecture of the offset hole convolution-self-attention mechanism module of the present invention;

[0044] Figure 7 Schematic diagram of the architecture of the dual-aggregation self-attention mechanism module of the present invention;

[0045] Figure 8 It is a schematic diagram of the architecture of the gating mechanism feedback network of the present invention;

[0046] Fig. 9 It is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0048] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0049] The present invention provides a method for super-resolution reconstruction and recognition of faces in video surveillance. The method uses advanced deep learning technology to process image frames in video streams, realizes efficient face detection, super-resolution reconstruction and face recognition functions, and can also perform visual display, recording and early warning.

[0050] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned face super-resolution reconstruction and recognition method applied in video surveillance includes the following steps S1 to S4. The specific implementation process is described in detail below.

[0051] S1. Obtain the original video stream captured by the image acquisition device in real time, and pre-process the original video stream.

[0052] Step S1 of the present invention is video stream access and preprocessing. In this step, an image acquisition device (such as a fixed camera) captures a real-time video stream and performs preprocessing steps such as format conversion and compression processing.

[0053] S2. In the target area, the face detection algorithm locates the face position of the preprocessed video stream and outputs a low-resolution face image with a detection frame for super-resolution reconstruction.

[0054] Step S2 of the present invention is face detection. In this step, the face detection algorithm quickly and accurately locates the face position in the preprocessed video stream and draws a detection frame. The face detection algorithm used here can adopt an existing face detection algorithm as long as it can realize face detection and output face detection results, which is not limited in the present invention.

[0055] S3, input the low-resolution face image into the trained face super-resolution reconstruction network, perform bilinear interpolation on the low-resolution face image to obtain an interpolated face image, process the low-resolution face image through the first point-by-point convolution layer to obtain an initial fused feature map, process the initial fused feature map through the backbone network and the super-resolution reconstruction module in turn to obtain a final fused feature map, perform feature stacking on the interpolated face image and the final fused feature map, and output a reconstructed super-resolution face image;

[0056] During the training process of the face super-resolution reconstruction network, an original data set consisting of face images is first obtained, and the original data set is preprocessed to obtain a preprocessed data set, and the face super-resolution reconstruction network is trained on the preprocessed data set;

[0057] The backbone network is formed by sequentially cascading multiple key modules and an offset dilated convolution-self-attention mechanism module, each of which is formed by sequentially cascading an offset dilated convolution-transposed residual block and a dual-aggregation self-attention mechanism module, the dual-aggregation self-attention mechanism module is formed by sequentially cascading a spatial aggregation module and a channel aggregation module, the spatial aggregation module includes a spatial self-attention mechanism, an activation function with adjustable thresholds, and a gating mechanism feedback network, and the channel aggregation module includes a channel self-attention mechanism, an activation function with adjustable thresholds, and a gating mechanism feedback network;

[0058] In the super-resolution reconstruction module, the feature map output by the backbone network is processed in sequence by an offset hole convolution-self-attention mechanism module, a second point-by-point convolution layer, a pixel shuffling operation, and a third point-by-point convolution layer to obtain the final fused feature map.

[0059] It should be noted that in step S3, the present invention designs a face super-resolution reconstruction network, and its architecture is as follows: Figure 3As shown in the figure. The network combines the local feature extraction capability of convolutional neural network (CNN) and the global context modeling capability of Transformer, and has the advantages of convolutional neural network and Transformer architecture, achieving efficient and effective face super-resolution reconstruction. Its core modules include point-by-point convolution, backbone network and super-resolution reconstruction module. Specifically, the backbone network adopts a dual-aggregation self-attention mechanism module, and the spatial reduction and channel increase operations are separated to achieve efficient downsampling. The dual-aggregation self-attention mechanism module introduces nonlinearity through the threshold-adjustable activation function ReLU-T, and solves the gradient vanishing problem through residual connection. The backbone network also adopts the offset hole convolution-self-attention mechanism module and the offset hole convolution-transposed residual block. These two modules use the offset hole point-by-point convolution operation to divide the input features into several parts for spatial offset to enhance local aggregation without adding additional computational cost. Among them, the offset hole convolution-transposed residual block enhances local feature aggregation by offset hole point-by-point convolution; the offset hole convolution-self-attention mechanism module improves efficiency by selectively calculating the self-attention of some features, and then processes it through the multi-head self-attention mechanism module (MHSA) and the feedforward neural network (FFN), and reorganizes the features through an offset hole point-by-point convolution. The super-resolution reconstruction module uses pixel shuffling to increase channels and spatially upsamples by rearranging channel elements to expand the image size while maintaining the minimum computational complexity. Finally, the face image after bilinear interpolation is residually connected with the feature map output by the super-resolution reconstruction module to achieve feature stacking and reconstruct the super-resolution face image.

[0060] The following is a detailed introduction to the core components of the face super-resolution reconstruction network for those skilled in the art to understand.

[0061] In step S3, the specific processing flow in the offset dilated convolution-transposed residual block is as follows: the feature map input to the offset dilated convolution-transposed residual block passes through the first offset dilated point-by-point convolution layer, separates the feature channels of the input feature map, and obtains a first feature map; the first feature map passes through the first depth convolution layer, the second offset dilated point-by-point convolution layer, the second depth convolution layer, the third offset dilated point-by-point convolution layer, and the third depth convolution layer in sequence to obtain a second feature map; the first feature map and the second feature map are added to obtain a third feature map; the input feature map is spliced ​​with the third feature map and the first feature map to obtain a first spliced ​​feature map; the first spliced ​​feature map is processed by the fourth offset dilated point-by-point convolution layer to obtain the output of the offset dilated convolution-transposed residual block.

[0062] In this embodiment, the convolution kernel sizes of the first depth convolution layer, the second depth convolution layer, and the third depth convolution layer are all 3×3.

[0063] In this embodiment, the implementation of the deep convolution layer belongs to the prior art and will not be described in detail. The shifted dilated convolution layer is implemented by sequentially performing shifted dilated convolution and point-by-point convolution (1×1 convolution). The shifted dilated convolution is implemented by superimposing a shift operation (Shift) on the dilated convolution, and the point-by-point convolution further aggregates features; among them, the dilated convolution, the shift operation, and the point-by-point convolution belong to the prior art and will not be described in detail.

[0064] like Figure 5 As shown in Figure 1, the offset atrous convolution-transposed residual block is a specific module in the face super-resolution reconstruction network for local feature aggregation. Specifically, the offset atrous point-by-point convolution operation in this module enhances the aggregation ability of local features by offsetting the input features in space. Among them, the atrous convolution (also known as dilated convolution) allows the face super-resolution reconstruction network to capture a wider receptive field without increasing the number of parameters, and the offset operation further enhances this ability by sampling features at different locations to improve the representation of features. In addition, the addition of residual connections in this module solves the gradient vanishing problem in the training of the face super-resolution reconstruction network. The residual connection allows the face super-resolution reconstruction network to directly skip some layers and add the input directly to the output of the subsequent layers, which can help the gradient propagate more effectively in the face super-resolution reconstruction network, thereby training a deeper face super-resolution reconstruction network. The design of this module is particularly suitable for applications that require fine local features and deep network structures, such as face super-resolution reconstruction.

[0065] In step S3, the specific processing flow in the offset hole convolution-self-attention mechanism module is as follows: the feature map input to the offset hole convolution-self-attention mechanism module passes through the fifth offset hole point-by-point convolution layer, separates the feature channels of the input feature map, and obtains a fourth feature map. The fourth feature map is input into the multi-head self-attention mechanism module (MHSA) to obtain a fifth feature map. The fourth feature map and the fifth feature map are added to obtain a sixth feature map. The sixth feature map is input into the feedforward neural network (FFN) to obtain a seventh feature map. The sixth feature map and the seventh feature map are added and then spliced ​​with the fourth feature map to obtain a second spliced ​​feature map. After the second spliced ​​feature map is processed by the sixth offset hole point-by-point convolution layer, the output of the offset hole convolution-self-attention mechanism module is obtained.

[0066] In this embodiment, the implementation of the multi-head self-attention mechanism module and the feedforward neural network belongs to the prior art and will not be described in detail.

[0067] like Figure 6As shown in the figure, the offset hole convolution-self-attention mechanism module combines the offset hole point-by-point convolution and the self-attention mechanism to improve the efficiency and performance of the face super-resolution reconstruction network. The features and functions of this module are as follows:

[0068] 1) Improved efficiency

[0069] The offset atrous convolution-self-attention mechanism module improves computational efficiency by selectively calculating self-attention for some features instead of all features. This approach can reduce unnecessary calculations, especially when processing high-resolution images.

[0070] 2) Local feature aggregation

[0071] The offset atrous convolution-self-attention mechanism module utilizes the offset atrous point-by-point convolution operation to enhance the aggregation ability of local features by sampling features at different positions.

[0072] 3) Self-attention processing

[0073] The offset atrous convolution-self-attention mechanism module uses a multi-head self-attention mechanism module. After selecting some features, it uses the multi-head self-attention mechanism to process these features, capture the feature relationship between different positions, and enhance the representation ability of the features.

[0074] 4) Feature transformation

[0075] After being processed by the multi-head self-attention mechanism module, the offset atrous convolution-self-attention mechanism module uses a feedforward neural network to further transform the features. The feedforward neural network usually contains linear transformations and activation functions to learn nonlinear combinations of features.

[0076] 5) Feature Recombination

[0077] After being processed by MHSA and FFN, the offset atrous convolution-self-attention mechanism module uses an offset atrous point-by-point convolution layer to reorganize the features, which helps to integrate information from different feature channels.

[0078] In general, the offset atrous convolution-partial self-attention mechanism is a module designed to improve the efficiency and performance of the face super-resolution reconstruction network. This module combines the local feature aggregation ability of the offset atrous point-by-point convolution and the global context modeling ability of the self-attention mechanism to improve the face super-resolution reconstruction network's ability to understand and represent facial image features, while improving computational efficiency through selective calculation and point-by-point convolution; by combining the processing of the multi-head self-attention mechanism module and the feedforward neural network, as well as feature reorganization, efficient and in-depth processing of features is achieved. This mechanism is particularly suitable for tasks that require consideration of both local and global information, such as image super-resolution reconstruction.

[0079] In step S3, Figure 7 As shown in the figure, the specific processing flow in the spatial aggregation module is as follows: the feature map input to the spatial aggregation module is sequentially normalized by layers to obtain a first layer of normalized feature maps, and the first layer of normalized feature maps is then passed through a feature projection layer to obtain a first spatial feature reshaping matrix; after processing the first spatial feature reshaping matrix, a spatial query matrix, a spatial key matrix, and a spatial value matrix are obtained, and the spatial query matrix and the spatial key matrix are matrix-multiplied and processed by a Softmax activation function using a spatial self-attention mechanism to output the first spatial feature map, and the spatial value matrix and the first spatial feature map are matrix-multiplied to obtain a second spatial feature reshaping matrix, and the second spatial feature reshaping matrix is ​​sequentially processed by a first point-by-point convolution layer, a threshold adjustable activation function, a second point-by-point convolution layer, and a Sigmoid activation function to obtain a first spatial weight map; the first spatial feature reshaping matrix is ​​processed by a first depth-separable convolution layer, Output the second spatial feature map, and process the second spatial feature map sequentially through the third point-by-point convolution layer, the threshold adjustable activation function, the fourth point-by-point convolution layer, and the Sigmoid activation function to obtain the second spatial weight map; perform element-wise multiplication of the second spatial weight map and the corresponding elements of the second spatial feature reshaping matrix to obtain the first spatial weighted feature, perform element-wise multiplication of the first spatial weight map and the corresponding elements of the second spatial feature map to obtain the second spatial weighted feature, add the first spatial weighted feature and the second spatial weighted feature and pass through a feature projection layer to obtain the third spatial feature reshaping matrix, add the first layer normalized feature map and the third spatial feature reshaping matrix to obtain the third spatial feature map, and process the third spatial feature map sequentially through layer normalization and the gating mechanism feedback network to obtain the fourth spatial feature map, and add the third spatial feature map and the fourth spatial feature map as the output of the spatial aggregation module.

[0080] In step S3, the channel aggregation module is implemented by replacing the spatial self-attention mechanism in the spatial aggregation module with the channel self-attention mechanism, and the input of the channel aggregation module is the output of the spatial aggregation module.

[0081] like Figure 7As shown in the figure, the specific processing flow in the channel aggregation module is as follows: the feature map input to the channel aggregation module is sequentially normalized by layers to obtain a second layer of normalized feature maps, and the second layer of normalized feature maps is then passed through a feature projection layer to obtain a first channel feature reshaping matrix; after processing the first channel feature reshaping matrix, a channel query matrix, a channel key matrix, and a channel value matrix are obtained, and the channel query matrix and the channel key matrix are matrix multiplied and processed by a Softmax activation function using a channel self-attention mechanism, and the first channel feature map is output, and the channel value matrix and the first channel feature map are matrix multiplied to obtain a second channel feature reshaping matrix, and the second channel feature reshaping matrix is ​​sequentially processed by a fifth point-by-point convolution layer, a threshold adjustable activation function, a sixth point-by-point convolution layer, and a Sigmoid activation function to obtain a first channel weight map; the first channel feature reshaping matrix is ​​processed by a second depth-separable convolution layer, Output the second channel feature map, and process the second channel feature map in turn through the seventh point-by-point convolution layer, the threshold adjustable activation function, the eighth point-by-point convolution layer, and the Sigmoid activation function to obtain the second channel weight map; perform element-wise multiplication of the second channel weight map and the corresponding elements of the second channel feature reshaping matrix to obtain the first channel weighted feature, perform element-wise multiplication of the first channel weight map and the corresponding elements of the second channel feature map to obtain the second channel weighted feature, add the first channel weighted feature and the second channel weighted feature and pass through a feature projection layer to obtain the third channel feature reshaping matrix, add the second layer normalized feature map and the third channel feature reshaping matrix to obtain the third channel feature map, and process the third channel feature map in turn through layer normalization and the gating mechanism feedback network to obtain the fourth channel feature map, and add the third channel feature map and the fourth channel feature map as the output of the channel aggregation module.

[0082] In step S3, the specific processing flow in the gating mechanism feedback network is as follows: after the feature map input to the gating mechanism feedback network passes through a linear projection layer, a threshold-adjustable activation function, and a feature separation operation along the channel dimension in sequence, the first part obtained by feature separation is processed by a third depth-separable convolution layer to obtain an eighth feature map, and the corresponding elements of the eighth feature map and the second part obtained by feature separation are element-wise multiplied to obtain a ninth feature map. After the ninth feature map passes through the linear projection layer, the output of the gating mechanism feedback network is obtained.

[0083] like Figure 8As shown in the figure, the gated mechanism feedback network contains a depth-separable convolution layer and element-wise multiplication (element-by-element multiplication). At the same time, the feature map is evenly divided into two parts along the channel dimension, and sent to the depth-separable convolution layer and the element-wise multiplication bypass respectively, so as to reduce channel redundancy. This operation can also effectively reduce the computational complexity and improve the operating efficiency of the face super-resolution reconstruction network. In addition, the gated mechanism feedback network introduces nonlinear spatial information through an activation function with adjustable thresholds, supplements the information processing capability of the feedforward neural network, and enhances the generalization capability of the face super-resolution reconstruction network. These characteristics make the gated mechanism feedback network have important application value in the face super-resolution reconstruction network.

[0084] It should be noted that in the dual-aggregation self-attention mechanism module, both the spatial aggregation module and the channel aggregation module use the threshold-adjustable activation function ReLU-T, denoted as , its function form is:

[0085]

[0086] In the formula, Indicates an adjustable threshold; Represents the characteristic value of the input activation function; Indicates the maximum value operation.

[0087] In the present invention, the threshold-adjustable activation function introduces nonlinear features, enhances the ability of the face super-resolution reconstruction network to learn complex objects, complex data and arbitrary function mappings, and enables the face super-resolution reconstruction network to learn more complex function mappings. ReLU (Rectified Linear Unit) is a commonly used activation function that sets all negative values ​​to 0, and the ReLU-T designed in the present invention is a variant of ReLU, which includes an adjustable threshold for further controlling the strength of nonlinearity.

[0088] In this embodiment, considering that the face image is affected by light, reducing the non-main brightness information has a certain improvement effect on the facial super-resolution reconstruction. Inspired by this, this embodiment designs the above threshold adjustable activation function through experimental verification, such as Figure 4As shown. When a low-resolution (LR) face image is input into the face super-resolution reconstruction network, in order to speed up the convergence of the training face super-resolution reconstruction network, the low-resolution face image will be converted into a value in the range of 0-1, that is, the maximum value in the low-resolution face image will not exceed 1. Therefore, when applying an activation function with an adjustable threshold, the adjustable threshold needs to be initialized to 1. In addition, the adjustable threshold is optimized by periodically evaluating the cumulative loss. When the cumulative loss increases, the adjustable threshold will decrease accordingly, and when the cumulative loss decreases, the adjustable threshold will increase accordingly, and finally a threshold that minimizes the cumulative loss is obtained.

[0089] It should be noted that in step S3 of the present invention, the number N of key modules included in the backbone network can be 3, 6, 9, etc., that is, N can be a positive integer and can be a multiple of 3. However, the larger the value of N, the deeper the face super-resolution reconstruction network is, the higher the accuracy is, but the larger the number of parameters is, and the higher the inference cost is. Therefore, the specific value of the key module can be selected by those skilled in the art according to the actual situation, and is not limited in the present invention.

[0090] Therefore, the face super-resolution reconstruction network of the present invention has the following characteristics: by offsetting the hole point-by-point convolution and pixel shuffling operations, spatial aggregation and upsampling are achieved without increasing too much computational burden; residual connection and ReLU-T activation function are used to improve the training efficiency and effect of the face super-resolution reconstruction network; the multi-head self-attention mechanism module and the dual-aggregation self-attention mechanism module are used to improve the processing ability of the face super-resolution reconstruction network for global information; in the downsampling process, the spatial reduction and channel increase operations are separated to achieve efficient downsampling. Specifically, the pixel shuffling operation is used for channel increase, which performs spatial upsampling by rearranging channel elements, expanding the image size while maintaining the minimum computational complexity, and the offset hole point-by-point convolution operation is used for spatial reduction, dividing the input features into several parts for spatial offset, and enhancing local aggregation without increasing additional computational cost.

[0091] It should be noted that in step S3 of the present invention, the original data set can be generated by existing public data sets, such as public data sets DIV2K, CelebA, CASIA-WebFace and LFW (Labeled Faces in the Wild), or web crawler tools can be used to capture face images from social media websites, news websites, blogs, etc., but the use of web crawler tools to obtain face images requires attention to copyright issues and privacy protection. In addition to these two methods, users can also collect face images by themselves through personal photography, cooperative projects, academic research, etc. to obtain more customized face images, but relevant ethical standards need to be followed, and it is also very important to obtain the explicit consent of the subject.

[0092] The following is an introduction to the above four public datasets. DIV2K is a commonly used high-quality image dataset, which contains a total of 1,000 high-definition images (2K resolution), of which 800 are used as training sets, 100 are used as validation sets, and 100 are used as test sets. Although DIV2K is not specifically for faces, suitable face images can still be screened out from it. CelebA is a large-scale face dataset containing more than 200,000 images, which is very suitable for research on face recognition and super-resolution reconstruction. CASIA-WebFace is a large-scale facial image dataset containing 494,413 images of 10,308 people, which can be used for face super-resolution reconstruction. LFW is a widely used facial recognition benchmark dataset, which contains 13,233 images of 5,749 different individuals.

[0093] In step S3 of the present invention, the Face Super-Resolution (FSR) task usually requires a large amount of high-quality training data (face images) to learn the mapping relationship from low resolution (LR) to high resolution (HR). However, since face images in the real world have diverse characteristics (such as different lighting conditions, expression changes, posture changes, etc.), it is often difficult to cover all situations relying solely on original face images. Therefore, data augmentation has become one of the important means to improve the generalization ability of the model.

[0094] The following are some common image enhancement methods in face super-resolution reconstruction tasks: 1) Geometric transformation, such as flip, rotation, and scaling. Flipping can be horizontal or vertical flipping of the image to increase data diversity; rotation refers to randomly rotating a certain angle to simulate changes in different perspectives; scaling refers to randomly scaling the image to simulate the effect of shooting at different distances. 2) Crop & Patch, such as Random Crop, Center Crop, and Patch Extraction. Random cropping refers to randomly selecting a part from the original image as input; center cropping refers to cropping a fixed-size area from the center of the image; patch extraction refers to extracting small areas from a high-resolution image to reduce computational costs. 3) Color transformation, such as brightness adjustment, contrast adjustment, and color jittering. Brightness adjustment refers to changing the brightness value of the image; contrast adjustment refers to changing the contrast of the image; color jittering refers to randomly changing the color channels of the image. 4) Noise addition, such as Gaussian Noise, Salt and Pepper Noise, and Blur. Gaussian noise refers to adding Gaussian noise to the image; Salt and Pepper Noise refers to randomly adding black and white dots to the image; Blur refers to using a blur filter (such as Gaussian blur) to simulate the camera lens effect. 5) Synthetic transformation. For example, Mixed Effects, which combines multiple transformation methods, such as rotating first and then scaling. 6) Occlusion. For example, Random Occlusion, which randomly occludes parts of the image to simulate occlusion. 7) Mixed Samples. For example, Style Transfer, which merges image features of different styles into the current image.

[0095] In addition to the enhancement methods listed above, the following data enhancement techniques can also be used: 1. Generative Adversarial Network (GAN). Use GAN to generate more diverse images, especially in the absence of a large amount of real data. 2. Simulate lighting conditions. Simulate different lighting conditions, such as daytime, nighttime, indoor lighting, etc., to increase the robustness of the model. 3. Expression and posture changes. Use synthesis methods or deep learning models to generate facial images with different expressions and postures.

[0096] Through the above methods, images can be effectively enhanced to help the model better learn the features of the face and improve the generalization and robustness of the model in practical applications. However, when implementing the above image enhancement, the following points should be noted: 1) Maintain authenticity: Try to keep the enhanced image close to the natural image and avoid introducing too many artificial traces. 2) Balance: Ensure that the proportion of various transformations in the enhanced data set is balanced to avoid a certain type of transformation being too frequent, causing the model to be biased in a certain direction. 3) Evaluate the impact: Regularly evaluate the impact of data enhancement on model performance to ensure that the gain is greater than the potential negative impact.

[0097] In step S3 of this embodiment, the preprocessing of the original data set specifically includes: 1) cleaning and filtering. Removing repeated, blurred, and face images with large angle deviations to ensure that the face image contains a complete face area. 2) Generating low-resolution face images, generating corresponding low-resolution face images from high-resolution face images, and using them to train the face super-resolution reconstruction network. Specifically, methods such as bicubic interpolation and Gaussian blurring can be used to reduce the resolution of face images.

[0098] In step S3 of this embodiment, when training the face super-resolution reconstruction network, the preprocessed data set is divided into a training set, a validation set, and a test set according to a ratio of 70%, 15%, and 15%. The ratio selection here is only a preferred method and can also be adjusted according to specific circumstances.

[0099] In step S3 of this embodiment, when training the face super-resolution reconstruction network, the hyperparameters are set as follows:

[0100] 1) Learning Rate: It is usually initialized to a small value, such as A lower learning rate can avoid large fluctuations in the early stages of training. As training progresses, the learning rate is gradually reduced to achieve learning rate decay and speed up training. The learning rate can be adjusted by Cosine Annealing in a cosine-periodic manner.

[0101] 2) Batch Size: It depends on the GPU memory size and image size. Common batch size settings are 16, 32 or 64. It should be noted that a larger batch size can speed up training, but it may also cause memory overflow.

[0102] 3) Optimizer: Use Adam optimizer.

[0103] 4) Number of training epochs: It depends on the size and complexity of the dataset and is usually set to 50 to 200. The Early Stopping strategy decides when to stop training based on the performance on the validation set.

[0104] 5) Weight Decay: This is achieved using L2 regularization, which helps prevent overfitting of the face super-resolution reconstruction network. Common weight decay coefficients are .

[0105] 6) Model Checkpoint frequency: The model can be saved every M epochs, or the best face super-resolution reconstruction network can be saved based on the performance on the validation set.

[0106] 7) Learning Rate Scheduler strategy: Use the CosineAnnealingLR strategy.

[0107] In step S3 of this embodiment, when training the face super-resolution reconstruction network, the following key points need to be noted:

[0108] 1. Evaluate model performance

[0109] A) Performance indicators: Determine key indicators for evaluating the effectiveness of the model, such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), etc., and can also combine human subjective evaluation to measure the quality of the reconstructed image. B) Benchmark comparison: Compare the performance of the current model with the benchmark model to find out the advantages and disadvantages of the current model. C) Error diagnosis: Analyze the error cases that occur in the model's prediction process, especially those recurring patterns, and find out the root causes of these errors.

[0110] 2. Clarify the optimization direction

[0111] A) Optimization goals: According to the requirements of specific applications, clarify the main optimization goals, such as improving the detail clarity of the image, reducing artifacts in the reconstruction process, and improving the realism of the image. B) Optimization sorting: List all possible improvement points and sort them according to their importance, giving priority to the most critical issues.

[0112] 3. Data Collection and Preprocessing

[0113] A) Data enhancement: Use a variety of data enhancement techniques, such as random cropping, flipping, rotation, etc., to increase the diversity and richness of the data set and enhance the generalization ability of the model. B) Data purification: Remove outliers and duplicates in the data set to ensure the purity and high quality of the data. C) Feature optimization: Optimize the feature representation of the input data through feature selection and feature construction, such as adjusting the image size and performing data standardization.

[0114] 4. Network structure adjustment

[0115] A) Architecture adjustment: According to the evaluation results of model performance, adjust the network structure, such as adding or removing convolutional layers, introducing residual connections, etc., to improve the expressiveness of the model. B) Hyperparameter optimization: Adjust key hyperparameters in the model training process, such as learning rate, batch size, and optimizer selection.

[0116] 5. Model Retraining

[0117] A) Initialization adjustment: Based on the results of the previous round of optimization, adjust the parameter settings of the model initialization. B) Training strategy: Use strategies such as learning rate decay and early stopping to optimize the training process and improve training efficiency. C) Validation set monitoring: Use the validation set to monitor the overfitting phenomenon that may occur during the training process to ensure that the model has good generalization ability.

[0118] 6. New model verification

[0119] A) Performance testing: Comprehensively evaluate the new model on an independent test data set and compare it with the previous model version. B) Result analysis: Carefully check the output of the new model, analyze its specific improvements over the old model, and confirm whether the predetermined optimization goals have been achieved. C) User feedback: If conditions permit, external opinions can be obtained through user surveys or expert reviews to further verify the improvement effect of the model.

[0120] 7. Deployment and Continuous Monitoring

[0121] A) Model deployment: Integrate the optimized model into the actual application system to ensure that it can run stably in the production environment. B) Real-time monitoring: After the model is put into actual use, continuously monitor its performance and collect field data to ensure that the model can cope with various actual scenarios. C) Performance evaluation: Regularly evaluate the performance of the model in actual applications, including but not limited to key indicators such as the model's response time, accuracy, and stability.

[0122] 8. Continuous Improvement and Iteration

[0123] A) Iterative optimization: Based on the performance of the model in actual deployment and user feedback, continuously optimize and iterate the model to continuously improve its performance. B) Experience accumulation: Record the specific improvement measures, effect evaluation results and lessons learned of each iteration to form a knowledge base to provide reference for subsequent optimization work. C) Teamwork: Encourage communication and cooperation among team members, share optimization experience and technical achievements, and promote the advancement of overall technical level.

[0124] S4. Perform face recognition on the reconstructed super-resolution face image using a face recognition algorithm, output the face recognition result, visualize the face recognition result and store the recognized person.

[0125] Step S4 of the present invention is face recognition. In this step, based on the high-quality face image reconstructed by super-resolution, accurate face recognition is achieved by a face recognition algorithm. The face recognition algorithm used here can adopt an existing face recognition algorithm, as long as it can achieve face recognition and output a corresponding recognition result, and is not limited in the present invention.

[0126] In this embodiment, for the super-resolution reconstruction and recognition results of faces in a fixed monitoring area, this embodiment establishes detailed archival records and instantly pushes relevant information to the mobile devices of relevant managers so that they can respond to and handle potential security incidents in a timely manner.

[0127] The present invention mainly focuses on super-resolution reconstruction and face recognition in fixed-point monitoring scenarios, and provides reliable verification basis for management departments through recording and archiving. The method of the present invention is not only suitable for community security application scenarios, but can also be widely used in commercial buildings, industrial areas, public facilities and other places. In addition, the method of the present invention also has the following characteristics: multi-scenario adaptability, applicable to a variety of monitoring environments; customized functions, functional modules can be customized according to specific needs; cross-platform compatibility, ensuring stable operation under different operating systems and hardware environments; user authority management, protecting personal privacy and data security; integrating third-party services to enhance functionality and convenience of use.

[0128] When integrating the trained face super-resolution reconstruction network into the face recognition system for actual application in video surveillance scenarios, it is necessary to ensure that it can run stably in the production environment. After the face super-resolution reconstruction network is put into actual use, continue to monitor its performance and collect field data to ensure that the face super-resolution reconstruction network can cope with various actual scenarios. Regularly evaluate the performance of the face super-resolution reconstruction network in actual applications, including but not limited to key indicators such as the model's response time, accuracy, and stability. Based on the performance of the model in actual deployment and user feedback, continue to optimize and iterate the face super-resolution reconstruction network to continuously improve its performance. Record the specific improvement measures, effect evaluation results, and lessons learned for each iteration to form a knowledge base to provide a reference for subsequent optimization work.

[0129] In addition, this embodiment also provides the back-end implementation process when applying the method of the present invention, such as Figure 2 As shown, the details are as follows:

[0130] (1) Initialization phase: The interface receives requests from the front end, including necessary parameters such as the camera address, the selected algorithm type (for example, whether to enable face super-resolution reconstruction and face recognition functions), and the callback address.

[0131] (2) Video stream capture: After receiving the parameters, the interface starts a new process to continuously capture image frames in the video stream from the specified camera address. These image frames will be temporarily stored in the memory database Redis and trigger the notification mechanism of the listening program at the same time.

[0132] (3) Image frame processing and algorithm instantiation: After receiving the notification, the listener reads the image frame from Redis and creates the corresponding face super-resolution reconstruction and face recognition algorithm instances. Subsequently, the algorithm instance begins to analyze and process the acquired image frames, including face detection and positioning, super-resolution reconstruction, and face recognition.

[0133] (4) Image analysis and result storage: The algorithm instance processes the image frame according to the preset business logic, including but not limited to tasks such as person recognition and trajectory tracking. After the analysis is completed, the results will be stored back in the Redis database and the listening program will be notified again.

[0134] (5) Result feedback: After receiving the notification of result update, the listening program reads the analysis results from Redis and returns the results to the front-end business interface through a pre-defined callback mechanism, completing a complete processing flow.

[0135] (6) Error handling and logging: Add an exception handling mechanism and record log information for subsequent troubleshooting and optimization. Resource management and optimization: Use load balancing technology to distribute processing load and optimize resource allocation.

[0136] (7) Security considerations: Use encrypted communication protocols to protect data security and implement access control strategies.

[0137] In order to better demonstrate the specific implementation and technical effects of the present invention, the face super-resolution reconstruction and recognition method used in video surveillance shown in steps S1 to S4 in the above preferred implementation manner is applied to a specific example.

[0138] Example

[0139] The specific implementation process of the face super-resolution reconstruction and recognition method used in video surveillance adopted in this embodiment is as described above and will not be repeated here.

[0140] In order to demonstrate the technical effect of the method of the present invention, this embodiment is tested on the open source dataset CASIA-WebFace. The above dataset is a large-scale facial image dataset, containing 494,413 images of 10,308 people. The test results of the present invention and the typical image super-resolution algorithm SRGAN on the above dataset are shown in Table 1.

[0141] Table 1. Average performance comparison results of the present invention and the comparative method on the test data set

[0142] method PSNR Metrics SSIM metric SRGAN 29.7090 0.9478 The present invention 30.7399 0.9609

[0143] In Table 1, PSNR and SSIM are two commonly used indicators for measuring image quality. PSNR is an indicator for measuring image distortion, usually used to quantify the difference between the reconstructed image and the original image. The higher the PSNR value, the closer the reconstructed image is to the original image. SSIM is an indicator for measuring the similarity between two images. This indicator takes into account three factors: brightness, contrast, and structure. An SSIM value close to 1 indicates that the two images are highly similar.

[0144] According to the performance comparison on the test data set, it can be seen that the method of the present invention can show good performance in face super-resolution.

[0145] In summary, the present invention proposes a method for super-resolution reconstruction and recognition of faces used in video surveillance, and its core goal is to use widely deployed fixed-point fixed cameras to achieve intelligent, high-precision super-resolution reconstruction of faces and personnel identification functions. By integrating advanced deep learning algorithms, the method of the present invention can automatically detect and capture facial images in video surveillance images, and then perform super-resolution reconstruction operations and accurate face recognition processes. This method not only provides a convenient, efficient and scalable information management platform for the non-contact attendance system and key security monitoring areas, but also relies on the powerful ability of deep learning to significantly improve the intelligent management efficiency of non-contact attendance, and ensure the efficient and stable operation of the face recognition system. More importantly, the method of the present invention has a high degree of flexibility and scalability, and can easily support the addition of subsequent functions and the optimization and upgrading of the performance of the face recognition system, so as to continuously adapt to the new trend of technological development. Specifically, the method has the following advantages:

[0146] 1) Good generalization ability: By training on large-scale datasets, the face super-resolution reconstruction network can improve its generalization ability in different scenarios;

[0147] 2) Efficient computing performance: With the development of deep learning hardware, such as the popularity of GPU, the computing efficiency of face super-resolution reconstruction network has been significantly improved;

[0148] 3) Robustness: The face super-resolution reconstruction network can improve the robustness to changes in input image quality by learning a large number of noisy and blurred images;

[0149] 4) Efficient interaction: This method can bring an efficient interactive experience in the application.

[0150] It should also be noted that the face super-resolution reconstruction and recognition method used in video surveillance in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a face super-resolution reconstruction and recognition system used in video surveillance corresponding to the face super-resolution reconstruction and recognition method used in video surveillance provided in the above embodiment, such as Fig. 9 As shown, it includes:

[0151] The data acquisition module is used to acquire the original video stream acquired by the image acquisition device in real time and pre-process the original video stream;

[0152] The detection module is used to locate the face position of the preprocessed video stream in the target area using a face detection algorithm, and output a low-resolution face image with a detection frame for super-resolution reconstruction;

[0153] A super-resolution reconstruction module is used to input a low-resolution face image into a trained face super-resolution reconstruction network, perform bilinear interpolation on the low-resolution face image to obtain an interpolated face image, process the low-resolution face image through a first point-by-point convolutional layer to obtain an initial fused feature map, process the initial fused feature map through a backbone network and a super-resolution reconstruction module in sequence to obtain a final fused feature map, perform feature stacking on the interpolated face image and the final fused feature map, and output a reconstructed super-resolution face image;

[0154] During the training process of the face super-resolution reconstruction network, an original data set consisting of face images is first obtained, and the original data set is preprocessed to obtain a preprocessed data set, and the face super-resolution reconstruction network is trained on the preprocessed data set; the backbone network is formed by sequentially cascading a plurality of key modules and an offset hole convolution-self-attention mechanism module, each of which is formed by sequentially cascading an offset hole convolution-transposed residual block and a dual-aggregation self-attention mechanism module, and the dual-aggregation self-attention mechanism module is formed by sequentially cascading a spatial aggregation module and a channel aggregation module, the spatial aggregation module includes a spatial self-attention mechanism, an activation function with adjustable thresholds, and a gating mechanism feedback network, and the channel aggregation module includes a channel self-attention mechanism, an activation function with adjustable thresholds, and a gating mechanism feedback network; in the super-resolution reconstruction module, the feature map output by the backbone network is sequentially processed by an offset hole convolution-self-attention mechanism module, a second point-by-point convolution layer, a pixel shuffling operation, and a third point-by-point convolution layer to obtain a final fused feature map;

[0155] The recognition module is used to perform face recognition on the reconstructed super-resolution face image using a face recognition algorithm, output the face recognition result, visualize the face recognition result and store the recognized person.

[0156] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0157] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A face super-resolution reconstruction and recognition method used in video surveillance, characterized in that: The following steps are involved: S1, obtaining the original video stream collected by the image acquisition device in real time, and preprocessing the original video stream; S2. In the target area, the face detection algorithm locates the face position of the preprocessed video stream and outputs a low-resolution face image with a detection frame for super-resolution reconstruction; S3, input the low-resolution face image into the trained face super-resolution reconstruction network, perform bilinear interpolation on the low-resolution face image to obtain an interpolated face image, process the low-resolution face image through the first point-by-point convolution layer to obtain an initial fused feature map, process the initial fused feature map through the backbone network and the super-resolution reconstruction module in turn to obtain a final fused feature map, perform feature stacking on the interpolated face image and the final fused feature map, and output a reconstructed super-resolution face image; During the training process of the face super-resolution reconstruction network, an original data set consisting of face images is first obtained, and the original data set is preprocessed to obtain a preprocessed data set, and the face super-resolution reconstruction network is trained on the preprocessed data set; the backbone network is formed by sequentially cascading a plurality of key modules and an offset hole convolution-self-attention mechanism module, each of which is formed by sequentially cascading an offset hole convolution-transposed residual block and a dual-aggregation self-attention mechanism module, and the dual-aggregation self-attention mechanism module is formed by sequentially cascading a spatial aggregation module and a channel aggregation module, the spatial aggregation module includes a spatial self-attention mechanism, an activation function with adjustable thresholds, and a gating mechanism feedback network, and the channel aggregation module includes a channel self-attention mechanism, an activation function with adjustable thresholds, and a gating mechanism feedback network; in the super-resolution reconstruction module, the feature map output by the backbone network is sequentially processed by an offset hole convolution-self-attention mechanism module, a second point-by-point convolution layer, a pixel shuffling operation, and a third point-by-point convolution layer to obtain a final fused feature map; S4. Perform face recognition on the reconstructed super-resolution face image using a face recognition algorithm, output the face recognition result, visualize the face recognition result and store the recognized person.

2. A method for super-resolution reconstruction and recognition of faces in video surveillance as claimed in claim 1, characterized in that: In step S3, the specific processing flow in the offset dilated convolution-transposed residual block is as follows: the feature map input to the offset dilated convolution-transposed residual block passes through the first offset dilated point-by-point convolution layer, separates the feature channels of the input feature map, and obtains a first feature map; the first feature map passes through the first depth convolution layer, the second offset dilated point-by-point convolution layer, the second depth convolution layer, the third offset dilated point-by-point convolution layer, and the third depth convolution layer in sequence to obtain a second feature map; the first feature map and the second feature map are added to obtain a third feature map; the input feature map is spliced ​​with the third feature map and the first feature map to obtain a first spliced ​​feature map; the first spliced ​​feature map is processed by the fourth offset dilated point-by-point convolution layer to obtain the output of the offset dilated convolution-transposed residual block.

3. A method for super-resolution reconstruction and recognition of faces in video surveillance as claimed in claim 2, characterized in that: The convolution kernel sizes of the first, second, and third depthwise convolutional layers are all 3×3.

4. The method for super-resolution reconstruction and recognition of faces in video surveillance according to claim 1, characterized in that: In step S3, the specific processing flow in the offset hole convolution-self-attention mechanism module is as follows: the feature map input to the offset hole convolution-self-attention mechanism module passes through the fifth offset hole point-by-point convolution layer, separates the feature channels of the input feature map, and obtains a fourth feature map. The fourth feature map is input into the multi-head self-attention mechanism module to obtain a fifth feature map. The fourth feature map and the fifth feature map are added to obtain a sixth feature map. The sixth feature map is input into the feedforward neural network to obtain a seventh feature map. The sixth feature map and the seventh feature map are added and then spliced ​​with the fourth feature map to obtain a second spliced ​​feature map. After the second spliced ​​feature map is processed by the sixth offset hole point-by-point convolution layer, the output of the offset hole convolution-self-attention mechanism module is obtained.

5. The method for super-resolution reconstruction and recognition of faces in video surveillance according to claim 1, characterized in that: In step S3, the specific processing flow in the spatial aggregation module is as follows: the feature map input to the spatial aggregation module is sequentially normalized by layers to obtain a first-layer normalized feature map, and the first-layer normalized feature map is then passed through a feature projection layer to obtain a first spatial feature reshaping matrix; After processing the first spatial feature reshaping matrix, a spatial query matrix, a spatial key matrix and a spatial value matrix are obtained. The spatial query matrix and the spatial key matrix are matrix multiplied and processed by the Softmax activation function using the spatial self-attention mechanism to output the first spatial feature map. The spatial value matrix and the first spatial feature map are matrix multiplied to obtain the second spatial feature reshaping matrix. The second spatial feature reshaping matrix is ​​sequentially processed by the first point-by-point convolution layer, the threshold adjustable activation function, the second point-by-point convolution layer, and the Sigmoid activation function to obtain the first spatial weight map. The first spatial feature reshaping matrix is ​​processed by the first depth-separable convolution layer to output the second spatial feature map. The second spatial feature map is sequentially processed by the third point-by-point convolution layer, the threshold adjustable activation function, the fourth point-by-point convolution layer, and the fifth point-by-point convolution layer. After point-by-point convolution layer and Sigmoid activation function processing, a second spatial weight map is obtained; the second spatial weight map is element-wise multiplied with the corresponding elements of the second spatial feature reshaping matrix to obtain the first spatial weighted feature, the first spatial weight map is element-wise multiplied with the corresponding elements of the second spatial feature map to obtain the second spatial weighted feature, the first spatial weighted feature and the second spatial weighted feature are added and then passed through a feature projection layer to obtain a third spatial feature reshaping matrix, the first layer normalized feature map is added to the third spatial feature reshaping matrix to obtain the third spatial feature map, the third spatial feature map is processed by layer normalization and gating mechanism feedback network in turn to obtain the fourth spatial feature map, and the third spatial feature map and the fourth spatial feature map are added as the output of the spatial aggregation module.

6. The method for super-resolution reconstruction and recognition of faces in video surveillance as claimed in claim 1, characterized in that: In step S3, the specific processing flow in the channel aggregation module is as follows: the feature map input to the channel aggregation module is sequentially normalized by layers to obtain a second-layer normalized feature map, and the second-layer normalized feature map is then passed through a feature projection layer to obtain a first channel feature reshaping matrix; After processing the first channel feature reshaping matrix, the channel query matrix, channel key matrix and channel value matrix are obtained. The channel query matrix and the channel key matrix are matrix multiplied and processed by the Softmax activation function after applying the channel self-attention mechanism to output the first channel feature map. The channel value matrix and the first channel feature map are matrix multiplied to obtain the second channel feature reshaping matrix. The second channel feature reshaping matrix is ​​sequentially processed by the fifth point-by-point convolution layer, the threshold adjustable activation function, the sixth point-by-point convolution layer, and the Sigmoid activation function to obtain the first channel weight map; the first channel feature reshaping matrix is ​​processed by the second depth-separable convolution layer to output the second channel feature map. The second channel feature map is sequentially processed by the seventh point-by-point convolution layer, the threshold adjustable activation function, the eighth point-by-point convolution layer, and the sigmoid activation function. After point-by-point convolution layer and Sigmoid activation function processing, the second channel weight map is obtained; the second channel weight map is element-wise multiplied with the corresponding elements of the second channel feature reshaping matrix to obtain the first channel weighted features; the first channel weight map is element-wise multiplied with the corresponding elements of the second channel feature map to obtain the second channel weighted features; the first channel weighted features and the second channel weighted features are added and then passed through a feature projection layer to obtain the third channel feature reshaping matrix; the second layer normalized feature map is added to the third channel feature reshaping matrix to obtain the third channel feature map; the third channel feature map is processed by layer normalization and gating mechanism feedback network in turn to obtain the fourth channel feature map; the third channel feature map and the fourth channel feature map are added as the output of the channel aggregation module.

7. A method for super-resolution reconstruction and recognition of faces in video surveillance as claimed in claim 5 or 6, characterized in that: The specific processing flow in the gating mechanism feedback network is as follows: the feature map input to the gating mechanism feedback network is sequentially subjected to a linear projection layer, an activation function with adjustable threshold, and a feature separation operation along the channel dimension, and the first part obtained by feature separation is processed by a third depth-wise separable convolution layer to obtain an eighth feature map, and the corresponding elements of the eighth feature map and the second part obtained by feature separation are element-wise multiplied to obtain a ninth feature map, and the ninth feature map is passed through a linear projection layer to obtain the output of the gating mechanism feedback network.

8. The method for super-resolution reconstruction and recognition of faces in video surveillance as claimed in claim 7, characterized in that: Activation function with adjustable threshold The functional form is: ； In the formula, Indicates an adjustable threshold; Represents the characteristic value of the input activation function; Indicates the maximum value operation.

9. The method for super-resolution reconstruction and recognition of faces in video surveillance as claimed in claim 1, characterized in that: The number of key modules is a positive integer and a multiple of 3.

10. A face super-resolution reconstruction and recognition system used in video surveillance, characterized in that: include: The data acquisition module is used to acquire the original video stream acquired by the image acquisition device in real time and pre-process the original video stream; The detection module is used to locate the face position of the preprocessed video stream in the target area using a face detection algorithm, and output a low-resolution face image with a detection frame for super-resolution reconstruction; A super-resolution reconstruction module is used to input a low-resolution face image into a trained face super-resolution reconstruction network, perform bilinear interpolation on the low-resolution face image to obtain an interpolated face image, process the low-resolution face image through a first point-by-point convolutional layer to obtain an initial fused feature map, process the initial fused feature map through a backbone network and a super-resolution reconstruction module in sequence to obtain a final fused feature map, perform feature stacking on the interpolated face image and the final fused feature map, and output a reconstructed super-resolution face image; During the training process of the face super-resolution reconstruction network, an original data set consisting of face images is first obtained, and the original data set is preprocessed to obtain a preprocessed data set, and the face super-resolution reconstruction network is trained on the preprocessed data set; the backbone network is formed by sequentially cascading a plurality of key modules and an offset hole convolution-self-attention mechanism module, each of which is formed by sequentially cascading an offset hole convolution-transposed residual block and a dual-aggregation self-attention mechanism module, and the dual-aggregation self-attention mechanism module is formed by sequentially cascading a spatial aggregation module and a channel aggregation module, the spatial aggregation module includes a spatial self-attention mechanism, an activation function with adjustable thresholds, and a gating mechanism feedback network, and the channel aggregation module includes a channel self-attention mechanism, an activation function with adjustable thresholds, and a gating mechanism feedback network; in the super-resolution reconstruction module, the feature map output by the backbone network is sequentially processed by an offset hole convolution-self-attention mechanism module, a second point-by-point convolution layer, a pixel shuffling operation, and a third point-by-point convolution layer to obtain a final fused feature map; The recognition module is used to perform face recognition on the reconstructed super-resolution face image using a face recognition algorithm, output the face recognition result, visualize the face recognition result and store the recognized person.

Images