Face gait feature layer fusion intelligent building monitoring system based on attention mechanism

Abstract

The present application relates to a kind of based on attention mechanism face gait feature layer fusion wisdom building monitoring system, the present application first collects the face and gait information of personnel, constructs dual-mode face and gait dataset;Then, the face and gait sample in dataset are used to train the face gait feature layer fusion algorithm based on attention mechanism;The algorithm uses spatial attention mechanism to set larger weight to the key area of face and gait high-order feature respectively, and set smaller weight to non-key area, effectively improve the difference between the high-order features of different personnel, the greater the difference between the features of different personnel, the easier to distinguish, and then effectively improve the recognition rate.The algorithm also uses a decision mechanism, channel attention mechanism and similarity objective function to solve the problem that face occlusion cannot detect personnel information in video monitoring, and effectively improves the personnel recognition rate of monitoring system.

Description

Technical Field

[0001] This invention relates to monitoring systems, and more particularly to a smart building monitoring system based on the fusion of facial gait features using an attention mechanism. Background Technology

[0002] With the continuous progress of society and the development of the intelligent era, office buildings, as places with high floors, complex personnel access, and large traffic flow, face increasing challenges in security management. Consequently, building intelligence technologies are receiving increasing attention, and smart buildings have become a major development trend. By utilizing next-generation information technologies such as machine learning, facial recognition, the Internet of Things, big data, and cloud computing, the way people interact with building systems can be transformed, significantly improving the effectiveness, flexibility, responsiveness, environmental friendliness, and resource and energy efficiency of building systems.

[0003] For smart building monitoring systems, the main function of facial recognition systems is to provide personnel identification technology support for building information technology applications and to provide protocol interfaces for other information systems such as public application systems, management application systems, business application systems, and intelligent information integration systems. Furthermore, various information application systems transmit acquired images to the facial recognition system, which processes the images and outputs the recognition results back to the system via interfaces. The system then makes further decisions based on the recognition results.

[0004] In smart building systems, commonly used monitoring systems rely on facial images captured by cameras to identify individuals. This method yields good results when the cameras capture clear facial images. However, when individuals deliberately masquerade by obscuring their faces, cameras struggle to capture clear images, rendering facial recognition-based monitoring systems ineffective. Secondly, gait recognition, a method that analyzes pedestrian walking postures to identify individuals, offers advantages such as being non-contact, long-distance, and difficult to spoof. Therefore, its improvement and innovation are imperative. Summary of the Invention

[0005] To address the above issues and overcome the shortcomings of existing technologies, the purpose of this invention is to provide a smart building monitoring system based on the fusion of facial gait features using an attention mechanism. This system can better identify personnel information within smart buildings. The attention mechanism is a feature processing method that focuses on key or important information in an image by weighting features, offering advantages such as flexibility and fewer parameters.

[0006] The technical solution solved by this invention is:

[0007] A smart building monitoring system based on attention mechanism and facial gait feature fusion, the monitoring system includes a field layer, a communication network layer and a monitoring processing layer;

[0008] The field layer includes image and audio acquisition devices installed within the building, and the image and audio acquisition devices include cameras and audio sensors;

[0009] The communication network layer includes multiple fiber optic switches, which form a fiber optic ring network to aggregate video data collected by the field layer and send it to the monitoring and processing layer. At the same time, the feedback signals from the monitoring and processing layer are also sent to the field layer.

[0010] The monitoring and processing layer includes a display and a main control module containing a face and gait feature fusion algorithm based on an attention mechanism, which is used to process face and gait information and provide feedback results.

[0011] Specifically, the cameras at the field layer capture video in real time, which is then transmitted to the main control module at the monitoring and processing layer via a fiber optic ring network.

[0012] The main control module first divides the video into two parts. One part of the video is then divided into frames and images, and the facial region is automatically extracted.

[0013] If a facial area exists in a video clip, the facial area will be automatically extracted.

[0014] If a face is not present in a video clip, a black background image is used instead.

[0015] The face image size is then modified to 124×124×3 and input into a residual network of a face gait feature layer fusion algorithm based on an attention mechanism to extract high-order face features.

[0016] For another video clip, if gait information is present in the video clip, it is processed normally; if gait information is absent, it is replaced with a black background image, then modified to 640×640×3, and input into a 3D-CNN based on an attention-based face gait feature layer fusion algorithm to extract high-order gait features. Then, the attention-based face gait feature layer fusion algorithm is used to classify people.

[0017] There are four possibilities at this point:

[0018] 1) When the video does not contain facial information or gait information, the monitoring system does not need to identify personnel information;

[0019] 2) When the video does not contain facial information but contains gait information, the algorithm recognizes people information through 3D-CNN, spatial attention and fully connected (FC); in addition, the algorithm uses the spatial attention mechanism to set large weights for key areas of gait high-order features and small weights for non-key areas, which effectively improves the difference between high-order features of different people, thereby effectively improving the recognition rate of people.

[0020] 3) When the video contains facial information but not gait information, the algorithm uses residual networks, spatial attention and fully connected (FC) to recognize people information. It also uses the spatial attention mechanism to set large weights on key areas of high-order facial features and small weights on non-key areas, which effectively improves the difference between high-order facial features of different people and thus effectively improves the recognition rate.

[0021] 4) When the video contains both facial and gait information, the algorithm increases the correlation between the gait and facial modalities of each person by utilizing channel attention and similarity objective functions, thereby improving the recognition rate. At the same time, spatial attention weights increase the differences between higher-order features of different people, which also effectively improves the recognition rate of people.

[0022] Finally, the face gait feature fusion algorithm based on the attention mechanism sends the recognition results to the main control module. The main control module issues decision instructions based on the results, which are then sent to the audio sensors at the field layer via the fiber optic switch of the fiber optic ring network to inform the person being identified.

[0023] The residual network of the face gait feature layer fusion algorithm based on the attention mechanism contains K residual blocks, each of which contains 2 convolutional layers (CL1 and CL2), 3 ReLU activation function layers, 2 normalization layers (BN), one addition operation, and one pooling layer.

[0024] Convolutional layers are used to extract facial features. For a convolutional layer l∈L, the convolution operation formula for two-dimensional face image data I(i,j) is as follows:

[0025]

[0026] Among them, S l,c (i,j) represents the two-dimensional face image data; I(i,j) represents the result of the c-th channel after convolution; K(m,n) represents the two-dimensional convolution kernel; i,j represents the index of the width and height of the two-dimensional image; m,n represents the index of the width and height of the two-dimensional convolution kernel;

[0027] The dimensional relationship between the output and input after a convolution operation is defined as follows:

[0028]

[0029] Among them, w l and h l S represents l,c The width and height dimensions of (i,j); w l-1 and h l-1 Let I(i,j) represent the width and height dimensions. and This indicates the size of the zero padding in the convolution operation; and This represents the stride size in convolution; and Describes the dimension of K(m,n); c l Indicates the number of channels;

[0030] The activation function layer achieves non-linearity of the features, where the activation function chosen is ReLU, which is defined as follows:

[0031] ReLU(x) = max(0,x) (3)

[0032] The normalization layer normalizes features by subtracting the mean and dividing by the variance, specifically defined as:

[0033]

[0034] Among them, Z bn,l,c [i,j] represents the output of the c-th channel of the normalization layer; μ is the coefficient (an infinitesimally small number) to prevent division by zero; l,c and Var l,c Let represent the mean and variance of the c-th channel, which are defined as follows:

[0035]

[0036]

[0037] Where F represents the batch size; This represents the convolution output of the f-th sample;

[0038] The addition operation alleviates the gradient vanishing problem by adding the input features and the convolution output features. The pooling layer reduces the feature dimension by downsampling. This step uses max pooling.

[0039] The face image is first used to extract the face region, which is then modified to 124×124×3. This modified region is then input into a residual network to extract higher-order face features. The output dimension of these higher-order face features is calculated using formula (2). The final output of the higher-order face features is as follows:

[0040] Preferably, the 3D-CNN of the face gait feature layer fusion algorithm based on the attention mechanism contains seven 3D convolutional layers and five pooling layers. When the 3D-CNN extracts high-order gait features, the gait video is first divided into frames, and then each frame is modified to 640×640×3 before being input into the 3D-CNN to extract high-order gait features. The final output high-order gait features are:

[0041] Preferred, high-level facial features and higher-order gait features Each is processed by a spatial attention mechanism to improve the recognition rate;

[0042] The spatial attention mechanism consists of a 1×1 convolution, a ReLU activation function, and a multiplication operation. For high-order facial features... Spatial attention weights are represented as:

[0043]

[0044] Among them, Conv 1×1 (·) represents a 1×1 convolution, meaning the kernel is 1×1, the stride is 1×1, and the number of channels is 1; ReLU(·) represents the activation function operation; Ξ,ξ represent the network parameters;

[0045] As shown in formula (2), after a 1×1 convolution, the dimension of the feature becomes 64×1, and after ReLU, the spatial attention weights of the high-order features of the face are obtained.

[0046] Spatial attention weights With higher-order facial features Multiplication yields spatial attention-weighted high-order facial features, represented as follows:

[0047]

[0048] Similarly, for higher-order gait features Spatial attention weights are represented as:

[0049]

[0050] Spatial attention weights With higher-order facial features Multiplication yields spatial attention-weighted higher-order gait features, represented as follows:

[0051]

[0052] Ultimately, spatial attention-weighted high-order facial features are obtained. and higher-order gait features

[0053] When the video does not contain facial information but contains gait information, and (Scenario 2);

[0054] This situation arises from the common problem of face occlusion in video surveillance based on facial features (facial information cannot be captured, but gait information can be obtained); in this case, spatial attention-weighted high-order facial features are used. However, spatial attention-weighted higher-order gait features Spatial attention-weighted gait higher-order features A 4-layer fully connected (FC) classification network is used; the output layer of the fully connected network has C neurons, and the activation function is the softmax function, achieving classification for C. The specific optimization function is defined as:

[0055]

[0056] Where θ represents the parameters of the fully connected (FC) network; Υ(·) represents the features output by the fully connected (FC) network; softmax(·) represents the prediction result of the FC network; K represents the number of training samples for gait information; y i The label is represented by L(·), which is the objective function, and its specific expression is as follows:

[0057]

[0058] Among them, y i,j and Let represent the label of the j-th class of the i-th gait data and the output of the softmax layer, respectively; C represents the total number of label classes.

[0059] During training, most images (background areas) of gait videos from different individuals are similar, with differences only in the gait-containing areas (key areas). Therefore, by using objective functions (11) and (12) to guide the spatial attention mechanism, large weights are assigned to key areas of gait high-order features, while smaller weights are assigned to non-key areas. This effectively improves the differences between high-order gait features of different individuals, thereby effectively improving the recognition rate.

[0060] When the video contains facial information but not gait information and (Case 3); Case 3 is similar to Case 2, using spatial attention-weighted high-order facial features. This algorithm classifies people using a 4-layer fully connected classification network. During training, most facial images (background regions) of different individuals are similar, with differences only appearing in the regions containing the face (key regions). Similarly, the algorithm utilizes an objective function to guide a spatial attention mechanism, assigning larger weights to key regions of high-order facial features and smaller weights to non-key regions. This effectively increases the difference in high-order features between different individuals, thereby improving the recognition rate.

[0061] Preferably, when the video contains both facial information and gait information, and (Scenario 4);

[0062] Spatial attention-weighted high-order facial features and higher-order gait features First, the sample is globally averaged and then passed through a channel attention network composed of an MLP (Multilayer Perceptron). The first layer of the MLP has 64 neurons, the second layer has 32, and the third layer has 64. Specifically:

[0063] High-level facial features and higher-order gait features The face features are obtained by global average pooling. and higher-order gait features These are input into the MLP to obtain channel attention weights. and It is defined as follows:

[0064]

[0065]

[0066] Here, FC(·) represents a fully connected operation.

[0067] Channel attention weight With higher-order facial features Multiplying these yields the channel attention-weighted high-order facial features, which are represented as follows:

[0068]

[0069] Channel attention weight higher-order features of gait Multiplying these yields the channel attention-weighted higher-order gait features, which are represented as follows:

[0070]

[0071] Finally, the channel- and spatial attention-weighted high-order face and gait features are optimized by a similarity objective function, which is defined as follows:

[0072]

[0073] The similarity objective function (17) optimizes the channel attention weights by finding the similarity between faces and gaits. The more similar the face and gait information of each person is, the greater the channel attention weight; the less similar the face and gait information of each person is, the smaller the channel attention weight. This increases the correlation between the gait and face modalities of each person. In the multimodal fusion process, the more correlated the modalities are, the better the fusion performance. Therefore, channel attention improves the recognition rate by increasing the correlation between faces and gaits. At the same time, spatial attention weights improve the differences between the higher-order features of different people, which also effectively improves the recognition rate of people.

[0074] Ultimately, the personnel classification in Case 4 is achieved using SVM (Support Vector Machine). SVM was originally designed for linearly separable sample data; however, in fields such as pattern recognition, image recognition, computer vision, natural language processing, and text processing, non-linear sample data is frequently encountered. To address this problem, SVM introduces a kernel function. This invention utilizes an SVM with a Gaussian kernel function because the Gaussian kernel function can solve the problem of non-linear sample data and contains only one hyperparameter; that is, the SVM has a Gaussian kernel function, the calculation expression of which is as follows:

[0075]

[0076] Among them, (x i ,x j )express and Features of [the text].

[0077] This system is a C-class classification problem, while SVM is a binary classification problem. Therefore, this paper adopts a "one-to-many method" to achieve multi-class classification using SVM. Specifically, the data of one person in the bimodal face and gait dataset is successively designated as the front face (labeled 1), and the data of other people in the bimodal face and gait dataset are designated as the back face (labeled -1). For a C-class classification problem, the one-to-many method only requires constructing C classifiers.

[0078] This invention's monitoring system first collects facial and gait information of individuals to construct a bimodal face and gait dataset. Then, it uses face and gait samples from the dataset to train a face-gait feature layer fusion algorithm based on an attention mechanism. Since most of the face images and gait videos (background areas) of different individuals are similar, differences only exist in the regions containing the face and gait (key areas). The characteristic of the spatial attention mechanism is to identify these key areas. This algorithm uses the spatial attention mechanism to assign greater weights to key areas of high-order face and gait features and less weights to non-key areas, effectively improving the differences between high-order features of different individuals. The greater the differences in features between different individuals, the easier it is to distinguish them, thus effectively improving the recognition rate. This algorithm also addresses the problem of undetectable personnel information due to facial occlusion in video surveillance by utilizing a decision mechanism, channel attention mechanism, and similarity objective function, effectively improving the personnel recognition rate of the monitoring system. Finally, the camera captures real-time facial and gait videos of personnel and transmits them to the master control mode via a fiber optic ring network. The master control mode then uses a facial and gait feature layer based on an attention mechanism to identify personnel information, makes a decision, and transmits the information to an audio sensor via the fiber optic ring network to inform the identified personnel. This not only solves the problem of facial occlusion in video surveillance but also effectively improves the recognition rate of the monitoring system. It is easy to use, effective, and represents an innovation in smart building monitoring systems, offering significant social and economic benefits. Attached Figure Description

[0079] Figure 1 This is a block diagram of the face gait feature layer fusion algorithm based on the attention mechanism of the present invention.

[0080] Figure 2 This is a schematic diagram of the high-order facial feature extraction network of the present invention: the residual network.

[0081] Figure 3 This is a schematic diagram of the gait high-order feature extraction network of the present invention: 3D-CNN.

[0082] Figure 4 A schematic diagram of the intelligent building monitoring system of the present invention. Detailed Implementation

[0083] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0084] like Figure 1-4 As shown, this invention discloses a smart building monitoring system based on attention mechanism and gait feature layer fusion, which includes a field layer, a communication network layer and a monitoring and processing layer;

[0085] The field layer includes image and audio acquisition devices installed within the building, and the image and audio acquisition devices include cameras and audio sensors;

[0086] The communication network layer includes multiple fiber optic switches, which form a fiber optic ring network to aggregate video data collected by the field layer and send it to the monitoring and processing layer. At the same time, the feedback signals from the monitoring and processing layer are also sent to the field layer.

[0087] The monitoring and processing layer includes a display and a main control module containing a face and gait feature fusion algorithm based on an attention mechanism, which is used to process face and gait information and provide feedback results.

[0088] Specifically, cameras at the field level capture video in real time, which is then transmitted via a fiber optic ring network to the main control module at the monitoring and processing layer; facial and gait information of personnel is collected to construct a dual-modal face and gait dataset.

[0089] For the face dataset, firstly, a video containing faces of people in the smart building is collected, and each video segment is divided into frames and images. Secondly, OpenCV is used to automatically locate the facial regions in the face photos and extract the facial regions, selecting 100 photos containing faces for each person. Then, the facial regions of all people are numbered from 1 to C. Finally, a face dataset of 100C people's photos is formed, and each person has a unique number. For example, for a certain person, the number is i, and the corresponding face photos are numbered from i_1 to i_100.

[0090] For the gait dataset, firstly, videos of each person walking normally were captured using cameras from 11 perspectives (0°–180°), with each perspective numbered 1–11. Secondly, the videos from each perspective were manually edited into 10-second continuous videos, and then the gait videos of each person were numbered from 1 to C. Finally, 11C gait databases of 10-second segments were obtained. For example, for a given person, numbered i, the corresponding gait videos are numbered i_1–i_11.

[0091] The person identification numbers are the same in both the face and gait datasets. For example, identification number 'i' refers to the same person in both datasets. Finally, the face and gait datasets are merged to form a bimodal face and gait dataset containing 100C face images and 11C 10s gait videos.

[0092] High-order facial features are extracted using a residual network. The residual network of the face gait feature fusion algorithm based on the attention mechanism contains K residual blocks, each of which contains two convolutional layers (CL1 and CL2), three ReLU activation function layers, two normalization layers (BN), one addition operation, and one pooling layer; specifically as follows... Figure 2 As shown;

[0093] Convolutional layers are used to extract facial features. For a convolutional layer l∈L, the convolution operation formula for two-dimensional face image data I(i,j) is as follows:

[0094]

[0095] Among them, S l,c (i,j) represents the two-dimensional face image data; I(i,j) represents the result of the c-th channel after convolution; K(m,n) represents the two-dimensional convolution kernel; i,j represents the index of the width and height of the two-dimensional image; m,n represents the index of the width and height of the two-dimensional convolution kernel;

[0096] The dimensional relationship between the output and input after a convolution operation is defined as follows:

[0097]

[0098] Among them, w l and h l S represents l,c The width and height dimensions of (i,j); w l-1 and h l-1 Let I(i,j) represent the width and height dimensions. and This indicates the size of the zero padding in the convolution operation; and This represents the stride size in convolution; and Describes the dimension of K(m,n); c l Indicates the number of channels;

[0099] The activation function layer achieves non-linearity of the features, where the activation function chosen is ReLU, which is defined as follows:

[0100] ReLU(x) = max(0,x) (21)

[0101] The normalization layer normalizes features by subtracting the mean and dividing by the variance, specifically defined as:

[0102]

[0103] Among them, Z bn,l,c [i,j] represents the output of the c-th channel of the normalization layer; μ is the coefficient (an infinitesimally small number) to prevent division by zero; l,c and Var l,c Let represent the mean and variance of the c-th channel, which are defined as follows:

[0104]

[0105]

[0106] Where F represents the batch size; This represents the convolution output of the f-th sample;

[0107] The addition operation alleviates the gradient vanishing problem by adding the input features and the convolution output features. The pooling layer reduces the feature dimension by downsampling. This step uses max pooling.

[0108] The face image is first used to extract the face region, which is then modified to 124×124×3. This region is then input into a residual network to extract higher-order face features. The specific parameters of the residual network are shown in Table 1. The output dimension of the higher-order face features is calculated using formula (2). The final output higher-order face features are:

[0109] Table 1. Parameters of the 2D-CNN network for high-order face feature extraction.

[0110]

[0111] Gait high-order features were extracted using 3D-CNN. The 3D-CNN based on the attention mechanism for face gait feature fusion contains seven 3D convolutional layers and five pooling layers. Specific parameters are shown in Table 2. Since two-dimensional convolutional neural networks (2D-CNN) can only extract spatial features of images and cannot extract temporal features, 3D-CNN was proposed and widely applied in video classification, action recognition, and other fields to simultaneously extract both spatial and temporal features. Furthermore, gait videos contain both spatial and temporal features; therefore, 3D-CNN is used to extract high-order gait features. While the convolutional kernels of 3D-CNN are three-dimensional, the pooling layers, activation layers, and batch normalization (BN) layers are similar to those of 2D-CNN. Specifically, when extracting high-order gait features using 3D-CNN, the gait video is first divided into frames, and then each frame is modified to 640×640×3 before being input into... Figure 3 The high-order gait features extracted from the 3D-CNN shown are as follows:

[0112] Table 2 Parameters of the Gait High-Order Feature Extraction Network

[0113]

[0114] Perform attention-based gait feature fusion:

[0115] High-level facial features and higher-order gait features The recognition rate is improved by using spatial attention mechanisms. The reason is as follows: most facial images and gait videos of different individuals (background areas) are similar; the differences lie only in the regions containing the face and gait (key areas). The spatial attention mechanism is designed to identify these key areas. During training, the objective function specifically guides the spatial attention mechanism to assign higher weights to key areas and lower weights to non-key areas. This effectively increases the difference between higher-order features of different individuals. The greater the difference between individual features, the easier it is to distinguish them; therefore, the spatial attention mechanism can effectively improve the recognition rate.

[0116] The spatial attention mechanism is as follows: Figure 1 As shown, it consists of a 1×1 convolution, a ReLU activation function, and a multiplication operation. This is used for high-order facial features. Spatial attention weights are represented as:

[0117]

[0118] Among them, Conv 1×1 (·) represents a 1×1 convolution, meaning the kernel is 1×1, the stride is 1×1, and the number of channels is 1; ReLU(·) represents the activation function operation; Ξ,ξ represent the network parameters;

[0119] As shown in formula (2), after a 1×1 convolution, the dimension of the feature becomes 64×1, and after ReLU, the spatial attention weights of the high-order features of the face are obtained.

[0120] Spatial attention weights With higher-order facial features Multiplication yields spatial attention-weighted high-order facial features, represented as follows:

[0121]

[0122] Similarly, for higher-order gait features Spatial attention weights are represented as:

[0123]

[0124] Spatial attention weights With higher-order facial features Multiplication yields spatial attention-weighted higher-order gait features, represented as follows:

[0125]

[0126] Ultimately, spatial attention-weighted high-order facial features are obtained. and higher-order gait features

[0127] In smart building real-time monitoring systems, it is sometimes difficult to capture facial information, which affects the system's performance. To address this issue, this paper designs a decision mechanism. The details are as follows:

[0128] The video captured by the surveillance system falls into the following four categories:

[0129] 1) The video does not contain facial information or gait information;

[0130] 2) The video does not contain facial information, but it does contain gait information;

[0131] 3) The video contains facial information but not gait information;

[0132] 4) The video contains facial information as well as gait information;

[0133] Of these, scenario 1) represents the normal situation, where the monitoring system does not need to identify personnel information. The monitoring system should focus on the latter three scenarios.

[0134] Specifically, when the monitoring system does not detect a face, the face modality information is replaced with a black background, meaning the RGB values ​​of the face modality are all 0. The higher-order features obtained after inputting into the residual network and spatial attention are also all 0. When the monitoring system detects a face, the face modality features are arbitrary, and the higher-order features obtained after inputting into the residual network and spatial attention are also arbitrary. Furthermore, when the monitoring system does not detect gait information, gait information is also replaced with a black background, meaning the RGB values ​​of the gait information are all 0. The higher-order features obtained after inputting into the 3D-CNN and spatial attention are also all 0. When the monitoring system detects gait, the gait modality features are arbitrary, and the higher-order features obtained after inputting into the 3D-CNN and spatial attention are also arbitrary. Therefore, the decision condition for this mechanism is set as: spatial attention-weighted higher-order face features. Or spatial attention-weighted higher-order gait features

[0135] Finally, the face gait feature fusion algorithm based on the attention mechanism sends the recognition results to the main control module. The main control module issues decision instructions based on the results, which are then sent to the audio sensors at the field layer via the fiber optic switch of the fiber optic ring network to inform the person being identified.

[0136] When the video does not contain facial information but contains gait information, and (Scenario 2);

[0137] This situation arises from the common problem of face occlusion in video surveillance based on facial features (facial information cannot be captured, but gait information can be obtained); in this case, spatial attention-weighted high-order facial features are used. However, spatial attention-weighted higher-order gait features Spatial attention-weighted gait higher-order features A 4-layer fully connected (FC) classification network is used; the output layer of the fully connected network has C neurons, and the activation function is the softmax function, achieving classification for C. The specific optimization function is defined as:

[0138]

[0139] Where θ represents the parameters of the fully connected (FC) network; Υ(·) represents the features output by the fully connected (FC) network; softmax(·) represents the prediction result of the FC network; K represents the number of training samples for gait information; y i The label is represented by L(·), which is the objective function, and its specific expression is as follows:

[0140]

[0141] Among them, y i,j and Let represent the label of the j-th class of the i-th gait data and the output of the softmax layer, respectively; C represents the total number of label classes.

[0142] During training, objective functions (11) and (12) guide the spatial attention mechanism to set large weights for key regions of gait high-order features and small weights for non-key regions. This effectively improves the differences between high-order features of different people, thereby effectively improving the recognition rate.

[0143] When the video contains facial information but not gait information and (Case 3); Case 3 is similar to Case 2, using spatial attention-weighted high-order facial features. Personnel classification is achieved through a 4-layer fully connected (FC) classification network. During training, the objective function guides the spatial attention mechanism to assign large weights to key regions of high-order facial features and smaller weights to non-key regions. This effectively improves the differences between high-order features of different personnel, thereby effectively improving the recognition rate.

[0144] When the video contains both facial information and gait information, and (Scenario 4);

[0145] Spatial attention-weighted high-order facial features and higher-order gait features First, the signal is globally averaged and then passed through a channel attention network composed of an MLP (Multilayer Perceptron). The first layer of the MLP has 64 neurons, the second layer has 32, and the third layer has 64. Specifically:

[0146] High-level facial features and higher-order gait features The face features are obtained by global average pooling. and higher-order gait features These are input into the MLP to obtain channel attention weights. and It is defined as follows:

[0147]

[0148]

[0149] Here, FC(·) represents a fully connected operation.

[0150] Channel attention weight With higher-order facial features Multiplying these yields the channel attention-weighted high-order facial features, which are represented as follows:

[0151]

[0152] Channel attention weight higher-order features of gait Multiplying these yields the channel attention-weighted higher-order gait features, which are represented as follows:

[0153]

[0154] Finally, the channel- and spatial attention-weighted high-order face and gait features are optimized by a similarity objective function, which is defined as follows:

[0155]

[0156] The similarity objective function (17) optimizes the channel attention weights by finding the similarity between faces and gaits. The more similar the face and gait information of each person is, the greater the channel attention weight; the less similar the face and gait information of each person is, the smaller the channel attention weight. This increases the correlation between the gait and face modalities of each person. In the multimodal fusion process, the more correlated the modalities are, the better the fusion performance. Therefore, channel attention improves the recognition rate by increasing the correlation between faces and gaits. At the same time, spatial attention weights improve the differences between the higher-order features of different people, which also effectively improves the recognition rate of people.

[0157] Ultimately, the personnel classification in Case 4 is achieved using SVM (Support Vector Machine). SVM was originally designed for linearly separable sample data; however, in fields such as pattern recognition, image recognition, computer vision, natural language processing, and text processing, non-linear sample data is frequently encountered. To address this problem, SVM introduces a kernel function. This invention utilizes an SVM with a Gaussian kernel function because the Gaussian kernel function can solve the problem of non-linear sample data and contains only one hyperparameter; that is, the SVM has a Gaussian kernel function, the calculation expression of which is as follows:

[0158]

[0159] Among them, (x i ,x j )express and Features of [the text].

[0160] Furthermore, this system is a C-class classification problem, while SVM is a binary classification problem. Therefore, this paper adopts a "one-to-many method" to achieve multi-class classification using SVM. Specifically, the data of one person in the bimodal face and gait dataset is successively designated as the front face (labeled 1), while the data of other people in the bimodal face and gait dataset are designated as the back face (labeled -1). For a C-class classification problem, the one-to-many method only requires constructing C classifiers.

[0161] Finally, the face gait feature fusion algorithm based on the attention mechanism sends the recognition results to the main control module. The main control module issues decision instructions based on the results, which are then sent to the audio sensors at the field layer via the fiber optic switch of the fiber optic ring network to inform the person being identified.

[0162] This fusion algorithm not only solves the problem of face occlusion in video surveillance, but also effectively improves the recognition rate of the surveillance system. Specifically:

[0163] For scenario 2: the video does not contain facial information but does contain gait information. This is a common problem of facial occlusion in face-based video surveillance, but current face-based video surveillance algorithms cannot identify people. Conversely, this algorithm utilizes a spatial attention mechanism guided by an objective function, assigning high weights to key regions of high-order gait features and lower weights to non-key regions. This effectively improves the differences in high-order features between different individuals, thereby significantly increasing the recognition rate. In other words, this algorithm not only solves the problem of face-based video surveillance algorithms being unable to identify people with occluded faces but also improves the recognition rate.

[0164] For scenario 3: the video contains facial information but not gait information, the algorithm uses the objective function to guide the spatial attention mechanism to set large weights on key areas of high-order facial features and small weights on non-key areas, which effectively improves the differences between high-order facial features of different people and thus effectively improves the recognition rate.

[0165] For scenario 4: the video contains both facial and gait information. This algorithm improves the recognition rate by utilizing channel attention and a similarity objective function to increase the correlation between each person's gait and facial modality. Simultaneously, spatial attention weights enhance the differences between higher-order features of different individuals, which also effectively improves the person recognition rate.

Claims

1. A smart building monitoring system based on attention mechanism and gait feature layer fusion, characterized in that, The monitoring system comprises a field layer, a communication network layer, and a monitoring and processing layer. The field layer includes image and audio acquisition devices installed within the building, and the image and audio acquisition devices include cameras and audio sensors; The communication network layer includes multiple fiber optic switches, which form a fiber optic ring network to aggregate video data collected by the field layer and send it to the monitoring and processing layer. At the same time, the feedback signals from the monitoring and processing layer are also sent to the field layer. The monitoring and processing layer includes a display and a main control module containing a face and gait feature fusion algorithm based on an attention mechanism, which is used to process face and gait information and provide feedback results. Specifically, the cameras at the field layer capture video in real time, which is then transmitted to the main control module at the monitoring and processing layer via a fiber optic ring network. The main control module first divides the video into two parts. One part of the video is then divided into frames and images, and the facial region is automatically extracted. If a facial area exists in a video clip, the facial area will be automatically extracted. If a video clip does not contain a face, a black background image is used instead. The face image size is then modified to 124×124×3 and input into a residual network of an attention-based face gait feature layer fusion algorithm to extract high-order face features. For another video clip, if gait information is present in the video clip, it is processed normally; if gait information is absent, it is replaced with a black background image, then modified to 640×640×3, and input into a 3D-CNN based on an attention-based face gait feature layer fusion algorithm to extract high-order gait features. Then, the attention-based face gait feature layer fusion algorithm is used to classify people. There are four possibilities at this point: 1) When the video does not contain facial information or gait information, the monitoring system does not need to identify personnel information; 2) When the video does not contain facial information but contains gait information, the algorithm recognizes people information through 3D-CNN, spatial attention and fully connected layers. In addition, the algorithm uses the spatial attention mechanism to set large weights for key areas of high-order gait features and small weights for non-key areas, which effectively improves the difference between high-order features of different people and thus effectively improves the recognition rate of people. 3) When the video contains facial information but not gait information, the algorithm uses residual networks, spatial attention and fully connected networks to recognize people information. It also uses spatial attention mechanism to set large weights on key areas of high-order facial features and small weights on non-key areas, which effectively improves the difference between high-order facial features of different people and thus effectively improves the recognition rate. 4) When the video contains both facial and gait information, the algorithm increases the correlation between the gait and facial modalities of each person by utilizing channel attention and similarity objective functions, thereby improving the recognition rate. At the same time, spatial attention weights increase the differences between higher-order features of different people, which also effectively improves the recognition rate of people. Finally, the face gait feature fusion algorithm based on the attention mechanism sends the recognition results to the main control module. The main control module issues decision instructions based on the results, which are then sent to the audio sensors at the field layer via the fiber optic switch of the fiber optic ring network to inform the person being identified.

2. The intelligent building monitoring system based on attention mechanism and gait feature layer fusion according to claim 1, characterized in that, The residual network of the face gait feature layer fusion algorithm based on the attention mechanism contains K residual blocks, each of which contains 2 convolutional layers, 3 ReLU activation function layers, 2 normalization layers, one addition operation, and one pooling layer. Convolutional layers are used to extract facial features. For a convolutional layer l∈L, the convolution operation formula for two-dimensional face image data I(i,j) is as follows: Among them, S l,c (i,j) represents the two-dimensional face image data; I(i,j) represents the result of the c-th channel after convolution; K(m,n) represents the two-dimensional convolution kernel; i,j represents the index of the width and height of the two-dimensional image; m,n represents the index of the width and height of the two-dimensional convolution kernel; The dimensional relationship between the output and input after a convolution operation is defined as follows: Among them, w l and h l S represents l,c The width and height dimensions of (i,j); w l-1 and h l-1 Let I(i,j) represent the width and height dimensions. and This indicates the size of the zero padding in the convolution operation; and This represents the stride size in convolution; and Describes the dimension of K(m,n); c l Indicates the number of channels; The activation function layer achieves non-linearity of the features, where the activation function chosen is ReLU, which is defined as follows: ReLU(x) = max(0,x) (3) The normalization layer normalizes features by subtracting the mean and dividing by the variance, specifically defined as: Among them, Z bn,l,c [i,j] represents the output of the c-th channel of the normalized layer; l,c μ is a coefficient to prevent divisor from being zero; l,c and Var l,c Let represent the mean and variance of the c-th channel, which are defined as follows: Where F represents the batch size; This represents the convolution output of the f-th sample; The addition operation alleviates the gradient vanishing problem by adding the input features and the convolution output features. The pooling layer reduces the feature dimension by downsampling. This step uses max pooling. The face image is first used to extract the face region, which is then modified to 124×124×3. This modified region is then input into a residual network to extract higher-order face features. The output dimension of these higher-order face features is calculated using formula (2). The final output of the higher-order face features is as follows:

3. The intelligent building monitoring system based on attention mechanism and gait feature layer fusion according to claim 2, characterized in that, The attention-based face gait feature fusion algorithm's 3D-CNN contains seven 3D convolutional layers and five pooling layers. When the 3D-CNN extracts high-order gait features, the gait video is first segmented into frames, then each frame is modified to 640×640×3 pixels and input into the 3D-CNN to extract high-order gait features. The final output high-order gait features are:

4. The intelligent building monitoring system based on attention mechanism and gait feature layer fusion according to claim 3, characterized in that, High-level facial features and higher-order gait features Each is processed by a spatial attention mechanism to improve the recognition rate; The spatial attention mechanism consists of a 1×1 convolution, a ReLU activation function, and a multiplication operation, which is effective for high-order facial features. Spatial attention weights are represented as: Among them, Conv 1×1 (·) represents a 1×1 convolution, meaning the kernel is 1×1, the stride is 1×1, and the number of channels is 1; ReLU(·) represents the activation function operation; Ξ,ξ represent the network parameters; As shown in formula (2), after a 1×1 convolution, the dimension of the feature becomes 64×1, and after ReLU, the spatial attention weights of the high-order features of the face are obtained. Spatial attention weights With higher-order facial features Multiplication yields spatial attention-weighted high-order facial features, represented as follows: For higher-order features of gait Spatial attention weights are represented as: Spatial attention weights With higher-order facial features Multiplication yields spatial attention-weighted higher-order gait features, represented as follows: Ultimately, spatial attention-weighted high-order facial features are obtained. and higher-order gait features 5. The intelligent building monitoring system based on attention mechanism and gait feature layer fusion according to claim 4, characterized in that, When the video does not contain facial information but contains gait information, and Spatial attention-weighted gait higher-order features A 4-layer fully connected classification network is used; the output layer of the fully connected network has C neurons, and the activation function is the softmax function, which achieves classification C. The specific optimization function is defined as follows: Where θ represents the parameters of the fully connected layer; Υ(·) represents the features output by the fully connected layer; softmax(·) represents the prediction result of the FC layer; K represents the number of training samples for gait information; y i The label is represented by L(·), which is the objective function, and its specific expression is as follows: Among them, y i,j and Let represent the label of the j-th class of the i-th gait data and the output of the softmax layer, respectively; C represents the total number of label classes. During training, most of the gait video images of different people are similar, and the differences are only in the gait regions. Therefore, by using the objective functions (11) and (12) to guide the spatial attention mechanism, we set a large weight for the key regions of the high-order gait features and a small weight for the non-key regions. This effectively improves the difference between the high-order gait features of different people, and thus can effectively improve the recognition rate.

6. The intelligent building monitoring system based on attention mechanism and gait feature layer fusion according to claim 5, characterized in that, When the video contains facial information but not gait information and Spatial attention-weighted high-order facial features The algorithm classifies people using a 4-layer fully connected classification network. During training, most of the facial images of different people are similar, and the differences are only in the regions containing the face. The algorithm uses the objective function and the spatial attention mechanism to set large weights on the key regions of the high-order features of the face and small weights on the non-key regions. This effectively improves the difference between the high-order features of different people, thereby effectively improving the recognition rate.

7. The intelligent building monitoring system based on attention mechanism and gait feature layer fusion according to claim 6, characterized in that, When the video contains both facial information and gait information, and Spatial attention-weighted high-order facial features and higher-order gait features First, it is subjected to global average pooling, and then passed through a channel attention network composed of MLPs. The first layer of the MLP has 64 neurons, the second layer has 32, and the third layer has 64. Specifically: High-level facial features and higher-order gait features The face features are obtained by global average pooling. and higher-order gait features These are input into the MLP to obtain channel attention weights. and It is defined as follows: Where FC(·) represents a fully connected operation; Channel attention weight With higher-order facial features Multiplying these yields the channel attention-weighted high-order facial features, which are represented as follows: Channel attention weight higher-order features of gait Multiplying these yields the channel attention-weighted higher-order gait features, which are represented as follows: Finally, the channel- and spatial attention-weighted high-order face and gait features are optimized by a similarity objective function, which is defined as follows: The similarity objective function (17) optimizes the channel attention weight by finding the similarity between face and gait. The more similar the face and gait information of each person is, the greater the channel attention weight is; the less similar the face and gait information of each person is, the smaller the channel attention weight is. This increases the correlation between the gait and face modalities of each person. In the process of multimodal fusion, the more correlated the modalities are, the better the fusion performance is. Therefore, channel attention improves the recognition rate by increasing the correlation between face and gait. At the same time, the spatial attention weight also improves the difference between the higher-order features of different people, which also effectively improves the recognition rate of people. Finally, the personnel classification in Case 4 is achieved using SVM (Support Vector Machine). The SVM has a Gaussian kernel function, and the calculation expression of the Gaussian kernel function is as follows: Among them, (x i ,x j )express and Features of [the text].

Images