Face feature extraction method and system based on local structure and global semantics, and storage medium

Abstract

The invention provides a face feature extraction method and system based on a local structure and global semanteme, and a storage medium, and the method comprises a face feature extraction step: in a training process, taking a thermal infrared image as the input of a face feature extraction module, carrying out the feature extraction on multiple scales, and obtaining a face feature extraction result; obtaining local structure information and global semantic information; and a comparative learning step: carrying out comparative learning on the extracted face feature information and positive and negative samples so as to close the similarity of the face features of the same identity and depart the distance of the face features of different information. The beneficial effects of the invention are that the method is better in robustness and generalization capability, and can achieve the extraction of human face features more accurately and quickly.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, system and storage medium for extracting facial features based on local structure and global semantics. Background Art

[0002] Facial feature extraction technology is an important research direction in the field of computer vision and pattern recognition. Its goal is to extract discriminative features from facial images for tasks such as face recognition, expression analysis, and identity verification. Facial feature extraction refers to extracting useful information from facial images in order to distinguish different individuals or perform other analyses. Facial features can be local (such as the position and shape of the eyes, nose, and mouth) or global (such as the overall facial texture).

[0003] Traditional facial feature extraction methods often focus on facial feature extraction based on visible light images. In harsh environments (such as extreme lighting environments, occlusion, interference from similar objects, etc.), facial feature extraction technology based on visible light cannot provide reliable facial features. Facial feature extraction based on thermal infrared images can provide more reliable and accurate facial features.

[0004] The core problem of thermal infrared facial features is how to extract features suitable for face recognition based on the characteristics of thermal infrared images, so that the features have good robustness against the difficulties in face recognition (such as changes in facial posture, expression, size, illumination, and partial occlusion). Different from images taken under visible light, thermal infrared images have different imaging principles. The response intensity of thermal infrared images is related to the temperature of the scene, so its images have different meanings from visible light images. Infrared images and visible light images provide different information. Thermal infrared has characteristics such as liveness detection and barrier perspective that are superior to visible light. These characteristics help solve the difficulties in face recognition. Therefore, how to combine the characteristics of thermal infrared imaging and make full use of its characteristics to extract features is a core issue in this research content.

[0005] The methods of facial feature extraction can be divided into feature extraction models based on manually designed features and feature extraction models based on deep neural networks. Feature extraction models based on manually designed features usually extract feature vectors from facial images using methods such as principal component analysis and linear discriminant analysis. In recent years, feature extraction models based on deep neural networks have achieved results far exceeding those of manually designed feature extraction models. Common feature extraction models based on deep neural networks include convolutional neural networks (CNN) and attention-based neural networks (Transformer).

[0006] Defects of background technology:

[0007] In terms of multi-scale feature extraction of faces: In classic face feature extraction technologies, most algorithms only consider face features of a single scale. Most existing multi-scale feature extraction technologies usually use feature pyramids to process features of different scales separately, and are unable to fully learn the local structure and global semantic information of face features during decoding.

[0008] In terms of face identification information: the face identification information contained in the fusion features extracted by the existing face feature extraction technology is often not sufficient, which will result in the extracted image features being unable to provide sufficient face identification information to meet the needs of the task.

[0009] In terms of model speed: The commonly used Transformer architecture has excellent performance, but the model has too many parameters and high computational complexity, which is not conducive to lightweight deployment. Summary of the invention

[0010] The present invention provides a method for extracting facial features based on local structure and global semantics, comprising the following steps:

[0011] Face feature extraction step: During the training process, the thermal infrared image is used as the input of the face feature extraction module, and features are extracted at multiple scales to obtain local structural information and global semantic information;

[0012] Contrastive learning step: The extracted facial feature information is contrasted with the positive and negative samples to narrow the similarity of facial features of the same identity and distance the facial features of different information.

[0013] As a further improvement of the present invention, the facial feature extraction module is based on a multi-scale codec architecture U-Net. The structure of U-Net consists of an encoder, a decoder and a jump connection. The encoder gradually extracts high-level features of the image while reducing the spatial resolution; the decoder gradually restores the spatial resolution and combines the high-resolution feature map of the encoder to reconstruct the output.

[0014] As a further improvement of the present invention, the facial features are further enhanced in the jump connection of the U-Net through the facial feature enhancement module;

[0015] The encoder is responsible for extracting local features of the image. The encoder processes the input image layer by layer to extract features at different levels. The decoder uses image merging operations to gradually reduce the spatial resolution of the feature map while increasing the level of abstraction of the features. By aggregating local information, the encoder can capture a larger range of local structures.

[0016] As a further improvement of the present invention, features at different levels include low-level features, intermediate features and high-level features. Low-level features: in the initial layer of the network, the encoder extracts edge, corner and texture features; intermediate features: capture local patterns and regional information, including the shape of the eyes and the curvature of the eyebrows; high-level features: high-level features can represent the structure and layout of the entire face, helping to identify the overall shape of the face and the relationship between different parts.

[0017] As a further improvement of the present invention, the decoder is the reverse operation of the encoder, which restores the low-resolution feature map to the same resolution as the input image; each decoding layer increases the spatial resolution of the feature map through an image expansion module, and in the process of restoring the feature map, the decoder not only retains and enhances the local detail features, but also combines the global semantic information extracted by the encoder; the jump connection transmits the feature maps of different levels of the encoder to the decoder, so that the decoder can combine local details and global semantic information; the high-resolution feature map extracted by the encoder is directly transmitted to the decoder through the jump connection.

[0018] As a further improvement of the present invention, the face feature extraction module adopts the long sequence modeling model Mamba with linear time complexity as the basic model. In the encoder and decoder of U-Net, after image merging or image expansion, the features will be further enhanced through multi-layer Mamba modules.

[0019] As a further improvement of the present invention, a jump connection transmits feature maps of different levels of the encoder to the decoder, so that the decoder can combine local details and global semantic information. A multi-layer Mamba network is introduced in the face feature enhancement module, and a residual connection is used; the face feature enhancement module enhances the multi-scale local detail information output from the U-Net encoder. In the face feature enhancement module, features of different scales will first pass through a feature enhancement layer constructed based on Mamba to obtain enhanced local face detail information of different scales, and then will be added to the initial input through a residual connection to obtain the final output.

[0020] As a further improvement of the present invention, in the comparative learning step, the facial features of the same identity are regarded as positive samples, and the facial features of different identities are regarded as negative samples. First, the similarity S between the facial features of different identities is calculated. i =[S i，1 , S i ,2 ,S i，3 ,...,S i，N ], the similarity calculation method between sample i and sample j is as follows:

[0021] S i，j =sim(F i , Fj )

[0022]

[0023] Where F represents facial features, sim(·) represents similarity calculation, and ||·||2 represents 12 regularization. In a training batch, based on the similarity between positive and negative samples, the loss function of the contrastive learning step is calculated as follows:

[0024]

[0025] The present invention also provides a facial feature extraction system based on local structure and global semantics, comprising: a memory, a processor, and a computer program stored on the memory, wherein the computer program is configured to implement the steps of the facial feature extraction method of the present invention when called by the processor.

[0026] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the facial feature extraction method of the present invention when called by a processor.

[0027] The beneficial effects of the present invention are: the present invention has better robustness and generalization ability, and can realize the extraction of facial features more accurately and quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is the algorithm architecture diagram;

[0029] Figure 2 This is the Mamba layer architecture diagram;

[0030] Figure 3 This is the architecture diagram of the facial feature enhancement module. DETAILED DESCRIPTION

[0031] The present invention discloses a facial feature extraction method based on local structure and global semantics, which extracts more multi-scale facial identification features from a face, thereby providing highly robust and highly discriminative facial features.

[0032] 1. Invention Architecture

[0033] The main modules of the algorithm proposed in the present invention include: a facial feature extraction step (facial feature extraction architecture) based on the local structure and global semantics of the multi-scale codec architecture U-Net, a facial feature enhancement module, and a comparative learning step (comparative learning module). During the training process, the thermal infrared image is used as the input of the facial feature extraction module, and feature extraction is performed at multiple scales to obtain local structural information and global semantic information. In order to successfully retain more facial detail information during the facial feature extraction process, the present invention introduces a facial feature enhancement module, and further enhances the facial features in the jump connection of U-Net, so that the fused facial features contain more detailed information. Finally, the extracted facial feature information will be compared with positive and negative samples for learning, so as to narrow the similarity of facial features of the same identity and push away the distance of facial features of different information.

[0034] 2. Facial feature extraction architecture based on U-Net’s local structure and global semantics

[0035] In order to extract facial features containing local structure and global semantic information, the present invention adopts U-Net based on multi-scale codec architecture as the backbone network for facial feature extraction. The structure of U-Net consists of encoder, decoder and skip connection. The encoder gradually extracts high-level features of the image while reducing the spatial resolution; the decoder gradually restores the spatial resolution and combines the high-resolution feature map of the encoder to reconstruct the output. The architecture of the present invention is as follows: Figure 1 shown.

[0036] In the present invention, the encoder is responsible for extracting local features of the image. The U-Net encoder extracts features at different levels by processing the input image layer by layer. These include: low-level features: in the initial layer of the network, the encoder extracts low-level features such as edges, corners and textures. These features are very important for distinguishing the basic structures of the face such as the position and shape of the eyes, nose and mouth. Intermediate features: as the number of layers increases, the features extracted by the encoder become more abstract, and can capture local patterns and regional information, such as the shape of the eyes, the curvature of the eyebrows, etc. High-level features: at a deeper level, the encoder extracts more abstract high-level features, which can represent the structure and layout of the entire face and help identify the overall shape of the face and the relationship between different parts. In addition, in the decoder of the present invention, the image merging operation is used to gradually reduce the spatial resolution of the feature map, while increasing the abstraction of the features, so that the encoder can capture a wider range of local structures by aggregating local information.

[0037] The role of the decoder of the present invention in facial feature extraction is mainly reflected in restoring the spatial resolution of the image, enhancing feature details, combining global semantic information, and generating high-quality feature maps. The decoder is the reverse operation of the encoder, restoring the low-resolution feature map to the same resolution as the input image. In the present invention, each decoding layer increases the spatial resolution of the feature map through an image expansion module. In addition, in the process of restoring the feature map, the decoder not only retains and enhances the local detail features, but also combines the global semantic information extracted by the encoder. The jump connection plays a key role in this process. The jump connection transmits the feature maps of different levels of the encoder to the decoder, so that the decoder can combine local details and global semantic information. This combination helps to generate a feature map with global understanding and rich details. The high-resolution feature map extracted by the encoder is directly transmitted to the decoder through the jump connection, avoiding the problem of information loss that may occur during the feature extraction process. In this way, the decoder can retain more original detail information when restoring the feature map.

[0038] 3. Mamba-based facial feature extraction module

[0039] The commonly used models in the face extraction module are convolutional neural networks and Transformer, but convolutional neural networks cannot efficiently model long sequences. Transformer can efficiently model long sequences, but the computational complexity and consumption are high. The present invention uses Mamba, a long sequence modeling model with linear time complexity, as the basic model of the face feature extraction module.

[0040] In the encoder and decoder of U-Net, after image merging or image expansion, the features will be further enhanced through multi-layer Mamba modules. The schematic diagram of Mamba module is as follows: Figure 2 As shown. The core principle of Mamba lies in its selective state space model (SSM), which is a recursive neural network (RNN) that can selectively update and process the state according to the current input. Unlike traditional RNN, SSM can flexibly choose which information to retain and which to discard, thereby effectively reducing computational redundancy and information overload. The time complexity of the Mamba module increases linearly with the improvement of image resolution, and the number of parameters is much lower than that of the Transformer model. The facial feature extraction module built based on Mamba has a lower number of parameters, less running consumption, and can achieve lightweight deployment, so that the facial feature extraction method proposed in the present invention can quickly extract reliable facial features.

[0041] 4. Facial feature enhancement module

[0042] The jump connection passes the feature maps of different levels of the encoder to the decoder, so that the decoder can combine local details and global semantic information. This method can make the extracted facial feature information contain more local details and global semantic information. In order to further enhance the detail information, the present invention introduces a multi-layer Mamba network in the face enhancement module and uses residual connections. The face feature enhancement module can enhance the multi-scale local detail information output in the U-Net encoder, so that the U-Net decoder retains more detail information during the multi-scale feature fusion process.

[0043] In the face feature enhancement module, features of different scales will first pass through the feature enhancement layer built based on Mamba to obtain enhanced local face detail information of different scales. After that, they will be added to the initial input through residual connection to obtain the final output. The schematic diagram of the face feature enhancement module is shown in the figure. Figure 3 shown.

[0044] 5. Comparative Learning Module

[0045] In order to enable the facial feature extraction method based on local structure and global semantics to extract more discriminative features, the present invention introduces a contrastive learning module, which calculates the similarity between the extracted facial features, shortens the similarity of faces with the same identity, and extends the similarity between faces with different identities.

[0046] In contrastive learning, facial features of the same identity are considered as positive samples, and facial features of different identities are considered as negative samples. The present invention first calculates the similarity S between different facial features. i =[S i，1 , S i，2 , S i，3 ,...,S i，N ], the similarity calculation method between sample i and sample j is as follows:

[0047] S i，j =sim(F i , F j )

[0048]

[0049] Among them, F represents facial features, sim(·) represents similarity calculation, and ||·||2 represents 12 regularization. In a training batch, based on the similarity between positive and negative samples, the loss function of the contrastive learning module is calculated as follows:

[0050]

[0051] Through contrastive learning, the discriminability and reliability of extracted facial features can be enhanced.

[0052] The specific innovations of the present invention are as follows:

[0053] 1. In order to retain more local structures and global semantic information in the process of facial feature extraction, a multi-scale encoder-decoder architecture U-Net is introduced to extract more multi-scale facial features through multi-scale fusion and jump connections.

[0054] 2. In order to retain more detailed information in the extracted facial features, a facial feature enhancement module is introduced. Through multi-scale feature enhancement, more facial detail information can be retained during the extraction process.

[0055] 3. In order to extract more discriminative and unique facial features from images, the present invention proposes a module based on contrastive learning, which shortens the distance between faces of the same identity by comparing positive and negative samples, thereby improving the discriminativeness of the features.

[0056] 4. In order to enable lightweight deployment of the facial feature extraction method, the present invention introduces a long sequence modeling network Mamba with linear complexity to achieve lightweight and fast facial feature extraction.

[0057] The beneficial effect of the present invention is that the facial feature extraction method based on local structure and global semantics has better robustness and generalization ability, and can extract facial features more accurately and quickly.

[0058] 1. The multi-scale codec structure is used to extract the local structure and global semantic features of the face, and more spatial information and details are retained during the face feature extraction process.

[0059] 2. By introducing the long sequence modeling network Mamba with linear complexity, lightweight and fast facial feature extraction is achieved, making the method lightweight to deploy.

[0060] 3. By introducing contrastive learning and face alignment modules, the similarity of faces with the same identity is shortened, and the discriminability of facial features extracted by the facial feature extraction method is improved.

[0061] 4. By proposing a facial feature enhancement module, multi-scale facial features are enhanced and combined through jump connections to retain more facial detail information as well as local structure and global semantic information.

[0062] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A facial feature extraction method based on local structure and global semantics, characterized in that: The steps include: Face feature extraction step: During the training process, the thermal infrared image is used as the input of the face feature extraction module, and features are extracted at multiple scales to obtain local structural information and global semantic information; Contrastive learning step: The extracted facial feature information is contrasted with the positive and negative samples to narrow the similarity of facial features of the same identity and distance the facial features of different information.

2. The facial feature extraction method according to claim 1, characterized in that: The face feature extraction module is based on the multi-scale codec architecture U-Net. The structure of U-Net consists of an encoder, a decoder and a jump connection. The encoder gradually extracts high-level features of the image while reducing the spatial resolution. The decoder gradually restores the spatial resolution and combines the encoder's high-resolution feature maps to reconstruct the output.

3. The facial feature extraction method according to claim 2, characterized in that: The facial features are further enhanced in the jump connection of U-Net through the facial feature enhancement module; The encoder is responsible for extracting local features of the image. The encoder processes the input image layer by layer to extract features at different levels. The decoder uses image merging operations to gradually reduce the spatial resolution of the feature map while increasing the level of abstraction of the features. By aggregating local information, the encoder can capture a larger range of local structures.

4. The method for extracting facial features according to claim 3, wherein: Features at different levels include low-level features, mid-level features and high-level features. Low-level features: In the initial layer of the network, the encoder extracts edge, corner and texture features; intermediate features: capture local patterns and regional information, including the shape of the eyes and the curvature of the eyebrows; high-level features: high-level features can represent the structure and layout of the entire face, helping to identify the overall shape of the face and the relationship between different parts.

5. The method for extracting facial features according to claim 3, characterized in that: The decoder is the reverse operation of the encoder, restoring the low-resolution feature map to the same resolution as the input image; each decoding layer increases the spatial resolution of the feature map through the image dilation module. In the process of restoring the feature map, the decoder not only retains and enhances the local detail features, but also combines the global semantic information extracted by the encoder; the skip connection passes the feature maps of different levels of the encoder to the decoder, allowing the decoder to combine local details and global semantic information; the high-resolution feature map extracted by the encoder is directly passed to the decoder through the skip connection.

6. The method for extracting facial features according to claim 3, characterized in that: The face feature extraction module uses the long sequence modeling model Mamba with linear time complexity as the basic model. In the encoder and decoder of U-Net, after image merging or image expansion, the features will be further enhanced through multi-layer Mamba modules.

7. The method for extracting facial features according to claim 6, wherein: The jump connection transfers the feature maps of different levels of the encoder to the decoder, allowing the decoder to combine local details and global semantic information. The multi-layer Mamba network is introduced in the face feature enhancement module, and residual connections are used; The face feature enhancement module enhances the multi-scale local detail information output by the U-Net encoder. In the face feature enhancement module, features of different scales will first pass through the feature enhancement layer built based on Mamba to obtain enhanced local face detail information of different scales, which will then be added to the initial input through a residual connection to obtain the final output.

8. The method for extracting facial features according to claim 1, characterized in that: In the contrastive learning step, facial features of the same identity are regarded as positive samples, and facial features of different identities are regarded as negative samples. First, the similarity S between different facial features is calculated. i =[S i，1 , S i，2 , S i，3 ,...,S i，N ], the similarity calculation method between sample i and sample j is as follows: S i，j =sim(F i ,F j ) Where F represents facial features, sim(·) represents similarity calculation, and ||·||2 represents 12 regularization. In a training batch, based on the similarity between positive and negative samples, the loss function of the contrastive learning step is calculated as follows:

9. A facial feature extraction system based on local structure and global semantics, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the facial feature extraction method according to any one of claims 1 to 8 when called by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the facial feature extraction method according to any one of claims 1 to 8 when called by a processor.

Images