A face recognition method and device based on biometric features and spatio-temporal fusion
By using a CNN-Transformer hybrid architecture face recognition model that combines PPG signals and network environment information, the accuracy and robustness of face recognition in complex environments are solved. This model achieves the fusion of dynamic multimodal biometrics and spatiotemporal context, thereby improving the accuracy and security of recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING GZT NETWORK TECH
- Filing Date
- 2025-04-02
- Publication Date
- 2026-05-12
AI Technical Summary
Current facial recognition technologies lack accuracy and robustness in complex environments, mainly because they rely on single-modal image information and do not integrate dynamic multimodal biometrics and spatiotemporal context information.
A face recognition model using a CNN-Transformer hybrid architecture combines PPG signals, face images, and access network environment information. It performs recognition through feature extraction, fusion, and classification modules. The model training uses adaptive weights and an L2 regularized loss function.
It improves the accuracy and security of facial recognition. Through dynamic multimodal biometric fusion and spatiotemporal context analysis, it enhances the robustness of the recognition system and prevents the forgery and tampering of static biometrics.
Smart Images
Figure CN120375439B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image recognition technology, specifically to a face recognition method and a face recognition device based on biometrics and spatiotemporal fusion. Background Technology
[0002] In existing technologies, facial recognition primarily relies on image processing and machine learning algorithms. The technical solution typically includes the following steps:
[0003] Facial image acquisition: capturing facial images using devices such as cameras.
[0004] Preprocessing: The acquired face images are subjected to preprocessing operations such as grayscale conversion, noise reduction, and normalization to improve the accuracy of subsequent processing.
[0005] Feature extraction: Using algorithms to extract information that can represent individual characteristics from preprocessed face images, such as facial contours, textures, and key points.
[0006] Feature matching and recognition: The extracted facial features are compared with feature templates in the database, a similarity score is calculated, and a decision is made based on the score to determine the identity.
[0007] The disadvantages of existing technology are:
[0008] In complex environments (such as changes in lighting, occlusion, and angle), the accuracy of face recognition is significantly affected because it relies on image information from only a single modality.
[0009] The lack of integration of dynamic multimodal biometrics (such as facial expressions and head posture) and spatiotemporal contextual information (such as changes between consecutive frames and changes in the position of the face in the image) results in insufficient robustness of the recognition system when facing complex scenes. Summary of the Invention
[0010] The purpose of this invention is to provide a face recognition method based on biometrics and spatiotemporal fusion to at least solve one of the above-mentioned technical problems.
[0011] One aspect of the present invention provides a face recognition method based on biometrics and spatiotemporal fusion, the face recognition method based on biometrics and spatiotemporal fusion comprising:
[0012] The PPG signal and facial image information are acquired through a camera device;
[0013] Obtain network environment information;
[0014] Obtain the trained face recognition model;
[0015] The PPG signal, facial image information, and network environment access information are input into the facial recognition model to obtain the recognition result.
[0016] Optionally, the face recognition model is a CNN-Transformer hybrid architecture face recognition model.
[0017] Optionally, the CNN-Transformer hybrid architecture face recognition model includes:
[0018] The input layer is used to acquire the PPG signal, face image information, and network environment access information.
[0019] The CNN feature extraction module is used to extract facial image features from the facial image information.
[0020] An LSTM sequence processing module is used to obtain PPG features based on the PPG signal and to obtain network environment information features based on the access network environment information.
[0021] The Transformer global dependency modeling module is used to receive the face image features, PPG features and network environment information features, and generate global features after obtaining global dependencies through a multi-head self-attention mechanism.
[0022] The feature fusion module is used to fuse the face image features, PPG features, network environment information features, and global features to obtain fused features.
[0023] A classification module is used to obtain the fusion features and obtain the recognition result based on the fusion features.
[0024] Optionally, the face recognition method based on biometrics and spatiotemporal fusion further includes:
[0025] The CNN-Transformer hybrid architecture face recognition model is trained.
[0026] Optionally, training the CNN-Transformer hybrid architecture face recognition model includes:
[0027] Obtain training data;
[0028] Obtain the CNN-Transformer hybrid architecture face recognition model;
[0029] The CNN-Transformer hybrid architecture face recognition model is trained using training data.
[0030] Optionally, the CNN-Transformer hybrid architecture face recognition model adopts the following loss function:
[0031] in,
[0032] L represents the loss function, x represents the feature vector output by the model (usually a high-dimensional vector containing features extracted from the input data), y represents the true label, and C represents the number of classes. i Let represent the i-th element in the true label vector y. If the sample belongs to the i-th class, then y... i =1, otherwise y i =0, W i The weight vector representing category i, Represents the weight vector W i The dot product with the feature vector x is used to measure the similarity between them, ||W i || represents the weight vector W i The L2 norm of x, ||x|| represents the L2 norm of the feature vector x, ∈ represents a positive number less than 10 to prevent the denominator from being zero and to ensure the stability of the calculation, α represents a positive coefficient used to control the influence of the adaptive weight term on the overall loss function, (1-y i ) is a constraint that ensures only categories without true labels are considered. If a sample belongs to the i-th category, then (1-y) i ) = 0, otherwise (1-y i ) = 1, This represents the weight vector W (where W is the weight vector for all classes). i The square of the L2 norm of the set is the sum of the squares of all weighted elements, and β represents a positive coefficient.
[0033] This application also provides a face recognition device based on biometrics and spatiotemporal fusion, the face recognition device based on biometrics and spatiotemporal fusion includes:
[0034] Information acquisition module, the information acquisition module is used to acquire facial image information, PPG signal and access network environment information transmitted by the camera device;
[0035] A face recognition model acquisition module, which is used to acquire a trained face recognition model;
[0036] The recognition result acquisition module is used to input the PPG signal, face image information and access network environment information into the face recognition model to obtain the recognition result.
[0037] Beneficial effects
[0038] The face recognition method based on biometrics and spatiotemporal fusion proposed in this application has the following advantages:
[0039] Dynamic Multimodal Biometric Recognition: This invention proposes to dynamically fuse multiple biometric features (such as facial recognition and PPG signals) to improve the accuracy and security of identity authentication. By dynamically monitoring and analyzing changes in different biometric features, the forgery and tampering of static biometric data can be effectively prevented.
[0040] Spatiotemporal context fusion: Introducing spatiotemporal contextual information enables deeper understanding and analysis of facial images. By modeling the temporal sequence and spatial relationships of facial images, dynamic changes and contextual information within the images can be effectively captured, improving the robustness of facial recognition. Attached Figure Description
[0041] Figure 1 This is a flowchart illustrating a face recognition method based on biometrics and spatiotemporal fusion according to an embodiment of this application.
[0042] Figure 2 This is a schematic diagram of an electronic device used to implement a face recognition method based on biometrics and spatiotemporal fusion according to an embodiment of this application. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0044] like Figure 1 The face recognition methods based on biometrics and spatiotemporal fusion shown include:
[0045] Step 1: Acquire PPG signals and facial image information using a camera device;
[0046] Step 2: Obtain network environment information;
[0047] Step 3: Obtain the trained face recognition model;
[0048] Step 4: Input the PPG signal, face image information, and access network environment information into the face recognition model to obtain the recognition result.
[0049] The face recognition method based on biometrics and spatiotemporal fusion proposed in this application has the following advantages:
[0050] Dynamic Multimodal Biometric Recognition: This invention proposes to dynamically fuse multiple biometric features (such as facial recognition and PPG signals) to improve the accuracy and security of identity authentication. By dynamically monitoring and analyzing changes in different biometric features, the forgery and tampering of static biometric data can be effectively prevented.
[0051] Spatiotemporal context fusion: Introducing spatiotemporal contextual information enables deeper understanding and analysis of facial images. By modeling the temporal sequence and spatial relationships of facial images, dynamic changes and contextual information within the images can be effectively captured, improving the robustness of facial recognition.
[0052] In this embodiment, the PPG signal can be obtained through the following method:
[0053] The basic principle of PPG signal acquisition is to use imaging equipment to acquire video of a region of the human skin. Changes in blood volume cause changes in the brightness of the skin region, and these changes in brightness cause changes in the grayscale value of each frame in the video. By analyzing these changes in grayscale values, the PPG signal can be obtained.
[0054] In this embodiment, accessing network environment information may include timestamps, geographical location, device fingerprints, network environment, and other information.
[0055] In this embodiment, the face recognition model is a CNN-Transformer hybrid architecture face recognition model.
[0056] In this embodiment, the CNN-Transformer hybrid architecture face recognition model includes:
[0057] The input layer is used to acquire the PPG signal, face image information, and network environment access information.
[0058] The CNN feature extraction module is used to extract facial image features from the facial image information.
[0059] An LSTM sequence processing module is used to obtain PPG features based on the PPG signal and to obtain network environment information features based on the access network environment information.
[0060] The Transformer global dependency modeling module is used to receive the face image features, PPG features and network environment information features, and generate global features after obtaining global dependencies through a multi-head self-attention mechanism.
[0061] The feature fusion module is used to fuse the face image features, PPG features, network environment information features, and global features to obtain fused features.
[0062] A classification module is used to obtain the fusion features and obtain the recognition result based on the fusion features.
[0063] In this embodiment, the face recognition method based on biometrics and spatiotemporal fusion further includes:
[0064] The CNN-Transformer hybrid architecture face recognition model is trained.
[0065] In this embodiment, training the CNN-Transformer hybrid architecture face recognition model includes:
[0066] Obtain training data;
[0067] Obtain the CNN-Transformer hybrid architecture face recognition model;
[0068] The CNN-Transformer hybrid architecture face recognition model is trained using training data.
[0069] In this embodiment, the CNN-Transformer hybrid architecture face recognition model uses the following loss function:
[0070] in,
[0071] L represents the loss function, x represents the feature vector output by the model (usually a high-dimensional vector containing features extracted from the input data), y represents the true label, and C represents the number of classes. i Let represent the i-th element in the true label vector y. If the sample belongs to the i-th class, then y... i =1, otherwise y i =0, W i The weight vector representing category i, Represents the weight vector W i The dot product with the feature vector x is used to measure the similarity between them, ||W i || represents the weight vector W iThe L2 norm of x, ||x|| represents the L2 norm of the feature vector x, ∈ represents a positive number less than 10 to prevent the denominator from being zero and to ensure the stability of the calculation, α represents a positive coefficient used to control the influence of the adaptive weight term on the overall loss function, (1-y i ) is a constraint that ensures only categories without true labels are considered. If a sample belongs to the i-th category, then (1-y) i ) = 0, otherwise (1-y i ) = 1, This represents the weight vector W (where W is the weight vector for all classes). i The square of the L2 norm of the set is the sum of the squares of all weighted elements, and β represents a positive coefficient that can be set as needed.
[0072] Using the loss function of this application has the following advantages:
[0073] 1. By converting the cross-entropy loss function into an angle-based form, the angular information in the face recognition task is better utilized, which helps to improve the recognition accuracy of the model.
[0074] 2. The adaptive weighting mechanism enables the model to dynamically adjust its contribution to model parameter updates based on the difficulty of the samples. This helps the model learn difficult samples more efficiently, thereby improving overall performance.
[0075] 3. The addition of L2 regularization helps prevent overfitting and improves the model's generalization ability.
[0076] This application also provides a face recognition device based on dynamic multimodal biometrics and spatiotemporal context fusion. The face recognition device includes an information acquisition module, a face recognition model acquisition module, and a recognition result acquisition module.
[0077] The information acquisition module is used to acquire facial image information, PPG signals, and network environment information transmitted by the camera device;
[0078] The face recognition model acquisition module is used to acquire a trained face recognition model;
[0079] The recognition result acquisition module is used to input the PPG signal, face image information, and access network environment information into the face recognition model to obtain the recognition result.
[0080] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of this embodiment, and will not be repeated here.
[0081] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the above-described face recognition method based on biometrics and spatiotemporal fusion.
[0082] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described face recognition method based on biometrics and spatiotemporal fusion.
[0083] Figure 2 This is an exemplary structural diagram of an electronic device capable of implementing a face recognition method based on biometrics and spatiotemporal fusion according to an embodiment of this application.
[0084] like Figure 2 As shown, the electronic device includes an input device 501, an input interface 502, a central processing unit 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, central processing unit 503, memory 504, and output interface 505 are interconnected via a bus 507. The input device 501 and output device 506 are connected to the bus 507 via the input interface 502 and output interface 505, respectively, and thus connected to other components of the electronic device. Specifically, the input device 501 receives input information from the outside and transmits it to the central processing unit 503 via the input interface 502. The central processing unit 503 processes the input information based on computer-executable instructions stored in the memory 504 to generate output information, temporarily or permanently storing the output information in the memory 504, and then transmitting the output information to the output device 506 via the output interface 505. The output device 506 outputs the output information to the outside of the electronic device for user use.
[0085] In other words, Figure 2 The illustrated electronic device may also be implemented as including: a memory storing computer-executable instructions; and one or more processors, which can be coupled when executing the computer-executable instructions. Figure 1 The described face recognition method is based on the fusion of biometrics and spatiotemporal features.
[0086] In one embodiment, Figure 2 The electronic device shown can be implemented as including: a memory 504 configured to store executable program code; and one or more processors configured to run the executable program code stored in the memory 504 to perform the face recognition method based on biometrics and spatiotemporal fusion in the above embodiments.
[0087] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0088] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0089] Computer-readable media include both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, DVD or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.
[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutively marked blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or the overall flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0091] In this embodiment, the processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0092] Memory can be used to store computer programs and / or modules. The processor implements various functions of the device / terminal equipment by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). In addition, memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0093] In this embodiment, if the modules / units integrated into the device / terminal equipment are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.
[0094] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0095] Furthermore, it is clear that the word "comprising" does not exclude other units or steps. Multiple units, modules, or devices recited in the apparatus claims may also be implemented by a single unit or overall apparatus via software or hardware.
[0096] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.
Claims
1. A face recognition method based on biometrics and spatiotemporal fusion, characterized in that, The face recognition method based on biometrics and spatiotemporal fusion includes: The PPG signal and facial image information are acquired through a camera device; Obtain network environment information; Obtain the trained face recognition model; The PPG signal, facial image information, and network environment access information are input into the facial recognition model to obtain the recognition result; The face recognition model is a CNN-Transformer hybrid architecture face recognition model; The CNN-Transformer hybrid architecture face recognition model includes: The input layer is used to acquire the PPG signal, face image information, and network environment access information. The CNN feature extraction module is used to extract facial image features from the facial image information. An LSTM sequence processing module is used to obtain PPG features based on the PPG signal and to obtain network environment information features based on the access network environment information. The Transformer global dependency modeling module is used to receive the face image features, PPG features and network environment information features, and generate global features after obtaining global dependencies through a multi-head self-attention mechanism. The feature fusion module is used to fuse the face image features, PPG features, network environment information features, and global features to obtain fused features. A classification module is used to obtain the fusion features and obtain the recognition result based on the fusion features.
2. The face recognition method based on biometrics and spatiotemporal fusion as described in claim 1, characterized in that, The face recognition method based on biometrics and spatiotemporal fusion further includes: The CNN-Transformer hybrid architecture face recognition model is trained.
3. The face recognition method based on biometrics and spatiotemporal fusion as described in claim 2, characterized in that, The training of the CNN-Transformer hybrid architecture face recognition model includes: Obtain training data; Obtain the CNN-Transformer hybrid architecture face recognition model; The CNN-Transformer hybrid architecture face recognition model is trained using training data.
4. The face recognition method based on biometrics and spatiotemporal fusion as described in claim 3, characterized in that, The CNN-Transformer hybrid architecture face recognition model uses the following loss function: ;in, L represents the loss function, x represents the feature vector output by the model (a high-dimensional vector containing features extracted from the input data), y represents the true label, and C represents the number of categories. : Represents the i-th element in the true label vector y. If the sample belongs to the i-th category, then... =1, otherwise =0、 The weight vector representing category i, Represents the weight vector The dot product with the feature vector x is used to measure the similarity between them. Represents the weight vector L2 norm The L2 norm of the eigenvector x is represented by... This represents a positive number less than 10, used to prevent the denominator from being zero and to ensure the stability of the calculation. This represents a positive coefficient used to control the impact of the adaptive weighting term on the overall loss function. This is a constraint to ensure that only categories with non-true labels are considered; if a sample belongs to the i-th category, then... =0, otherwise =1、 Represents the weight vector W The square of the L2 norm, i.e., the sum of the squares of all weighted elements. β It represents a positive coefficient.
5. A face recognition device based on biometrics and spatiotemporal fusion, characterized in that, The facial recognition device based on biometrics and spatiotemporal fusion includes: Information acquisition module, the information acquisition module is used to acquire facial image information, PPG signal and access network environment information transmitted by the camera device; A face recognition model acquisition module, which is used to acquire a trained face recognition model; The recognition result acquisition module is used to input the PPG signal, face image information and access network environment information into the face recognition model to obtain the recognition result; The face recognition model is a CNN-Transformer hybrid architecture face recognition model; The CNN-Transformer hybrid architecture face recognition model includes: The input layer is used to acquire the PPG signal, face image information, and network environment access information. The CNN feature extraction module is used to extract facial image features from the facial image information. An LSTM sequence processing module is used to obtain PPG features based on the PPG signal and to obtain network environment information features based on the access network environment information. The Transformer global dependency modeling module is used to receive the face image features, PPG features and network environment information features, and generate global features after obtaining global dependencies through a multi-head self-attention mechanism. The feature fusion module is used to fuse the face image features, PPG features, network environment information features, and global features to obtain fused features. A classification module is used to obtain the fusion features and obtain the recognition result based on the fusion features.