A data processing method, apparatus, device, medium, and program product

Abstract

Embodiments of the present application provide a data processing method, device, equipment, medium and program product, which are used to improve the recognition accuracy of living body data. It can be applied to the fields of artificial intelligence, image recognition, living body detection, etc. Including: obtaining a to-be-processed image, the to-be-processed image including a biological feature of a to-be-recognized object; calling an encoder of a target image generation network to perform feature extraction on the to-be-processed image to obtain a to-be-processed feature; performing discretization processing on the to-be-processed feature based on a target codebook vector set to obtain a discretized feature; calling a decoder of the target image generation network to perform decoding processing on the discretized feature to obtain a reconstructed image of the to-be-processed image; determining a feature residual based on the to-be-processed feature and the discretized feature, and determining a pixel residual based on the reconstructed image and the to-be-processed image; calling a target classification network to perform recognition processing on the feature residual and the pixel residual to obtain a recognition result of the to-be-processed image.

Description

Technical Field

[0001] This application relates to artificial intelligence, and more particularly to a data processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, liveness detection, as an important technology in the field of identity verification, has become a key research direction for preventing forgery attacks (such as photos, videos, masks, etc.). More and more liveness detection technologies are being applied to real-world products, such as smart facial recognition payment and access control services.

[0003] Most current liveness detection technologies are based on deep learning classification models. However, the detection performance of deep learning-based classification models is largely determined by the quality and quantity of samples; the models can only make good judgments on samples that have been trained on. For newly emerging negative samples, it is necessary to manually accumulate and construct negative samples, and to continuously optimize and iterate the model through the accumulated negative samples in order to be able to perceive new negative samples. The inability to perceive negative samples that have not been trained on results in insufficient recognition accuracy of deep learning-based models.

[0004] Therefore, there is an urgent need to provide a solution that can improve the accuracy of liveness data identification. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, medium, and program product for improving the accuracy of liveness data identification.

[0006] In view of this, this application provides a data processing method, comprising: acquiring an image to be processed, the image to be processed including biometric features of an object to be identified; invoking the encoder of a target image generation network to extract features from the image to be processed to obtain features to be processed, the target image generation network being a pre-trained network; discretizing the features to be processed based on a target codebook vector set to obtain discretized features, the target codebook vector set being a feature space trained on a real image and including feature vectors of the real image; invoking the decoder of the target image generation network to decode the discretized features to obtain a reconstructed image of the image to be processed; determining feature residuals based on the features to be processed and the discretized features, and determining pixel residuals based on the reconstructed image and the image to be processed; and invoking a target classification network to perform recognition processing on the feature residuals and the pixel residuals to obtain a recognition result of the image to be processed, the target classification network being a pre-trained network.

[0007] Another aspect of this application provides a data processing apparatus, including: an acquisition module for acquiring an image to be processed, the image to be processed including biometric features of an object to be identified;

[0008] The processing module is used to call the encoder of the target image generation network to extract features from the image to be processed, thereby obtaining the features to be processed. The target image generation network is a pre-trained network. The module then discretizes the features to be processed based on a target codebook vector set, which is a feature space trained on real images and includes feature vectors from real images. Next, the module calls the decoder of the target image generation network to decode the discretized features, thereby obtaining a reconstructed image of the image to be processed. Finally, the module determines feature residuals based on the features to be processed and the discretized features, and determines pixel residuals based on the reconstructed image and the image to be processed. Finally, the module calls a target classification network to perform recognition processing on the feature residuals and the pixel residuals, thereby obtaining the recognition result of the image to be processed. The target classification network is a pre-trained network.

[0009] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to perform a fusion process on the feature residual and the pixel residual to obtain a fused residual;

[0010] The target classification network is invoked to perform recognition processing on the fusion residual in order to obtain the recognition result of the image to be processed.

[0011] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to perform fusion processing on the feature residual and the pixel residual based on an attention mechanism to obtain the fused residual.

[0012] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to perform feature learning on the feature residual based on the first network structure and the attention mechanism to obtain a first intermediate feature;

[0013] Based on the second network structure and the attention mechanism, the first intermediate feature and the pixel residual are fused to obtain the fused residual.

[0014] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to extract features from the feature residual through a first convolutional layer to obtain a second intermediate feature, wherein the first convolutional layer is included in the first network structure.

[0015] Based on the attention mechanism, feature learning is performed on the second intermediate feature and the feature to be processed to obtain the first intermediate feature.

[0016] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to upsample the first intermediate feature through a sampling network to obtain a third intermediate feature, wherein the sampling network is included in the second network structure.

[0017] The pixel residual is used to extract features through the second convolutional layer and the global pooling layer to obtain the fourth intermediate feature. The second convolutional layer and the global pooling layer are included in the second network structure.

[0018] The third and fourth intermediate features are concatenated to obtain the fifth intermediate feature;

[0019] Attention is calculated on the fifth intermediate feature in both the channel dimension and the spatial location dimension to obtain the location attention feature and the channel attention feature.

[0020] The positional attention feature and the channel attention feature are multiplied and added together to obtain the fusion residual.

[0021] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to call the index prediction network to perform index prediction on the feature to be processed in order to obtain a first feature index, wherein the index prediction network is a pre-trained network.

[0022] Based on the first feature index, a query is performed in the target codebook vector set to obtain the discretized feature.

[0023] In one possible design, in another implementation of another aspect of the embodiments of this application, the acquisition module is used to acquire a training image set, an initial image generation network, an initial classification network, and the target codebook vector set. The training image set includes a first training image and a second training image, wherein the first label of the first training image is a real image, and the second label of the second training image is a fake image.

[0024] This processing module is used for

[0025] The initial image generation network is invoked to extract features from the training images in the training image set to obtain the first feature;

[0026] The first feature is discretized based on the target codebook vector set to obtain the second feature;

[0027] The initial image generation network is invoked to reconstruct the image based on the second feature to obtain the first reconstructed image;

[0028] Based on the first reconstructed image, the first training image, and the second training image, the training pixel residual and the training feature residual are obtained;

[0029] The initial classification network is invoked to classify the training pixel residual and the training feature residual to obtain the training classification result;

[0030] Calculate the classification loss value based on the training classification result, the first label, and the second label;

[0031] The reconstruction loss value is calculated based on the first feature, the second feature, the first reconstructed image, and the training images in the training image set;

[0032] The initial classification network and the initial image generation network are trained based on the classification loss value and the reconstruction loss value to obtain the target classification network and the target image generation network.

[0033] In one possible design, in another implementation of another aspect of the embodiments of this application, the acquisition module is used to acquire a third training image, an original image generation network, and an initial codebook vector set, wherein the third label of the third training image is a real image.

[0034] The original image generation network is invoked to extract features from the third training image to obtain the third feature;

[0035] The third feature is discretized based on the initial codebook vector set to obtain the fourth feature;

[0036] The original image generation network is invoked to reconstruct the image based on the fourth feature, resulting in a second reconstructed image.

[0037] The original image generation network and the initial codebook vector set are trained based on the third feature, the fourth feature, the second reconstructed image, and the third training image to obtain the initial image generation network and the target codebook vector set.

[0038] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to traverse each feature vector in the initial codebook vector set and calculate the distance value between each feature vector and each feature vector in the third feature.

[0039] The feature vector with the smallest distance value is selected to replace the feature vector in the third feature to obtain the fourth feature.

[0040] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to perform similarity calculation based on the fourth feature and the third feature to obtain a first loss value;

[0041] Regularized loss is calculated based on the second reconstructed image and the third training image to obtain a second loss value and a third loss value, wherein the second loss value is the image pixel loss value and the third loss value is the image semantic loss value.

[0042] Based on the second reconstructed image and the third training image, a discrimination loss is calculated to obtain a fourth loss value;

[0043] The initial image generation network and the initial codebook vector set are trained based on the first loss value, the second loss value, the third loss value, and the fourth loss value to obtain the initial image generation network and the target codebook vector set.

[0044] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to call the initial index prediction network to perform index prediction on the first feature to obtain the feature index;

[0045] Based on this feature index, a query is performed in the target codebook vector set to obtain the second feature.

[0046] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to determine an index prediction loss based on the feature index, and the index prediction loss is used to train the initial classification network, the initial image generation network and the initial index prediction network.

[0047] The processing module is used to train the initial index prediction network, the initial classification network, and the initial image generation network based on the index prediction loss, the classification loss value, and the reconstruction loss value, so as to obtain the index prediction network, the target classification network, and the target image generation network.

[0048] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to call the initial image generation network to extract features from the first training image to obtain a fifth feature;

[0049] The initial index prediction network is invoked to predict the index of the fifth feature in order to obtain the label index of the first training image;

[0050] When the training image corresponding to the second feature has a label index, the index prediction loss is determined based on the feature index and the label index.

[0051] When the training image corresponding to the second feature does not have a label index, the prediction loss for that index is determined to be zero.

[0052] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to perform similarity calculation based on the first feature and the second feature to obtain a fifth loss value;

[0053] Based on the first reconstructed image and each training image in the training image set, regularization loss is calculated to obtain a sixth loss value and a seventh loss value, wherein the sixth loss value is the image pixel loss value and the seventh loss value is the image semantic loss value.

[0054] The discrimination loss is calculated based on the first reconstructed image and each training image in the training image set to obtain the eighth loss value;

[0055] The fifth, sixth, seventh, and eighth loss values ​​are weighted and summed to obtain the reconstruction loss value.

[0056] This application also provides a computer device, including: a memory, a processor, and a bus system;

[0057] The memory is used to store programs;

[0058] The processor is used to execute programs in memory, and the processor is used to execute the methods mentioned above according to the instructions in the program code;

[0059] Bus systems are used to connect memory and processor to enable communication between them.

[0060] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0061] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0062] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: Feature vectors are extracted from the image to be processed through a target image generation network, and these feature vectors are discretized based on the target codebook vector set to obtain discretized features; then, the discretized features are reconstructed based on the reconstruction technology of the target image generation network to obtain a reconstructed image; finally, liveness detection is achieved through the pixel and feature residuals between the reconstructed image and the image to be processed. Since the target codebook vector set has learned the distribution and feature differences of real biometric features through training, when generating the reconstructed image through the image generation network, the reconstructed features of the real image are similar to the input features, with a small residual; while the forged image, because it did not participate in training, has a large feature difference from the features in the target codebook vector set, resulting in a large residual. Thus, when using a classification network for classification and recognition, effective perception of unknown forged images can be achieved, thereby improving the accuracy of liveness data recognition. At the same time, the target codebook vector set only includes feature vectors from real images, so training can be performed using only real images, thus avoiding the drawback of relying on a large number of forged image samples for training. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of an application architecture for the data processing method in the embodiments of this application;

[0064] Figure 2 This is a schematic diagram of a model training process in an embodiment of this application;

[0065] Figure 3 This is a schematic diagram of the first stage of model training in an embodiment of this application;

[0066] Figure 4 This is a schematic diagram of the second stage of model training in an embodiment of this application;

[0067] Figure 5a This is a schematic diagram of the QRAM processing flow in an embodiment of this application;

[0068] Figure 5b This is a schematic diagram of the DRAM processing flow in an embodiment of this application;

[0069] Figure 6 This is a schematic diagram of one embodiment of the data processing method in this application;

[0070] Figure 7 This is a schematic diagram of one embodiment of the data processing device in this application.

[0071] Figure 8 This is a schematic diagram of one embodiment of the server in this application;

[0072] Figure 9This is a schematic diagram of one embodiment of the terminal in this application. Detailed Implementation

[0073] This application provides a data processing method, apparatus, device, medium, and program product for improving the accuracy of liveness data identification.

[0074] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0075] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0076] With the rapid development of artificial intelligence and computer vision technologies, liveness detection, as an important technology in the field of identity verification, has become a key research direction for preventing forgery attacks (such as photos, videos, masks, etc.). More and more liveness detection technologies are being applied to real-world products, such as smart facial recognition payment and access control services. Currently, most liveness detection technologies are based on deep learning classification models. However, the detection performance of deep learning-based classification models is largely determined by the quality and quantity of samples; the model can only make good judgments on samples that have been trained. For newly emerging negative samples, it is necessary to continuously accumulate and construct negative samples manually, and to continuously optimize and iterate the model through accumulated negative samples in order to be able to perceive new negative samples. The inability to perceive negative samples that have not been trained results in insufficient recognition accuracy of deep learning-based models. Therefore, there is an urgent need to provide a solution that can improve the accuracy of liveness detection.

[0077] To address this technical problem, this application provides the following technical solution: acquiring an image to be processed, which includes the biometric features of the object to be identified; invoking the encoder of a target image generation network to extract features from the image to be processed, wherein the target image generation network is a pre-trained network; discretizing the features to be processed based on a target codebook vector set to obtain discretized features, wherein the target codebook vector set is a feature space trained on real images and includes feature vectors from real images; invoking the decoder of the target image generation network to decode the discretized features to obtain a reconstructed image of the image to be processed; determining feature residuals based on the features to be processed and the discretized features, and determining pixel residuals based on the reconstructed image and the image to be processed; invoking a target classification network to perform recognition processing on the feature residuals and the pixel residuals to obtain the recognition result of the image to be processed, wherein the target classification network is a pre-trained network. Because this target codebook vector set learns the distribution and feature differences of real biometric features through training, when the image generation network generates reconstructed images, the reconstructed features of real images are similar to the input features, resulting in smaller residuals. In contrast, forged images, not having participated in training, have larger feature differences from the target codebook vector set, leading to larger residuals. This allows for effective perception of unknown forged images when using a classification network for identification, thereby improving the accuracy of identifying real images and liveness data. Furthermore, since the target codebook vector set only includes feature vectors from real images, training can be performed using only real images, avoiding the drawbacks of relying on a large number of negative data samples.

[0078] The model training methods in the various optional embodiments of this application can be implemented based on artificial intelligence (AI) technology. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making functions.

[0079] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0080] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the vision field, such as Swin-transformer, ViT, V-MOE, and MAE, can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0081] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence (AI) and the fundamental way to endow computers with intelligence; its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning. With the research and advancement of AI technology, it is being researched and applied in multiple fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with technological development, AI will be applied in even more fields and play an increasingly important role.

[0082] This application also relates to cloud technology. Cloud technology refers to a managed technology that unifies hardware, software, network, and other system resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing.

[0083] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology applied to the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of internet behavior, every item may possess its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing. The cloud technology involved in this application mainly refers to the transmission and storage of data such as images to be processed, training image sets, and models (including target image generation networks, target codebook vector sets, and target classification network models) between terminal devices or servers via the "cloud."

[0084] This application provides a data processing method, apparatus, device, medium, and program product to improve the accuracy of recognizing real images and liveness data. The following describes exemplary applications of the electronic devices provided in this application. These electronic devices can be implemented as various types of user terminals or as servers.

[0085] The electronic device improves the accuracy of liveness detection by running the data processing method provided in the embodiments of this application. Specifically, after deploying a trained liveness detection model (i.e., the model including the target image generation network, the target codebook vector set, and the target classification network), the electronic device improves its accuracy in recognizing palm prints or face images.

[0086] The above solution can be applied to many artificial intelligence fields, including image recognition, mobile payment, and security. When using a liveness detection model (trained using the model training method provided in this application embodiment) to help users perform palmprint recognition or face recognition, the liveness detection model can be implemented as an independent online application, installed on the user's computer device or backend server, making it convenient for users to use the liveness detection model for palmprint recognition or face recognition.

[0087] In one exemplary solution, within a mobile payment scenario, the user's palmprint features are first registered. The user captures an image of their palm using their mobile phone camera or a dedicated palmprint scanning device. It should be understood that during the palm image capture process, a real-time feedback mechanism can be used to guide the user to adjust the position, angle, and distance of their palm to ensure the quality of the captured image. For example, the system might prompt the user to place their palm in a designated area to ensure the palmprint is complete and clear; or it might prompt the user to capture the palmprint from multiple angles and under different lighting conditions to enhance the robustness of subsequent recognition. The palm image is then cropped to obtain the palmprint region. The target image generation network in the deployed liveness detection model is then used to extract features from this palmprint region, generating a corresponding feature vector; this feature vector represents the user's identity. Finally, this feature vector is encrypted and stored locally on the mobile device or on a cloud server to ensure user privacy is not compromised.

[0088] During mobile payment, the user holds their palm in front of the payment device (such as a mobile phone or POS machine), and the device captures a palmprint image in real time via a camera. The captured palm image is then processed by a Region of Interest (ROI) extraction module, which automatically crops the ROI. The target image generation network in the liveness detection model is then used to extract features from the extracted palmprint region to obtain a palmprint feature vector. This palmprint feature vector has the same dimension as the feature vector stored during the registration phase. Then, based on the target codebook vector set in the liveness detection model, the discretized features of the palmprint feature vector are obtained. The target image generation network of the liveness detection model is then used to reconstruct these discretized features to obtain a reconstructed image. Pixel residuals are determined based on the palmprint region and the reconstructed image, and feature residuals are determined based on the palmprint feature vector and the discretized features. Finally, the classification network of the liveness detection model is used to identify the pixel residuals and feature residuals to determine whether the palmprint region is a real palm. If it is a real image, the current user's palmprint feature vector is matched with the feature vector stored during the registration phase. Efficient similarity calculations (such as cosine similarity or Euclidean distance) are used to determine whether they belong to the same user. If the similarity exceeds a preset threshold, authentication is successful; otherwise, authentication fails. Once palmprint authentication is successful, the system triggers the payment process to complete the transaction. If the palmprint region is determined to be forged data or palmprint authentication fails, the system can prompt the user to retry or choose another payment method (such as password input).

[0089] In another exemplary solution, within a security context, the user's facial features are first registered. The user's facial image is captured on a registration device (such as a facial scanner or access control terminal). It should be understood that during the facial image capture process, a real-time feedback mechanism can be used to guide the user in adjusting their head position, angle, and distance to ensure image quality. For example, the system might prompt the user to place their face in a designated area to ensure a complete and clear image; or it might prompt the user to capture the face from multiple angles and under different lighting conditions to enhance the robustness of subsequent recognition. The facial image is then cropped to obtain the facial region. A target image generation network based on a deployed liveness detection model is then used to extract features from this facial region, generating a corresponding feature vector that represents the user's identity. Finally, this feature vector is encrypted and stored locally on the mobile device or in a cloud server to ensure user privacy is not compromised.

[0090] When a user needs to enter the access control area, the system completes identity verification through the following steps. The user places their face in front of the access control terminal's camera or face scanning device, and the device captures the face image in real time through the camera. The captured face image is then processed by the ROI extraction module to automatically crop out the region of interest (ROI) of the face. The target image generation network of the liveness detection model is then invoked to extract features from the extracted face region to obtain a face feature vector. This face feature vector has the same dimension as the feature vector stored during the registration phase. Then, based on the target codebook vector set in the liveness detection model, the discretized features of the facial feature vector are obtained. The target image generation network of the liveness detection model is then used to reconstruct these discretized features to obtain a reconstructed image. Pixel residuals are determined based on the facial region and the reconstructed image, and feature residuals are determined based on the facial feature vector and the discretized features. Finally, the classification network of the liveness detection model is used to identify the pixel residuals and feature residuals to determine whether the facial region is a real face. If it is a real image, the current user's facial feature vector is matched with the feature vectors stored during the registration phase. Efficient similarity calculations (such as cosine similarity or Euclidean distance) are used to determine whether they belong to the same user. If the similarity exceeds a preset threshold, authentication is successful; otherwise, authentication fails. After successful facial authentication, the system triggers the payment process to complete the transaction. If the facial region is determined to be a non-real face or facial authentication fails, the system can prompt the user to retry or choose another access method (such as password input).

[0091] It should be understood that, in this application, liveness data refers to the input data used for training, validating and testing liveness detection models, and mainly includes real data and fake data.

[0092] The "real data" refers to images or videos, including biometric features, collected from real objects in natural environments. For example, when the real data is real human facial data, it typically possesses natural facial textures, light reflection, depth information, and features consistent with the physical characteristics of a real human face. Its collection conditions are usually diverse, involving various devices (such as mobile phones and cameras) and environments (such as different lighting and backgrounds).

[0093] This forged data refers to fake data generated through various means to attack liveness detection systems. Its types typically include photo attacks, video attacks, 3D mask attacks, high-resolution attacks, and synthetic data attacks. Photo attacks use flat photographs as forgery samples (such as printed paper photos or photos on electronic screens); such data lacks natural lighting reflections and depth information. Video attacks use recorded videos played on electronic devices for forgery; such data may contain dynamic features, but texture and lighting remain unnatural. 3D mask attacks use 3D masks (such as silicone masks) to forge faces; such data has some depth information, but texture and lighting are abnormal. High-resolution attacks use forged data generated with high-quality equipment, potentially containing more realistic lighting and texture features. Synthetic data attacks use virtual face data generated using techniques such as generative adversarial networks; such data may visually resemble real faces, but exhibits anomalies in detail and feature distribution.

[0094] Of course, in addition to the above-mentioned scenarios, the liveness detection model provided in this application embodiment can also be applied to other scenarios that require image recognition. This application embodiment does not limit the specific application scenarios.

[0095] See Figure 1 , Figure 1This is an optional architecture diagram of the model training scheme and online application (i.e., data processing method) of the liveness detection model provided in this application embodiment. To support a data processing method and a model training scheme, the terminal device 100 connects to the server 300 via the network 200, and the server 300 connects to the database 400. The network 200 can be a wide area network (WAN), a local area network (LAN), or a combination of both. The client used to implement the model training scheme and data processing method is deployed on the terminal device 100. The client can run on the terminal device 100 through a browser or as a standalone application (APP). The specific form of the client is not limited here. The server 300 involved in this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Terminal device 100 can be a smartphone, tablet, laptop, PDA, personal computer, smart TV, smartwatch, in-vehicle device, wearable device, smart voice interaction device, smart home appliance, aircraft, etc., but is not limited to these. Terminal device 100 and server 300 can be directly or indirectly connected via network 200 through wired or wireless communication, which is not limited in this application. The number of server 300 and terminal device 100 is also not limited. The solution provided in this application can be completed independently by terminal device 100, independently by server 300, or jointly by terminal device 100 and server 300, which is not specifically limited in this application. In short, database 400 can be regarded as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data in the files. The so-called "database" is a collection of data stored together in a certain way, which can be shared by multiple users, has the lowest possible redundancy, and is independent of the application. A Database Management System (DBMS) is a computer software system designed to manage databases, and it generally has basic functions such as storage, retrieval, security, and backup.Database management systems can be categorized based on the database model they support, such as relational or Extensible Markup Language (XML); or based on the type of computer they support, such as server clusters or mobile phones; or based on the query language used, such as Structured Query Language (SQL) or XQuery; or based on performance priorities, such as maximum scale or highest operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages ​​simultaneously. In this application, database 400 can be used to store data such as training image sets and initial liveness detection models. Of course, the storage location for storing training image sets and initial liveness detection models is not limited to a database; for example, it can also be stored in terminal device 100, a blockchain, or the distributed file system of server 300.

[0096] In some embodiments, both the server 300 and the terminal device 100 can execute the data processing method and model training method provided in the embodiments of this application.

[0097] Based on the above introduction, such as Figure 2 The illustrated model training flowchart explains the training process of the liveness detection model in this application. In this application, the liveness detection model includes a target image generation network, a target codebook vector set, an index prediction network, a feature fusion network, and a target classification network. The target image generation network can be called a Vector Quantized Generative Adversarial Network (VQGAN). It can employ any encoder and decoder architecture, such as a U-Shaped Network (Unet), a Denoising Diffusion Implicit Models Transformer (Dit), or a Vision Transformer (VIT). The target classification network can be any architecture, such as VIT, a Convolutional Neural Network (CNN), or a Transformer.

[0098] like Figure 2As shown, the training process of the liveness detection model can be divided into two stages. The first stage is used to train the target codebook vector set, while simultaneously performing part of the image generation network training. The second stage is used to train the target image generation network and the target classification network. It should be understood that when the liveness detection model introduces an index prediction network, this index prediction network also needs to be trained simultaneously during the second stage of the training process.

[0099] The following section uses the server as the execution entity to explain the two training phases of this liveness detection model.

[0100] In the first training phase, the process can be as follows: Figure 3 As shown.

[0101] 301. The server obtains the third training image, the original image generation network, and the initial codebook vector set, wherein the third label of the third training image is the real image.

[0102] In this embodiment, the third training image can be publicly available open-source data or authorized data collected by a third party; no specific limitation is made here.

[0103] It should be understood that, in order to improve the data processing efficiency during model training, the third training image can be set as a preprocessed normalized training image.

[0104] In one exemplary scheme, the image preprocessing scheme may be as follows: performing biometric detection and cropping on the acquired real image to obtain a cropped image; adjusting the cropped image to a preset size; and then normalizing the pixel values ​​in the image to a preset range to obtain the first training image.

[0105] The original image generation network can be the network structure illustrated above, and will not be elaborated further here.

[0106] The initial codebook vector set (also known as the codebook) is a core component of vector quantization. It is a collection of a fixed number of vectors called "codebook vectors" or "codewords." Essentially, it is a predefined vector dictionary containing K discrete representation vectors, where the feature dimension D of each representation vector can be determined based on the specific situation. The purpose of this initial codebook vector set is to map continuous high-dimensional input vectors to a finite set of discrete vectors. In practical applications, its workflow can be as follows: for the original input vector, find the vector in the codebook vector set that is most similar to it (usually with the smallest Euclidean distance); then replace the original input vector with the found codebook vector. To reduce information storage, the codebook vector set typically only stores the index of the codebook vector, so the codebook vector in the codebook vector set can also be found using the index.

[0107] Based on the above description of the codebook vector set, it is clear that the settings of K and D in this initial codebook vector set directly affect the performance, reconstruction quality, memory usage, and computational complexity of the liveness detection model. Therefore, the settings of K and D need to be adjusted according to the actual situation.

[0108] Here, K represents the number of discrete feature vectors in the initial codebook vector set, where each discrete feature vector represents a specific feature pattern. Therefore, the size of K determines the granularity of the feature representation. A smaller K indicates fewer discrete feature types and limited feature representation capability (which may lead to overly coarse reconstruction); a larger K indicates more discrete feature types and enhanced feature representation capability, but it increases the computational and storage costs of the model. Therefore, the value of K can be set according to task complexity, input image resolution, model storage limitations, and empirical values ​​(e.g., 256 or 512).

[0109] D represents the dimension of each discrete feature vector, which determines the representational power of each feature vector. A smaller D indicates that the representational power of the discrete feature vector is limited, which may result in the features being insufficient to express highly complex image content. A larger D indicates that the representational power of the discrete feature vector is enhanced, but it increases the number of model parameters and computational complexity. Therefore, the value of D can be set according to the complexity of the input image content, the resolution of the input image, hardware resource limitations, and empirical values ​​(such as 64 or 128).

[0110] Based on the above description, in an exemplary scheme, assuming the liveness detection model is a simple task (low-resolution face image) with an input image resolution of 128×128, and a task complexity of medium (low detail), the recommended parameters for the initial codebook vector set can be set to K=512, D=128, and storage size=512×128=65536.

[0111] 302. The server calls the original image generation network to extract features from the third training image to obtain the third feature.

[0112] During training, the server inputs the third training image into the encoder of the original image generation network to obtain continuous features (i.e., the third feature). In an exemplary scheme, the third feature can be represented as Z. i =encoder(I). Where Z... i The shape can be represented as [H, W, D], where H and W represent the spatial dimensions of the third feature and D represents the depth of the third feature.

[0113] 303. The server discretizes the third feature based on the initial codebook vector set to obtain the fourth feature.

[0114] After obtaining the third feature, the server maps the third feature to the initial codebook vector set to obtain the fourth feature, which is a discretized feature.

[0115] In this embodiment, when mapping the third feature to the initial codebook vector set, the following method can be used: the server calculates the distance value between the third feature and each codebook vector in the initial codebook vector set; then, the codebook vector with the smallest distance value is selected as the fourth feature. The distance value can be Euclidean distance, cosine distance, etc., and is not specifically limited here, as long as it can represent the similarity between two feature vectors.

[0116] Based on the above description, this mapping process can be represented as Z q =argmin‖Z i -Z k ‖. Among them, the Z i Used to indicate the third feature, the Z k Z is used to indicate the codebook vectors in the initial codebook vector set. q Used to indicate this fourth feature.

[0117] 304. The server calls the original image generation network to reconstruct the image of the fourth feature to obtain the second reconstructed image.

[0118] After obtaining the fourth feature, the server inputs the fourth feature into the decoder of the original image generation network. The decoder performs reconstruction processing on the fourth feature to obtain the second reconstructed image.

[0119] In this embodiment, the decoder typically decodes discrete features into a reconstructed image using deconvolution or self-attention mechanisms. The specific process will not be detailed here.

[0120] 305. The server trains the original image generation network and the initial codebook vector set based on the third feature, the fourth feature, the second reconstructed image, and the third training image to obtain the initial image generation network and the target codebook vector set.

[0121] In this embodiment, after the server obtains the reconstructed feature (i.e., the fourth feature) and the reconstructed image (i.e., the second reconstructed image), the server can train the initial codebook vector set and the original image generation network according to the designed loss function.

[0122] The loss function can be designed as follows: the server calculates the similarity between the fourth feature and the third feature to obtain a first loss value; it calculates a regularization loss between the second reconstructed image and the third training image to obtain a second loss value and a third loss value, wherein the second loss value is the image pixel loss value and the third loss value is the image semantic loss value; it calculates a discriminative loss between the first reconstructed image and the third training image to obtain a fourth loss value; and it trains the original image generation network and the initial codebook vector set based on the first loss value, the second loss value, the third loss value, and the fourth loss value to obtain the initial image generation network and the target codebook vector set.

[0123] Specifically, this first loss value can be understood as vector quantization loss, which is used to optimize the consistency between the third feature of the encoder output and the discretized feature (fourth feature) after discretization. Its loss function can be expressed as:

[0124] L vq =‖sg(Z i )-Z q ‖+β‖Z i -sg(Z q )‖

[0125] Among them, the ||sg(Z) i )-Z q The "|" symbol is used to constrain the consistency between the encoder's output features and reconstructed features; the "|Z" symbol... i -sg(Z qThe β is used to constrain the consistency between the reconstructed features and the output features of the encoder; the β is a constant used to control the importance of the corresponding loss; where sg represents the stop gradient propagation operation.

[0126] This second loss value can be understood as image reconstruction loss (i.e., image pixel loss), which is used to ensure pixel similarity between the reconstructed image and the original image. Its loss function can be expressed as:

[0127]

[0128] Wherein, I is used to represent the original image (i.e., the third training image), and Used to represent the reconstructed image (i.e., the second reconstructed image).

[0129] This third loss value can be understood as image perceptual loss (i.e., image semantic loss), which is used to ensure the high-level semantic similarity between the reconstructed image and the original image. Its loss function can be expressed as:

[0130]

[0131] Wherein, I is used to represent the original image (i.e., the third training image), and Used to represent the reconstructed image (i.e., the second reconstructed image).

[0132] This fourth loss value can be understood as adversarial loss, used to employ a discriminator to distinguish between the original image and the reconstructed image, thereby improving the generation quality of the image generation network through adversarial training. Its loss function can be expressed as:

[0133]

[0134] Wherein, I is used to represent the original image (i.e., the third training image), and Used to represent the reconstructed image (i.e., the second reconstructed image).

[0135] In summary, in this first training phase, the overall training loss can be expressed as:

[0136] L code =L1+L per +γ adv L adv +L vq

[0137] Based on the aforementioned training loss, the original image generation network and the initial codebook vector set are trained to construct a target codebook vector set containing only real biological features and an image generation network that has been initially trained (i.e., the initial image generation network).

[0138] In the second training phase, the process can be as follows: Figure 4 As shown.

[0139] 401. The server obtains a training image set, an initial classification network, a target codebook vector set, and an initial image generation network, wherein the training image set includes a first training image and a second training image, wherein the first label of the first training image is a real image, and the second label of the second training image is a fake image.

[0140] In this embodiment, the training image set can be publicly available open-source data or authorized data collected by a third party; no specific limitation is made here.

[0141] It should be understood that during the second phase of training, in order to train a better classification network, the training image set should include both positive samples (i.e., the first training image labeled as a real image) and negative samples (i.e., the second training image labeled as a fake image). The fake image can be understood as forged data in the liveness detection dataset. The first training image can be the same as the third training image from the first training phase, or it can be a training image different from the third training image; this is not limited here. Similarly, the second training image can be a fake image of the first training image, or it can be a fake image unrelated to the first training image; this is not limited here either.

[0142] It should be understood that, in order to improve the data processing efficiency during model training, each training image in this training image set is a preprocessed, normalized training image.

[0143] In one exemplary scheme, the image preprocessing scheme may be as follows: performing biometric detection and cropping on the acquired real image to obtain a cropped image; adjusting the cropped image to a preset size; and then normalizing the pixel values ​​in the image to a preset range to obtain each training image in the training image set.

[0144] During this stage, the target codebook vector set is a pre-trained codebook vector set, and therefore no longer participates in data updates.

[0145] 402. The server calls the initial image generation network to extract features from the training images in the training image set to obtain the first feature.

[0146] During training, the server inputs the first training image into the encoder of the initial image generation network to obtain continuous features (i.e., the first feature). In an exemplary scheme, the first feature can be represented as Z. j =encoder(I). Where Z... jThe shape can be represented as [H, W, D], where H and W represent the spatial dimensions of the first feature and D represents the depth of the first feature.

[0147] 403. The server discretizes the first feature based on the target codebook vector set to obtain the second feature.

[0148] After obtaining the first feature, the server maps the first feature to the target codebook vector set to obtain the second feature, which is a discretized feature.

[0149] In this embodiment, when mapping the first feature to the target codebook vector set, the following method can be used: the server calculates the distance between the first feature and each codebook vector in the target codebook vector set; then, the codebook vector with the smallest distance value is selected as the second feature. The distance value can be Euclidean distance, cosine distance, etc., and is not specifically limited here, as long as it can represent the similarity between two feature vectors.

[0150] Based on the above description, this mapping process can be represented as Z p =argmin‖Z j -Z k ‖. Among them, the Z j Used to indicate the first feature, the Z k Z is used to indicate the codebook vectors in the target codebook vector set. p Used to indicate this second feature.

[0151] In practical applications, since the feature distribution in the target codebook vector set has changed during the first training phase, using minimum distance to find similar features is no longer effective. Therefore, to achieve more accurate feature reconstruction, this embodiment can also introduce an index prediction network and use this network to search for codebook vectors. The specific process is as follows: the server calls the initial index prediction network to predict the index of the first feature to obtain the feature index; based on the feature index, a query is performed in the target codebook vector set to obtain the second feature.

[0152] For example, suppose a trained codebook contains 5 discrete feature vectors, numbered by indices [0, 1, 2, 3, 4], and each feature vector has a dimension of 3. In an exemplary scheme, this codebook can be represented as follows:

[0153] Codebook = {

[0154] 0: [0.1, 0.2, 0.3],

[0155] 1:[0.4, 0.5, 0.6],

[0156] 2: [0.7, 0.8, 0.9],

[0157] 3: [0.2, 0.3, 0.4],

[0158] 4: [0.6, 0.7, 0.8]

[0159] }

[0160] The input image is encoded to generate feature 1. Assuming feature 1 has a feature dimension of 2, and the feature vectors for each dimension are [0.35, 0.45, 0.55] and [0.65, 0.75, 0.85], feature 1 is then input into the index prediction network. Assuming this index prediction network is a multi-layer Transformer architecture, it can capture the global relationships of features through multiple Transformers and predict the optimal index for each feature dimension. Assuming the output of the index prediction network predicts the index [1, 4], then the first feature vector of feature 1 is predicted to be closest to index 1 in the codebook, and the second feature vector is predicted to be closest to index 4 in the codebook. Then, based on the predicted index [1, 4], the corresponding codebook vector is queried from the codebook, and the result can be represented as the reconstructed features of feature 1 being [0.4, 0.5, 0.6] and [0.6, 0.7, 0.8].

[0161] It should be understood that after introducing the index prediction network, the second training phase can also be divided into two parts. The first part is used to add label indices to the first training image. Specifically, the process can be as follows: the first training image undergoes feature discretization processing in both the initial image generation network and the initial index prediction network to obtain the label index of the first training image; this label index is then used to calculate the index prediction loss in subsequent training processes. The second part is used to train the overall liveness detection model based on the first and second training images.

[0162] 404. The server calls the initial image generation network to reconstruct the image based on the second feature to obtain the first reconstructed image.

[0163] After obtaining the second feature, the server inputs the second feature into the decoder of the initial image generation network, and the decoder performs reconstruction processing on the second feature to obtain the first reconstructed image.

[0164] In this embodiment, the decoder typically decodes discrete features into a reconstructed image using deconvolution or self-attention mechanisms. The specific process will not be detailed here.

[0165] 405. The server obtains training pixel residuals and training feature residuals based on the first reconstructed image, the first training image, and the second training image.

[0166] After the server obtains the first feature, the second feature, the first reconstructed image, the second training image, and the third training image, it determines the training feature residual based on the first feature and the second feature, and determines the training pixel residual based on the first reconstructed image and the first training image or the second training image.

[0167] For example, the second feature is Z p The first feature is Z j The first reconstructed image is I′, and the original input image is I. input Then the residual of the training feature can be expressed as Z. res =||Z j -Z p || 2 The training pixel residual can be expressed as I. res =‖I′-I input || 2 .

[0168] Since each feature vector in the target codebook vector set is constructed based on training on real images, when the original image is a real image, the feature residual between the continuous feature and the discrete feature is small, and the image residual between the reconstructed image and the original image is small. When the original image is a fake image, the feature residual between the continuous feature and the discrete feature is large, and the image residual between the reconstructed image and the original image is large.

[0169] 406. The server calls the initial classification network to classify the training pixel residual and the training feature residual to obtain the training classification result.

[0170] The server inputs the training pixel residual and the training feature residual into the initial classification network, so that the initial classification network performs classification processing based on the training pixel residual and the training feature residual to obtain the training classification result.

[0171] Understandably, in order to further improve the recognition ability of the classification network, this embodiment also provides feature fusion of training pixel residuals and training feature residuals to obtain fused residuals; finally, the fused residuals are input into the initial classification network, so that the initial classification network performs classification processing on the fused residuals to obtain the training classification result.

[0172] Specifically, the server may employ several possible implementation methods for feature fusion of the training pixel residual and the training feature residual.

[0173] In one exemplary scheme, the server can introduce an attention mechanism to fuse the training pixel residual and the training feature residual to obtain the fused residual. In another exemplary scheme, the server can introduce a Query Residual Attention Module (QRAM) (i.e., the first network structure, which includes a first convolutional layer and an attention computation layer) and a Dual Residual Attention Module (DRAM) (i.e., the second network structure, which includes a sampling network, a second convolutional layer, a global pooling layer, and an attention computation layer), and fuse the training pixel residual and the training feature residual based on these two modules. The process can be as follows: Figure 5a and Figure 5b As shown.

[0174] Based on this Figure 5a The illustrated process (i.e., the processing flow of the QRAM) involves the server extracting features from the training feature residuals using a first convolutional layer to obtain second intermediate features; then, based on an attention mechanism, feature learning is performed on the second intermediate features and the feature to be processed to obtain first intermediate features. In an exemplary scheme, the training feature residuals can represent Z. res This continuous feature is represented by Z. j In the QRAM processing flow, when processing the training feature residuals and continuous features based on this attention mechanism, it can be represented as follows:

[0175] Q = Conv(Z) j ),K=Conv(Z res V = Conv(Z) res )

[0176]

[0177] Here, Conv is used to indicate the convolution operation, and the first intermediate feature obtained by the above attention mechanism can be represented as follows:

[0178] Z g =Conv(A(Q,K,V))+Z j

[0179] Among them, the Z g Used to indicate the first intermediate feature, the Z j Continuous features used to indicate the encoder output of the image generation network.

[0180] After obtaining the first intermediate feature of the training feature residual through the QRAM processing flow, the first intermediate feature is then used as... Figure 5bThis is an input to the DRAM processing flow shown. The processing flow can be as follows: The first intermediate feature is upsampled through a sampling network to obtain a third intermediate feature; the training pixel residual is extracted using a second convolutional layer and a global pooling layer to obtain a fourth intermediate feature; the third and fourth intermediate features are concatenated to obtain a fifth intermediate feature; attention is calculated on the fifth intermediate feature in both the channel dimension and the spatial position dimension to obtain positional attention features and channel attention features; the positional attention features and the channel attention features are multiplied and added together to obtain the fused residual.

[0181] In one exemplary scheme, suppose the training pixel residual is represented as I res In this DRAM processing flow, the location attention feature and the channel attention feature can be represented as follows:

[0182]

[0183] Here, Ac is used to indicate the location attention feature, and Ap is used to indicate the channel attention feature.

[0184] Then, during feature fusion, the calculation process can be as follows:

[0185] Zc = f R (Z′ ca *Ac+Z′ ca )

[0186] Zp = f R (Ap*Z′ ca +Z′ ca )

[0187] Zfuse = Zc + Zp

[0188] Where Zc represents the position feature; Zp represents the channel feature; and Zfuse represents the fusion of the position feature and the channel feature, which is represented as the fusion residual.

[0189] In another exemplary approach, the server can directly concatenate the training pixel residual and the training feature residual to obtain the fused residual. This involves concatenating the training pixel residual and the training feature residual along a specific dimension to form a joint feature vector. The resulting concatenated feature vector retains the independent information from both residuals.

[0190] In another exemplary scheme, the server can perform a weighted concatenation of the training pixel residual and the training feature residual to obtain the fused residual. This involves assigning weights to the training pixel residual and the training feature residual respectively, and then concatenating them along a specific dimension to form a weighted joint feature vector. The concatenated feature retains the independent information of the two residuals. The weights can be fixed values ​​or dynamically set.

[0191] In another exemplary scheme, the server can introduce a self-attention mechanism to adaptively allocate weights based on the features of the input, thereby dynamically fusing the training pixel residuals and the training feature residuals.

[0192] 407. The server calculates the classification loss value based on the training classification result, the first label, and the second label.

[0193] In this embodiment, after obtaining the training classification result, cross-entropy loss is calculated based on the training classification result and the labels (i.e., real images or fake images) of each training image in the training image set to obtain the classification loss value.

[0194] In one exemplary scheme, the formula for calculating the classification loss value can be expressed as follows:

[0195]

[0196] Among them, the L cls Used to indicate the classification loss value, y i The p is used to indicate the label of each training image in the training image set. i Used to indicate the predicted classification result of each training image in the training image set.

[0197] 408. The server calculates the reconstruction loss value based on the first feature, the second feature, the first reconstructed image, and the training images in the training image set.

[0198] In this embodiment, after the server obtains the reconstructed feature (i.e., the second feature) and the reconstructed image (i.e., the first reconstructed image), the server can calculate the reconstruction loss according to the designed loss function.

[0199] The loss function can be designed as follows: similarity is calculated based on the first feature and the second feature to obtain a fifth loss value; regularization loss is calculated based on the first reconstructed image and each training image in the training image set to obtain a sixth loss value and a seventh loss value, wherein the sixth loss value is the image pixel loss value and the seventh loss value is the image semantic loss value; discriminative loss is calculated based on the second reconstructed image and each training image in the training image set to obtain an eighth loss value; and the fifth loss value, the sixth loss value, the seventh loss value and the eighth loss value are weighted and summed to obtain the reconstruction loss value.

[0200] Specifically, this fifth loss value can be understood as vector quantization loss, which is used to optimize the consistency between the first feature of the encoder output and the discretized feature (second feature) after discretization. Its loss function can be expressed as:

[0201] L vq =‖sg(Z j )-Z p ‖+β‖Z j -sg(Z p )‖

[0202] Among them, the ||sg(Z) j )-Z p The "|" symbol is used to constrain the consistency between the encoder's output features and reconstructed features; the "|Z" symbol... j -sg(Z p The β is used to constrain the consistency between the reconstructed features and the output features of the encoder; the β is a constant used to control the importance of the corresponding loss; where sg represents the stop gradient propagation operation.

[0203] This sixth loss value can be understood as image reconstruction loss (i.e., image pixel loss), which is used to ensure pixel similarity between the reconstructed image and the original image. Its loss function can be expressed as:

[0204] L1 = ||I input -I′‖

[0205] Wherein, I is used to represent the original image (i.e., the first training image), Used to represent the reconstructed image (i.e., the first reconstructed image).

[0206] This seventh loss value can be understood as image perceptual loss (i.e., image semantic loss), which is used to ensure the high-level semantic similarity between the reconstructed image and the original image. Its loss function can be expressed as:

[0207]

[0208] Among them, the I inputThe I′ is used to represent the original image (i.e., the first training image), and the I′ is used to represent the reconstructed image (i.e., the first reconstructed image).

[0209] This eighth loss value can be understood as adversarial loss, used to employ a discriminator to distinguish between the original image and the reconstructed image, thereby improving the generation quality of the image generation network through adversarial training. Its loss function can be expressed as:

[0210] L adv =[log(D(I input ))+log(1-D(I′))]

[0211] Among them, the I input The I′ is used to represent the original image (i.e., the first training image), and the I′ is used to represent the reconstructed image (i.e., the first reconstructed image).

[0212] In summary, in this second training phase, the reconstruction loss can be expressed as:

[0213] L 重构 =L1+L per +γ adv L adv +L vq

[0214] 409. The server trains the initial classification network and the initial image generation network based on the classification loss value and the reconstruction loss value to obtain the target classification network and the target image generation network.

[0215] In this embodiment, after the server obtains the classification loss value and the reconstruction loss value, the classification loss value and the reconstruction loss value are weighted and summed to obtain the overall loss value of the liveness detection model. Finally, the initial classification network and the initial image generation network are trained based on the overall loss value. After reaching the convergence condition, the target classification network and the target image generation network are output. It should be understood that the liveness detection model should include the target image generation network, the target codebook vector set, and the target classification network.

[0216] It should be understood that if an index prediction network is introduced when discretizing features, then the index prediction loss should also be introduced during training, and the liveness detection model should be trained by combining the index prediction loss, the classification loss value, and the reconstruction loss value.

[0217] The prediction loss for this index is explained below.

[0218] After introducing the index prediction network, in the first part of the second training phase, a label index is generated for the first training image; then in the second part of the second training phase, the server calls the initial index prediction network to perform index prediction on the first feature to obtain the feature index; when the training image corresponding to the second feature has a label index, the index prediction loss is determined based on the feature index and the label index; when the training image corresponding to the second feature does not have a label index, the index prediction loss is determined to be zero.

[0219] It should be understood that when determining the index prediction loss based on the feature index and the label index, it can be either cross-entropy loss or feature consistency loss; the specific choice is not limited here.

[0220] Based on the above introduction, the data processing method in this application will be described below, taking the server as the execution entity. Please refer to [link / reference needed]. Figure 6 One embodiment of the data processing method in this application includes:

[0221] 601. Obtain the image to be processed, which includes the biometric features of the object to be identified.

[0222] The server acquires image data from front-end image acquisition devices, such as data captured in real-time by a camera or data uploaded by the user. It should be understood that the image to be processed should include the biometric features of the object to be identified, such as facial data or palm print data.

[0223] Preferably, the server can also preprocess the image data to obtain the image to be processed. In one exemplary scheme, the preprocessing scheme can be as described above. Figure 3 The process described is not elaborated here. For example, if the image data is a 256×256 resolution face image, after image preprocessing, the output image to be processed can be an image that includes the face region and has a resolution of 224×224.

[0224] 602. Call the encoder of the target image generation network to extract features from the image to be processed to obtain the features to be processed. The target image generation network is a pre-trained network.

[0225] After obtaining the image to be processed, the server calls the encoder of the target image generation network to extract features from the image to obtain the features to be processed (also known as continuous features).

[0226] 603. Discretize the feature to be processed based on the target codebook vector set to obtain the discretized feature. The target codebook vector set is a feature space trained on real images and includes feature vectors of real images.

[0227] In this embodiment, the server performs this step in the same way as described above. Figure 3 or Figure 4 The scheme shown is the same, and will not be described in detail here.

[0228] 604. Call the decoder of the target image generation network to decode the discretized features to obtain the reconstructed image of the image to be processed.

[0229] In this embodiment, the server performs this step in the same way as described above. Figure 3 or Figure 4 The scheme shown is the same, and will not be described in detail here.

[0230] 605. Determine the feature residuals based on the feature to be processed and the discretized feature, and determine the pixel residuals based on the reconstructed image and the image to be processed.

[0231] In this embodiment, the server performs this step in the same way as described above. Figure 3 or Figure 4 The scheme shown is the same, and will not be described in detail here.

[0232] 606. Call the target classification network to identify the feature residual and the pixel residual to obtain the recognition result of the image to be processed. The target classification network is a pre-trained network.

[0233] In this embodiment, the server performs this step in the same way as described above. Figure 3 or Figure 4 The scheme shown is the same, and will not be described in detail here.

[0234] The recognition result is used to indicate whether the image to be processed is a real image or a fake image. Based on the recognition result, subsequent payment verification or access control verification operations are then performed. For example, it verifies whether the face indicated by the image to be processed matches the registered face. If they match, subsequent facial recognition payment operations can be performed.

[0235] The data processing apparatus in this application is described in detail below. Please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of one embodiment of the data processing apparatus in this application. The data processing apparatus 20 includes:

[0236] The acquisition module 201 is used to acquire an image to be processed, which includes the biometric features of the object to be identified.

[0237] The processing module 202 is used to call the encoder of the target image generation network to extract features from the image to be processed, so as to obtain the features to be processed. The target image generation network is a pre-trained network. The features to be processed are discretized based on the target codebook vector set to obtain discretized features. The target codebook vector set is a feature space trained on real images and includes feature vectors of real images. The decoder of the target image generation network is called to decode the discretized features to obtain a reconstructed image of the image to be processed. The feature residuals are determined based on the features to be processed and the discretized features, and the pixel residuals are determined based on the reconstructed image and the image to be processed. The target classification network is called to perform recognition processing on the feature residuals and the pixel residuals to obtain the recognition result of the image to be processed. The target classification network is a pre-trained network.

[0238] This application provides a data processing apparatus. Using this apparatus, since the target codebook vector set has learned the distribution and feature differences of real biometric features through training, when generating reconstructed images using the image generation network, the reconstructed features of real images are similar to the input features, resulting in a small residual. In contrast, forged images, not having participated in training, have a large feature difference from the target codebook vector set, resulting in a large residual. Thus, when using a classification network for classification and recognition, effective perception of unknown forged images can be achieved, thereby improving the accuracy of liveness detection. Furthermore, since the target codebook vector set only includes feature vectors from real images, training can be performed using only real images, avoiding the drawbacks of relying on a large number of forged image samples for training.

[0239] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0240] The processing module 202 is used to perform a fusion process on the feature residual and the pixel residual to obtain a fused residual;

[0241] The target classification network is invoked to perform recognition processing on the fusion residual in order to obtain the recognition result of the image to be processed.

[0242] In this embodiment, a data processing apparatus is provided. Using this apparatus, pixel residuals and feature residuals are fused, combining information from local pixel anomalies and high-dimensional feature distribution anomalies, thereby enhancing the detection capability of liveness data.

[0243] Optionally, in the above Figure 7Based on the corresponding embodiment, in another embodiment of the data processing device 20 provided in this application, the processing module 202 is used to perform fusion processing on the feature residual and the pixel residual based on an attention mechanism to obtain the fused residual.

[0244] In this embodiment, a data processing apparatus is provided. Using this apparatus, an attention mechanism is introduced to dynamically weight the feature residuals, highlighting key information relevant to liveness data and suppressing irrelevant or redundant features. Simultaneously, pixel residuals and feature residuals are concatenated using channel attention features and position attention features, thereby effectively acquiring local detail features and global distribution features, thus enhancing the detection capability of liveness data.

[0245] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing device 20 provided in this application, the processing module 202 is used to perform feature learning on the feature residual based on the first network structure and the attention mechanism to obtain the first intermediate feature;

[0246] Based on the second network structure and the attention mechanism, the first intermediate feature and the pixel residual are fused to obtain the fused residual. In this embodiment, a data processing apparatus is provided. Using the above apparatus, an attention mechanism is introduced to dynamically weight the feature residual, highlighting key information related to liveness data and suppressing irrelevant or redundant features. Simultaneously, the pixel residual and the feature residual are concatenated on channel attention features and position attention features, thereby effectively acquiring local detail features and global distribution features, thus enhancing the detection capability of liveness data.

[0247] Optionally, in the above Figure 7 Based on the corresponding embodiment, in another embodiment of the data processing device 20 provided in this application, the processing module 202 is used to extract features from the feature residual through the first convolutional layer to obtain the second intermediate feature;

[0248] Based on the attention mechanism, feature learning is performed on the second intermediate feature and the feature to be processed to obtain the first intermediate feature.

[0249] In this embodiment, a data processing apparatus is provided. Using this apparatus, an attention mechanism is introduced to dynamically weight the feature residuals, highlighting key information relevant to liveness data and suppressing irrelevant or redundant features. Simultaneously, pixel residuals and feature residuals are concatenated using channel attention features and position attention features, thereby effectively acquiring local detail features and global distribution features, thus enhancing the detection capability of liveness data.

[0250] Optionally, in the above Figure 7 Based on the corresponding embodiment, in another embodiment of the data processing device 20 provided in this application, the processing module 202 is used to upsample the first intermediate feature through a sampling network to obtain a third intermediate feature, wherein the sampling network is included in the second network structure;

[0251] The pixel residual is used to extract features through the second convolutional layer and the global pooling layer to obtain the fourth intermediate feature. The second convolutional layer is included in the second network structure.

[0252] The third and fourth intermediate features are concatenated to obtain the fifth intermediate feature;

[0253] Attention is calculated on the fifth intermediate feature in both the channel dimension and the spatial location dimension to obtain the location attention feature and the channel attention feature.

[0254] The positional attention feature and the channel attention feature are multiplied and added together to obtain the fusion residual.

[0255] In this embodiment, a data processing apparatus is provided. Using this apparatus, an attention mechanism is introduced to dynamically weight the feature residuals, highlighting key information relevant to liveness data and suppressing irrelevant or redundant features. Simultaneously, pixel residuals and feature residuals are concatenated using channel attention features and position attention features, thereby effectively acquiring local detail features and global distribution features, thus enhancing the detection capability of liveness data.

[0256] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0257] The processing module 202 is used to call the index prediction network to predict the index of the feature to be processed in order to obtain the first feature index. The index prediction network is a pre-trained network.

[0258] Based on the first feature index, a query is performed in the target codebook vector set to obtain the discretized feature.

[0259] In this embodiment, a data processing apparatus is provided. Using this apparatus, an index prediction network is introduced to predict feature indexes, thereby optimizing the process of querying the most similar features from the target codebook vector set, improving the accuracy of feature reconstruction, and thus enhancing the detection capability of liveness data.

[0260] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0261] The acquisition module 201 is used to acquire a training image set, an initial image generation network, an initial classification network, and the target codebook vector set. The training image set includes a first training image and a second training image, wherein the first label of the first training image is a real image, and the second label of the second training image is a fake image.

[0262] The processing module 202 is used to call the first image generation network to extract features from the training images in the training image set to obtain the first feature;

[0263] The first feature is discretized based on the target codebook vector set to obtain the second feature;

[0264] The initial image generation network is invoked to reconstruct the image based on the second feature to obtain the first reconstructed image;

[0265] Based on the first reconstructed image, the first training image, and the second training image, the training pixel residual and the training feature residual are obtained;

[0266] The initial classification network is invoked to classify the training pixel residual and the training feature residual to obtain the training classification result;

[0267] Calculate the classification loss value based on the training classification result, the first label, and the second label;

[0268] The reconstruction loss value is calculated based on the first feature, the second feature, the first reconstructed image, and the training images in the training image set;

[0269] The initial classification network and the initial image generation network are trained based on the classification loss value and the reconstruction loss value to obtain the target classification network and the target image generation network.

[0270] In this embodiment, a data processing apparatus is provided. Using this apparatus, during training, each feature vector in a codebook vector set containing only real biometric features is discretized to obtain discretized features. Then, the discretized features are used for reconstruction to obtain a reconstructed image. Liveness detection is achieved by comparing the pixel and feature residuals between the reconstructed image and the training image, thus enabling effective perception of unknown forged images and improving the accuracy of liveness detection.

[0271] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing device 20 provided in this application, the acquisition module 201 is used to acquire a third training image, an original image generation network, and an initial codebook vector set, wherein the third label of the third training image is a real image;

[0272] The processing module 202 is used to call the original image generation network to extract features from the third training image to obtain the third feature;

[0273] The third feature is discretized based on the initial codebook vector set to obtain the fourth feature;

[0274] The original image generation network is invoked to reconstruct the image based on the fourth feature, resulting in a second reconstructed image.

[0275] The original image generation network and the initial codebook vector set are trained based on the third feature, the fourth feature, the second reconstructed image, and the third training image to obtain the initial image generation network and the target codebook vector set.

[0276] In this application embodiment, a data processing apparatus is provided. Using this apparatus, real images are used to train the codebook vectors, meaning that training can be performed using only real images, thus avoiding the drawbacks of relying on a large number of fake samples for training.

[0277] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing device 20 provided in this application, the processing module 202 is used to traverse each feature vector in the initial codebook vector set and calculate the distance value between each feature vector and each feature vector in the third feature.

[0278] The feature vector with the smallest distance value is selected to replace the feature vector in the third feature to obtain the fourth feature.

[0279] In this embodiment, a data processing apparatus is provided. Using this apparatus, the distance between features is calculated, and a minimum distance matching mechanism is employed. This allows for the direct calculation of the distance between the input sample and the known true distribution, effectively capturing feature deviations between samples and ensuring that the codebook vector can learn effective features from the real image.

[0280] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing device 20 provided in this application, the processing module 202 is used to perform similarity calculation based on the fourth feature and the third feature to obtain a first loss value;

[0281] Regularized loss is calculated based on the second reconstructed image and the third training image to obtain a second loss value and a third loss value, wherein the second loss value is the image pixel loss value and the third loss value is the image semantic loss value.

[0282] Based on the second reconstructed image and the third training image, a discrimination loss is calculated to obtain a fourth loss value;

[0283] The original image generation network and the initial codebook vector set are trained based on the first loss value, the second loss value, the third loss value, and the fourth loss value to obtain the initial image generation network and the target codebook vector set.

[0284] In this embodiment, a data processing apparatus is provided. Using this apparatus, the similarity between input features and reconstructed features, the image pixel loss between the input image and the reconstructed image, as well as semantic and discriminative losses are simultaneously constrained. This allows for the learning of a more comprehensive feature distribution of real images, enabling the target codebook vector set to establish feature representations containing only real images. This ensures that the discretized features accurately represent real images, thereby improving the subsequent detection capability for liveness data.

[0285] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0286] The processing module 202 is used to call the initial index prediction network to predict the index of the first feature in order to obtain the feature index;

[0287] Based on this feature index, a query is performed in the target codebook vector set to obtain the second feature.

[0288] In this embodiment, a data processing apparatus is provided. Using this apparatus, an index prediction network is introduced to predict feature indexes, thereby optimizing the process of querying the most similar features from the target codebook vector set, improving the accuracy of feature reconstruction, and thus enhancing the detection capability of liveness data.

[0289] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0290] The processing module 202 is used to determine the index prediction loss based on the feature index, and the index prediction loss is used to train the initial classification network, the initial image generation network and the initial index prediction network.

[0291] The processing module 202 is used to train the initial index prediction network, the initial classification network, and the initial image generation network based on the index prediction loss, the classification loss value, and the reconstruction loss value, so as to obtain the index prediction network, the target classification network, and the target image generation network.

[0292] In this embodiment, a data processing apparatus is provided. By employing the above apparatus and simultaneously constraining the classification loss, reconstruction loss, and index prediction loss, the model can better capture the global features of the image, more accurately reconstruct features, and more accurately classify the data, thereby improving its ability to distinguish between live data.

[0293] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0294] The processing module 202 is used to call the initial image generation network to extract features from the first training image to obtain the fifth feature;

[0295] The initial index prediction network is invoked to predict the index of the fifth feature in order to obtain the label index of the first training image;

[0296] When the training image corresponding to the second feature has a label index, the index prediction loss is determined based on the feature index and the label index.

[0297] When the training image corresponding to the second feature does not have a label index, the prediction loss for that index is determined to be zero.

[0298] In this embodiment, a data processing apparatus is provided. Using this apparatus, index prediction is performed on real images to determine label indices. Then, during training, when the input image is a real image, there is an index prediction loss, which is constrained, enabling the model to better reconstruct features and thus improve its ability to distinguish between live data.

[0299] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0300] The processing module 202 is used to perform similarity calculation based on the first feature and the second feature to obtain a fifth loss value;

[0301] Based on the first reconstructed image and each training image in the training image set, regularization loss is calculated to obtain a sixth loss value and a seventh loss value, wherein the sixth loss value is the image pixel loss value and the seventh loss value is the image semantic loss value.

[0302] The discrimination loss is calculated based on the first reconstructed image and each training image in the training image set to obtain the eighth loss value;

[0303] The fifth, sixth, seventh, and eighth loss values ​​are weighted and summed to obtain the reconstruction loss value.

[0304] In this embodiment, a data processing apparatus is provided. Using this apparatus, the similarity between input features and reconstructed features, the image pixel loss between the input image and the reconstructed image, as well as semantic and discriminative losses are simultaneously constrained. This allows for the learning of a more comprehensive feature distribution of real images, enabling the target codebook vector set to establish feature representations containing only real images. This ensures that the discretized features accurately represent real images, thereby improving the subsequent detection capability for liveness data.

[0305] The data processing apparatus provided in this application can be used on a server; please refer to [link / reference]. Figure 8 , Figure 8 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations stored in the storage media 330 on the server 300.

[0306] Server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0307] The steps performed by the server in the above embodiments can be based on this Figure 8 The server structure shown.

[0308] The data processing apparatus provided in this application can be used in terminal devices; please refer to [link / reference]. Figure 9 For ease of explanation, only the parts relevant to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. In the embodiments of this application, a smartphone is used as an example for illustration:

[0309] Figure 9 This is a block diagram illustrating a portion of the structure of a smartphone related to the terminal device provided in the embodiments of this application. (Reference) Figure 9 The smartphone includes components such as a radio frequency (RF) circuit 410, a memory 420, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a wireless fidelity (WiFi) module 470, a processor 480, and a power supply 490. Those skilled in the art will understand that... Figure 9 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0310] The following is combined Figure 9 A detailed introduction to the various components of a smartphone:

[0311] RF circuit 410 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 480; additionally, it transmits uplink data to the base station. Typically, RF circuit 410 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 410 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Message Service (SMS), etc.

[0312] The memory 420 can be used to store software programs and modules. The processor 480 executes various functions and data processing of the smartphone by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, phonebook, etc.). In addition, the memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0313] The input unit 430 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the smartphone. Specifically, the input unit 430 may include a touch panel 431 and other input devices 432. The touch panel 431, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 431), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 431 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 480, and can also receive and execute commands sent by the processor 480. In addition, the touch panel 431 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 431, the input unit 430 may also include other input devices 432. Specifically, other input devices 432 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0314] Display unit 440 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. Display unit 440 may include display panel 441, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, touch panel 431 may cover display panel 441. When touch panel 431 detects a touch operation on or near it, it transmits the information to processor 480 to determine the type of touch event. Subsequently, processor 480 provides corresponding visual output on display panel 441 based on the type of touch event. Although in Figure 9 In this embodiment, the touch panel 431 and the display panel 441 are two separate components to realize the input and output functions of the smartphone. However, in some embodiments, the touch panel 431 and the display panel 441 can be integrated to realize the input and output functions of the smartphone.

[0315] The smartphone may also include at least one sensor 450, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 441 according to the ambient light level, and the proximity sensor can turn off the display panel 441 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the smartphone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the smartphone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0316] Audio circuit 460, speaker 461, and microphone 462 provide an audio interface between the user and the smartphone. Audio circuit 460 converts received audio data into electrical signals and transmits them to speaker 461, where speaker 461 converts them into sound signals for output. On the other hand, microphone 462 converts collected sound signals into electrical signals, which are received by audio circuit 460, converted into audio data, and then processed by processor 480 before being transmitted via RF circuit 410 to, for example, another smartphone, or the audio data can be output to memory 420 for further processing.

[0317] WiFi is a short-range wireless transmission technology. Smartphones, through their WiFi modules (470), can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 9 WiFi module 470 is shown, but it is understood that it is not an essential component of a smartphone and can be omitted as needed without changing the nature of the invention.

[0318] The processor 480 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 420, and by calling data stored in the memory 420, thereby providing overall monitoring of the smartphone. Optionally, the processor 480 may include one or more processing units; optionally, the processor 480 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 480.

[0319] The smartphone also includes a power supply 490 (such as a battery) that supplies power to various components. Optionally, the power supply can be logically connected to the processor 480 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0320] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0321] The steps performed by the terminal device in the above embodiments can be based on this Figure 9 The terminal device structure is shown.

[0322] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0323] This application also provides a computer program product including a program, which, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0324] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0325] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0326] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0327] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0328] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0329] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Acquire an image to be processed, the image to be processed including the biometric features of the object to be identified; The encoder of the target image generation network is invoked to extract features from the image to be processed in order to obtain the features to be processed. The target image generation network is a pre-trained network. The features to be processed are discretized based on the target codebook vector set to obtain discretized features. The target codebook vector set is a feature space trained on real images and includes feature vectors of real images. The decoder of the target image generation network is invoked to decode the discretized features to obtain the reconstructed image of the image to be processed; The feature residuals are determined based on the features to be processed and the discretized features, and the pixel residuals are determined based on the reconstructed image and the image to be processed. The target classification network is invoked to perform recognition processing on the feature residuals and the pixel residuals to obtain the recognition result of the image to be processed. The target classification network is a pre-trained network.

2. The method according to claim 1, characterized in that, The step of calling the target classification network to perform recognition processing on the feature residuals and the pixel residuals to obtain the recognition result of the image to be processed includes: The feature residual and the pixel residual are fused to obtain a fused residual; The target classification network is invoked to perform recognition processing on the fusion residual to obtain the recognition result of the image to be processed.

3. The method according to claim 2, characterized in that, The process of fusing the feature residual and the pixel residual to obtain the fused residual includes: The feature residual and the pixel residual are fused based on an attention mechanism to obtain the fused residual.

4. The method according to claim 3, characterized in that, The process of fusing the feature residual and the pixel residual based on the attention mechanism to obtain the fused residual includes: Based on the first network structure and the attention mechanism, feature learning is performed on the feature residuals to obtain the first intermediate features; Based on the second network structure and the attention mechanism, the first intermediate feature and the pixel residual are fused to obtain the fused residual.

5. The method according to claim 4, characterized in that, Based on the first network structure and the attention mechanism, feature learning is performed on the feature residuals to obtain the first intermediate features, including: The feature residuals are extracted by a first convolutional layer to obtain a second intermediate feature, wherein the first convolutional layer is included in the first network structure; Feature learning is performed on the second intermediate feature and the feature to be processed based on an attention mechanism to obtain the first intermediate feature.

6. The method according to claim 4, characterized in that, The feature fusion of the first intermediate feature and the pixel residual based on the second network structure and the attention mechanism to obtain the fused residual includes: The first intermediate feature is upsampled by a sampling network to obtain a third intermediate feature, wherein the sampling network is included in the second network structure; The pixel residuals are used to extract features through a second convolutional layer and a global pooling layer to obtain a fourth intermediate feature. The second convolutional layer and the global pooling layer are included in the second network structure. The third intermediate feature and the fourth intermediate feature are concatenated to obtain the fifth intermediate feature; Attention is calculated on the fifth intermediate feature in both the channel dimension and the spatial position dimension to obtain positional attention features and channel attention features; The positional attention features and the channel attention features are multiplied and added together to obtain the fusion residual.

7. The method according to claim 1, characterized in that, The discretization of the features to be processed based on the target codebook vector set to obtain discretized features includes: The index prediction network is invoked to predict the index of the feature to be processed, so as to obtain the first feature index. The index prediction network is a pre-trained network. Based on the first feature index, a query is performed in the target codebook vector set to obtain the discretized feature.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Obtain a training image set, an initial image generation network, an initial classification network, and the target codebook vector set. The training image set includes a first training image and a second training image, wherein the first label of the first training image is a real image, and the second label of the second training image is a fake image. The initial image generation network is invoked to extract features from the training images in the training image set to obtain the first feature; The first feature is discretized based on the target codebook vector set to obtain the second feature; The initial image generation network is invoked to reconstruct the image based on the second feature to obtain a first reconstructed image; The training pixel residual and training feature residual are obtained based on the first reconstructed image, the first training image and the second training image; The initial classification network is invoked to classify the training pixel residuals and the training feature residuals to obtain the training classification results; Calculate the classification loss value based on the training classification result, the first label, and the second label; The reconstruction loss value is calculated based on the first feature, the second feature, the first reconstructed image, and the training images in the training image set; The initial classification network and the initial image generation network are trained based on the classification loss value and the reconstruction loss value to obtain the target classification network and the target image generation network.

9. The method according to claim 8, characterized in that, The method further includes: Obtain a third training image, an original image generation network, and an initial codebook vector set, wherein the third label of the third training image is a real image; The original image generation network is invoked to extract features from the third training image to obtain the third feature; The third feature is discretized based on the initial codebook vector set to obtain the fourth feature; The original image generation network is invoked to reconstruct the image of the fourth feature to obtain a second reconstructed image; The original image generation network and the initial codebook vector set are trained based on the third feature, the fourth feature, the second reconstructed image, and the third training image to obtain the initial image generation network and the target codebook vector set.

10. The method according to claim 9, characterized in that, The discretization of the third feature based on the initial codebook vector set to obtain the fourth feature includes: Traverse each feature vector in the initial codebook vector set and calculate the distance between each feature vector and each feature vector in the third feature; The feature vector with the smallest distance value is selected to replace the feature vector in the third feature to obtain the fourth feature.

11. The method according to claim 9, characterized in that, The original image generation network and the initial codebook vector set are trained based on the third feature, the fourth feature, the second reconstructed image, and the third training image to obtain the initial image generation network and the target codebook vector set, including: A similarity calculation is performed between the fourth feature and the third feature to obtain a first loss value; Regularized loss is calculated based on the second reconstructed image and the third training image to obtain a second loss value and a third loss value, wherein the second loss value is the image pixel loss value and the third loss value is the image semantic loss value; A fourth loss value is obtained by calculating the discrimination loss based on the second reconstructed image and the third training image; The original image generation network and the initial codebook vector set are trained based on the first loss value, the second loss value, the third loss value, and the fourth loss value to obtain the initial image generation network and the target codebook vector set.

12. The method according to claim 8, characterized in that, The discretization of the first feature based on the target codebook vector set to obtain the second feature includes: The initial index prediction network is invoked to predict the index of the first feature to obtain the feature index; The second feature is obtained by querying the target codebook vector set based on the feature index.

13. The method according to claim 12, characterized in that, The method further includes: The index prediction loss is determined based on the feature index, and the index prediction loss is used to train the initial classification network, the initial image generation network, and the initial index prediction network. Training the initial classification network and the initial image generation network based on the classification loss value and the reconstruction loss value to obtain the target classification network and the target image generation network includes: The initial index prediction network, the initial classification network, and the initial image generation network are trained based on the index prediction loss, the classification loss value, and the reconstruction loss value to obtain the index prediction network, the target classification network, and the target image generation network.

14. The method according to claim 13, characterized in that, The determination of index prediction loss based on the feature index includes: The initial image generation network is invoked to extract features from the first training image to obtain the fifth feature; The initial index prediction network is invoked to predict the index of the fifth feature in order to obtain the label index of the first training image; When a label index exists in the training image corresponding to the second feature, the index prediction loss is determined based on the feature index and the label index. When the training image corresponding to the second feature does not have a label index, the index prediction loss is determined to be zero.

15. The method according to claim 8, characterized in that, The reconstruction loss value is calculated based on the first feature, the second feature, the first reconstructed image, and the training images in the training image set, including: A similarity calculation is performed based on the first feature and the second feature to obtain a fifth loss value; Based on the first reconstructed image and each training image in the training image set, a regularization loss is calculated to obtain a sixth loss value and a seventh loss value, wherein the sixth loss value is the image pixel loss value and the seventh loss value is the image semantic loss value. The discrimination loss is calculated based on the first reconstructed image and each training image in the training image set to obtain the eighth loss value; The fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value are weighted and summed to obtain the reconstruction loss value.

16. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire an image to be processed, the image to be processed including the biometric features of the object to be identified; The processing module is used to call the encoder of the target image generation network to extract features from the image to be processed in order to obtain the features to be processed. The target image generation network is a pre-trained network. The features to be processed are discretized based on the target codebook vector set to obtain discretized features. The target codebook vector set is a feature space trained on real images and includes feature vectors of real images. The decoder of the target image generation network is called to decode the discretized features to obtain a reconstructed image of the image to be processed. Feature residuals are determined based on the features to be processed and the discretized features, and pixel residuals are determined based on the reconstructed image and the image to be processed. The target classification network is invoked to perform recognition processing on the feature residuals and the pixel residuals to obtain the recognition result of the image to be processed. The target classification network is a pre-trained network.

17. A computer device, characterized in that, include: Memory, processor, and bus system; The memory is used to store programs; The processor is configured to execute a program in the memory, and the processor is configured to execute the method of any one of claims 1 to 15 according to instructions in the program code; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

18. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 15.

19. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor using the method as described in any one of claims 1 to 15.

Images