Biological feature recognition model training method and device, equipment and storage medium

Abstract

The invention provides a biological feature recognition model training method and device, equipment and a storage medium. The method comprises the following steps: acquiring a to-be-trained biological feature recognition model sent by service party equipment; acquiring a plurality of image samples sent by at least one terminal device; carrying out at least one round of training on the to-be-trained biological feature recognition model based on the plurality of image samples to obtain a local biological feature recognition model; determining the model parameter offset of the local biological feature recognition model relative to the to-be-trained biological feature recognition model; and encrypting the model parameter offsets to obtain encrypted data, sending the encrypted data to the service party device, so that the service party device decrypts the encrypted data sent by the plurality of training party devices to obtain a plurality of model parameter offsets, and generating a global biological feature recognition model based on the plurality of model parameter offsets. According to the invention, the iteration efficiency and robustness of the biological feature recognition model can be improved on the basis of protecting data privacy.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular, to a method, apparatus, device, and storage medium for training a biometric recognition model. Background Art

[0002] In related technologies, the training of biometric recognition models is deployed on a centralized server. It is necessary to aggregate the biometric image (such as palmprint) datasets of users. Therefore, there is a risk of leakage of users' privacy information. At the same time, biometric image data belongs to sensitive data. To improve the recognition rate, a large amount of real data is required to train the model. However, these data are difficult to obtain, and the model upgrade of the biometric recognition model requires a large amount of computing resources, which will incur a large cost and expense. Summary of the Invention

[0003] Embodiments of this application provide a method, apparatus, device, computer program product, and computer-readable storage medium for training a biometric recognition model, which can protect user privacy and improve the iteration efficiency and robustness of the biometric recognition model.

[0004] The technical solution of the embodiments of this application is implemented as follows:

[0005] Embodiments of this application provide a method for training a biometric recognition model, which is applied to a training party device. A plurality of the training party devices and a service party device form a distributed training system; the method includes:

[0006] Obtain the biometric recognition model to be trained sent by the service party device;

[0007] Obtain a plurality of image samples sent by at least one terminal device, where the plurality of image samples are obtained by adding noise to a plurality of biometric images collected by the terminal device respectively;

[0008] Based on the plurality of image samples, perform at least one round of training on the biometric recognition model to be trained to obtain a local biometric recognition model;

[0009] Determine the model parameter offset of the local biometric recognition model relative to the biometric recognition model to be trained;

[0010] Encrypt the model parameter offset to obtain encrypted data, and send the encrypted data to the service party device, so that

[0011] The service party device decrypts the encrypted data sent by the plurality of training party devices to obtain multiple copies of the model parameter offset, and generates a global biometric recognition model based on the multiple copies of the model parameter offset.

[0012] An embodiment of the present application provides a method for training a biometric recognition model, which is applied to a service device, and the service device and multiple training devices form a distributed training system; the method includes:

[0013] Sending the biometric recognition model to be trained to the multiple training devices, so that each training device performs the following processing:

[0014] Obtaining multiple image samples sent by at least one terminal device, where the multiple image samples are obtained by adding noise to multiple biometric images collected by the terminal device respectively;

[0015] Based on the multiple image samples, performing at least one round of training on the biometric recognition model to be trained to obtain a local biometric recognition model;

[0016] Determining the model parameter offset of the local biometric recognition model relative to the biometric recognition model to be trained,

[0017] Encrypting the model parameter offset to obtain encrypted data, and sending the encrypted data to the service device;

[0018] Receiving the encrypted data, decrypting the encrypted data to obtain multiple copies of the model parameter offset;

[0019] Generating a global biometric recognition model based on multiple copies of the model parameter offset.

[0020] An embodiment of the present application provides a training device for a biometric recognition model, including:

[0021] An acquisition module, configured to acquire the biometric recognition model to be trained sent by a service device;

[0022] The acquisition module is further configured to acquire multiple image samples sent by at least one terminal device, where the multiple image samples are obtained by adding noise to multiple biometric images collected by the terminal device respectively;

[0023] A training module, configured to perform at least one round of training on the biometric recognition model to be trained based on the multiple image samples to obtain a local biometric recognition model;

[0024] The training module is further configured to determine the model parameter offset of the local biometric recognition model relative to the biometric recognition model to be trained;

[0025] An encryption module, configured to encrypt the model parameter offset to obtain encrypted data, and send the encrypted data to the service device, so that

[0026] The service provider device decrypts the encrypted data sent by the multiple training provider devices to obtain multiple copies of the model parameter offsets, and generates a global biometric recognition model based on the multiple copies of the model parameter offsets;

[0027] The training module is further configured to, in response to the number of multiple image samples sent by the at least one terminal device reaching a preset training threshold, transfer to the step of training the biometric recognition model to be trained based on the multiple image samples for at least one round;

[0028] The encryption module is further configured to obtain the public key in the key pair of each training provider device, where different training provider devices have different key pairs, and the key pair includes a private key and the public key; encrypt the model parameter offset through the public key to obtain the encrypted data of each training provider device.

[0029] An embodiment of the present application provides a training device for a biometric recognition model, including:

[0030] A sending module is configured to send a biometric recognition model to be trained to multiple training provider devices, so that each training provider device performs the following processing: obtain multiple image samples sent by at least one terminal device, where the multiple image samples are obtained by adding noise to multiple biometric images collected by the terminal device respectively; train the biometric recognition model to be trained based on the multiple image samples for at least one round to obtain a local biometric recognition model; determine the model parameter offset of the local biometric recognition model relative to the biometric recognition model to be trained; encrypt the parameter offset to obtain encrypted data, and send the encrypted data to the service provider device.

[0031] A receiving module is configured to receive the encrypted data and decrypt it to obtain multiple copies of the model parameter offsets;

[0032] A generating module is configured to generate a global biometric recognition model based on multiple copies of the model parameter offsets;

[0033] The sending module is further configured to send the global biometric recognition model to multiple terminal devices, so that the terminal devices identify the biometric features of the object to be detected based on the global biometric recognition model, and send the biometric features of the object to be detected to the service provider device, so that the service provider device compares the biometric features of the object to be detected with the biometric features in the registered biometric feature library, and sends the comparison result to the terminal device;

[0034] The sending module is further configured to send the global biometric recognition model to a third-party trusted device, so that the third-party trusted device performs a security assessment on the global biometric recognition model. In response to the security assessment of the global biometric recognition model by the third-party trusted device passing, the biometric features of the registered object are obtained through the global biometric recognition model to form a registered biometric database, and the registered biometric database is synchronized to the service-side device;

[0035] The receiving module is further configured to obtain the private key corresponding to each training-party device; decrypt the encrypted data through the private key to obtain the model parameter offset of each training-party device;

[0036] The generating module is further configured to perform a weighted sum process on the model parameter offsets of the multiple training-party devices to obtain a global parameter offset; generate the global biometric recognition model through the global parameter offset.

[0037] An embodiment of the present application provides an electronic device, which includes:

[0038] A memory for storing computer-executable instructions;

[0039] A processor, when executing the computer-executable instructions stored in the memory, implements the method for training a biometric recognition model provided by the embodiment of the present application.

[0040] An embodiment of the present application provides a computer-readable storage medium, storing a computer program or computer-executable instructions, which are used to implement the method for training a biometric recognition model provided by the embodiment of the present application when executed by a processor.

[0041] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions, which implement the method for training a biometric recognition model provided by the embodiment of the present application when the computer program or computer-executable instructions are executed by a processor.

[0042] The embodiments of the present application have the following beneficial effects:

[0043] Noise is added to the biometric image through at least one terminal device to obtain an image sample, which realizes the randomness and diversity of biometrics for training on the basis of protecting privacy, and further improves the robustness of the local biometric recognition model. A distributed training system is composed of multiple training party devices and one service party device. The calculation task of the model parameter offset is dispersed and implemented on multiple training party devices, and the service party device generates a global biometric recognition model by combining the model parameter offsets of multiple local biometric recognition models. It no longer depends on a centralized server, is easy to deploy, can be flexibly applied to different training scales, improves the efficiency of data processing during training, and improves the model generalization ability. Description of the Drawings

[0044] Figure 1 It is a schematic structural diagram of the training system architecture of the biometric recognition model provided by the embodiment of the present application;

[0045] Figure 2 It is a schematic structural diagram of the training party device provided by the embodiment of the present application;

[0046] Figure 3 It is a schematic structural diagram of the service party device provided by the embodiment of the present application;

[0047] Figure 4A It is a schematic diagram of the first process of the training method of the biometric recognition model provided by the embodiment of the present application;

[0048] Figure 4B It is a schematic diagram of the second process of the training method of the biometric recognition model provided by the embodiment of the present application;

[0049] Figure 4C It is a schematic diagram of the third process of the training method of the biometric recognition model provided by the embodiment of the present application;

[0050] Figure 4D It is a schematic diagram of the fourth process of the training method of the biometric recognition model provided by the embodiment of the present application;

[0051] Figure 4E It is a schematic diagram of the fifth process of the training method of the biometric recognition model provided by the embodiment of the present application;

[0052] Figure 4F It is a schematic diagram of the sixth process of the training method of the biometric recognition model provided by the embodiment of the present application;

[0053] Figure 4G It is a schematic diagram of the seventh process of the training method of the biometric recognition model provided by the embodiment of the present application;

[0054] Figure 4HIt is the eighth process schematic diagram of the training method of the biometric recognition model provided by the embodiments of the present application;

[0055] Figure 4I It is the ninth process schematic diagram of the training method of the biometric recognition model provided by the embodiments of the present application;

[0056] Figure 5 It is the schematic diagram of the application scenario of the biometric recognition model provided by the embodiments of the present application;

[0057] Figure 6 It is the schematic diagram of the principle of federated learning provided by the embodiments of the present application;

[0058] Figure 7 It is the schematic diagram of the training and deployment of the biometric recognition model provided by the embodiments of the present application;

[0059] Figure 8 It is the schematic diagram of the framework of the image generation model provided by the embodiments of the present application;

[0060] Figure 9 It is the schematic diagram of the principle of the generative adversarial network provided by the embodiments of the present application. Detailed implementation manners

[0061] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be construed as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0062] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0063] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the function of the module or unit.

[0064] In the embodiments of this application, when collecting and processing relevant data during actual application, it should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing behaviors within the scope authorized by laws, regulations and the personal information subject.

[0065] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the technical field to which this application pertains. The terms used in the embodiments of this application are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] Before further elaborating on the embodiments of this application, the nouns and terms involved in the embodiments of this application are described. The nouns and terms involved in the embodiments of this application are subject to the following explanations.

[0067] 1) Federated learning, a distributed machine learning method that allows training on multiple devices or data centers without the need to centralize the dataset in one place. In federated learning, each participating party trains the model locally and then aggregates the updated model parameters to a central server for aggregation, thus forming a global model. This method can protect data privacy and at the same time improve training efficiency by distributing the computing tasks to multiple devices.

[0068] 2) Differential privacy, a technology for protecting personal privacy that protects sensitive personal information by adding noise to query results. Specifically, it adds some random noise to the training data to mask the sensitive information of individuals in the data, thereby preventing attackers from intercepting or inferring personal privacy information. Differential privacy is a widely used privacy protection technology that has been applied in many fields, such as data mining, machine learning, and social network analysis.

[0069] 3) Deep learning, a machine learning method based on neural networks that simulates the working mode of human brain neurons through multiple layers of neural networks to achieve learning and prediction of large-scale data. It can automatically learn features, extract high-level abstract features from complex data, and has been widely applied in various fields, such as image recognition, speech recognition, and natural language processing.

[0070] 4) Biometric image refers to the digital image of biometric features used for authentication and identification. Biometric features can be, for example, a person's face, palm print, fingerprint, iris, etc.

[0071] 5) The privacy parameter, also known as the privacy budget or privacy budget parameter, is a value representing the degree of privacy protection, used to measure the amount of privacy information leaked by the differential privacy algorithm when processing data, usually denoted by ε. The size of the privacy parameter reflects the maximum degree of privacy leakage allowed by the differential privacy algorithm.

[0072] 6) The third-party trusted device is a device from a third-party trusted structure, such as a terminal or a server. The third-party trusted institution is an institution trusted by both the service provider and the training party participating in the model training.

[0073] 7) The privacy algorithm, also known as privacy computing, is a computing theory and method for the full-life-cycle protection of privacy information. Specifically, when processing information such as video, audio, image, graphics, text, numerical values, and ubiquitous network behavior information flows, operations such as describing, measuring, evaluating, and fusing the privacy information involved are performed to form a set of privacy computing theories, algorithms, and application technologies that are symbolic, formulaic, and have a quantitative evaluation standard, supporting the privacy information protection of multi-system integration.

[0074] There are some problems in the training of biometric recognition models in the related technologies:

[0075] 1) It is difficult to protect user privacy. The training of current biometric recognition models needs to be carried out on a centralized server, and it is necessary to aggregate the palmprint image data sets of users. To a certain extent, it is difficult to protect users' biometric information. If the data of this institution is leaked, the privacy information of users will not be guaranteed. Users may not trust the institutions providing biometric recognition services and worry that the institutions will leak users' privacy such as biometrics (such as palmprints, faces). At the same time, relevant regulatory departments may restrict the storage and use of users' biometrics by biometric recognition institutions;

[0076] 2) It is difficult to obtain data. Since biometric data belongs to sensitive data, biometric recognition institutions need a large amount of real data to improve the recognition rate, and these data are difficult to obtain;

[0077] 3) The training cost is high. The iterative upgrade of biometric recognition models requires a large amount of computing resources, which will cost a large amount of cost and overhead.

[0078] To solve the above problems, the embodiments of the present application provide a training method, device, equipment, computer-readable storage medium, and computer program product for a biometric recognition model, which can protect user privacy and improve the iterative efficiency and robustness of the biometric recognition model.

[0079] The electronic device provided by the embodiments of the present application can be implemented as various types of user terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), smart phones, smart speakers, smart watches, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc.

[0080] Taking the application of the embodiments of the present application to the access control verification scenario as an example, biometric features such as fingerprints or faces can be obtained through the global biometric recognition model provided by the embodiments of the present application, and the recognized biometric features are compared with the biometric features in the registered biometric feature library to obtain a verification result. In response to the verification result indicating that the currently recognized biometric feature exists in the registered biometric feature library, the access control verification is successful and relevant personnel are released. In response to the verification result indicating that the currently recognized biometric feature does not exist in the registered biometric feature library, the access control verification fails and relevant personnel are not allowed to pass. The embodiments of the present application can also be applied to scenarios such as electronic payment and identity verification for logging in to application programs. The training process of the global biometric recognition model provided by the embodiments of the present application is described below.

[0081] See Figure 1 , Figure 1 is a schematic structural diagram of the training system architecture of the biometric recognition model provided by the embodiments of the present application. By way of example, Figure 1 involves a training party device 100, a terminal device 200, a network 300, and a service party device 400 (for the sake of brief description, here the Figure 1 terminal device 200-1, terminal device 200-5, etc. in

[0082] In some embodiments, the training system of the biometric recognition model provided by the embodiments of the present application can be realized by the cooperation of the service provider device, the training party device, and the terminal device. For example, in response to the instruction to continue training the biometric recognition model issued by the service provider device 400, multiple terminal devices 200 respectively collect multiple biometric images, perform noise addition processing to obtain multiple image samples, and transmit the image samples to their respective corresponding training party devices 100 through the network 300. When the number of image samples reaches the preset training threshold, the corresponding training party device 100 trains the local biometric recognition model to obtain a model parameter offset, and encrypts and transmits the model parameter offset to the service provider device 400. The service provider device 400 decrypts and integrates multiple model parameter offsets to obtain the global biometric recognition model. The service provider device 400 issues the global biometric recognition model to multiple terminal devices 200 for biometric recognition (such as fingerprint recognition, face recognition, etc.) in the above access control verification scenario. Among them, before the training party device 100 trains the local biometric recognition model, the terminal device 200 transmits the pre-trained biometric recognition model to the training party device 100 while transmitting the image samples, or in response to the instruction to continue training the biometric recognition model issued by the service provider device 400, the service provider device 400 sends the pre-trained biometric recognition model to multiple training party devices 100. After the service provider device 400 obtains the global biometric recognition model, a third-party trusted device (from a third-party trusted institution) can be introduced to verify the security of the global biometric recognition model, and the third-party trusted device extracts features through the global biometric recognition model that has passed the security verification, and sends the extracted features to the service provider device 400 to update the biometric database.

[0083] In some embodiments, the terminal device, the service provider device, or the training party device can implement the training method of the biometric recognition model provided by the embodiments of the present application by running various computer-executable instructions or computer programs. For example, the computer-executable instructions can be commands at the microprogram level, machine instructions, or software instructions. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as an application that requires biometric verification to be used; it can also be a small program embedded in any APP, that is, a program that only needs to be downloaded to the browser environment to run. In short, the above computer-executable instructions can be instructions in any form, and the above computer programs can be applications, modules, or plugins in any form.

[0084] In some embodiments, the training device 100 or the service device 400 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. Among them, the cloud service may be an interactive processing service for the terminal to call.

[0085] In some embodiments, multiple servers can form a blockchain, and the training device 100 or the service device 400 is a node on the blockchain. There can be an information connection between each node in the blockchain, and information can be transmitted between nodes through the above information connection. Among them, the data related to the training method of the biometric recognition model provided in the embodiments of the present application can be saved on the blockchain.

[0086] The embodiments of the present application can be implemented with the help of Artificial Intelligence (AI) technology. Artificial intelligence is to use a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, and is a theory, method, technology, and application system that perceives the environment, acquires knowledge, and uses knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling the machines to have the functions of perception, reasoning, and decision-making.

[0087] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model is also called the large model or the basic model, and can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0088] The embodiments of the present application can be implemented with the help of computer vision technology (CV). Computer vision is a science that studies how to enable machines to "see". More specifically, it refers to using cameras and computers to replace human eyes to perform machine vision such as target recognition, tracking, and measurement, and further perform graphic processing to make the computer-processed images more suitable for human eyes to observe or be transmitted to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies and attempts to establish artificial intelligence systems that can obtain information from images or multi-dimensional data. The large model technology has brought important changes to the development of computer vision technology. Pre-trained models in the field of vision such as swin-transformer, ViT, V-MOE, and MAE can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology usually includes technologies such as image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, etc., and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0089] Taking the training party device for training a local biometric recognition model as an example, refer to Figure 2 , Figure 2 which is a schematic structural diagram of the training party device provided by the embodiments of the present application. Figure 2 The training party device 100 shown includes: at least one processor 110, a memory 130, and at least one network interface 120. Each component in the training party device 100 is coupled together through a bus system 140. It can be understood that the bus system 140 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 140 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 2 all kinds of buses are labeled as the bus system 140.

[0090] The processor 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0091] The memory 130 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memories, hard disk drives, optical disc drives, etc. The memory 130 optionally includes one or more storage devices that are physically located far from the processor 110.

[0092] The memory 130 includes volatile memory or non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 130 described in the embodiments of the present application is intended to include any suitable type of memory.

[0093] In some embodiments, the memory 130 is capable of storing data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof, which are exemplarily described below.

[0094] The operating system 131 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0095] The network communication module 132 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 120. Exemplary network interfaces 120 include: Bluetooth, wireless fidelity (WiFi), and universal serial bus (USB), etc.;

[0096] In some embodiments, the device provided by the embodiments of the present application can be implemented in software. Figure 2 Shown is a training device 133 for a biometric recognition model stored in the memory 130, which may be software in the form of programs and plugins, etc., including the following software modules: an acquisition module 1331, a training module 1332, and an encryption module 1333. These modules are logical, and thus can be arbitrarily combined or further split according to the functions to be implemented. The functions of each module will be described below.

[0097] Taking the service-side device for training and deploying a biometric recognition model as an example, see FIG. 3. Figure 3 It is a schematic structural diagram of the service-side device provided by the embodiments of the present application. Figure 3 The service-side device 400 shown includes: at least one processor 410, a memory 430, and at least one network interface 420. Each component in the service-side device 400 is coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between these components. The bus system 440 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 3 all kinds of buses are labeled as the bus system 440. For the specific descriptions of the processor 410 and the memory 430, refer to the above, and details will not be repeated here.

[0098] In some embodiments, the device provided by the embodiments of the present application may be implemented in software. Figure 3 The training device 433 of the biometric recognition model stored in the memory 430 is shown, which may be software in the form of a program and a plug-in, etc., including the following software modules: a sending module 4341, a receiving module 4342, and a generating module 4343. These modules are logical, so they can be combined arbitrarily or further split according to the functions to be implemented. The functions of each module will be described below.

[0099] In other embodiments, the device provided by the embodiments of the present application may be implemented in hardware. As an example, the device provided by the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the training method of the biometric recognition model provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may employ one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0100] Next, the training method of the biometric recognition model provided by the embodiments of the present application will be described. Refer to Figure 4A , Figure 4A which is the first process schematic diagram of the training method of the biometric recognition model provided by the embodiments of the present application. A plurality of training party devices and one service party device form a distributed training system, and the steps shown in Figure 4A will be described in combination.

[0101] In step 101, the training party device 100 obtains the biometric recognition model to be trained sent by the service party device 400.

[0102] In some embodiments, in response to the triggering operation of the instruction to continue training the biometric recognition model of the service party device 400, the service party device 400 sends the biometric recognition model to be trained to a plurality of training party devices 100.

[0103] In step 102, the training device 100 obtains a plurality of image samples sent by the terminal device 200, where the plurality of image samples are obtained by adding noise to a plurality of collected biometric images respectively by the terminal device 200.

[0104] In some embodiments, the terminal device 200 collects biometric images through devices such as cameras, webcams, or scanners, or obtains biometric images uploaded by the user locally, so as to obtain a plurality of biometric images. Before the process of collecting or obtaining biometric images, the terminal device 200 prompts the user about the use of the biometric images (for training a biometric recognition model after adding noise to the biometric images), and after the user is informed and consents to the collection, the collection process is carried out.

[0105] In some embodiments, referring to Figure 4B , Figure 4A in step 102 shown, the plurality of image samples are generated by the terminal device 200 through the following processing of steps 1021 to 1023 for each biometric image, which is specifically described below.

[0106] In step 1021, a biometric vector of the biometric image is obtained.

[0107] In some embodiments, for example, a deep neural network (DNN) (such as a convolutional neural network, a Residual Network (ResNet), etc.) can be used as an encoder to calculate through forward propagation to obtain the output feature vector of a certain layer in the neural network, so as to extract the biometric vector of the biometric image. After obtaining the biometric vector, the biometric vector is normalized, such as L2 normalization, to eliminate the scale difference in the feature vector and reduce the influence of noise.

[0108] In step 1022, the biometric vector is processed by adding noise perturbation to obtain a noise-added image feature vector.

[0109] In some embodiments, referring to Figure 4C , Figure 4B step 1022 shown, it can be realized by taking the biometric vector as the target biometric vector and the image sample corresponding to the biometric vector as the target biometric image, and performing the following steps 10221 to 10225, which is specifically described below.

[0110] In step 10221, a reference biometric image is determined from a plurality of biometric images.

[0111] In some embodiments, referring to Figure 4D , Figure 4CThe shown step 10221 can be implemented through the following steps 102211 to 102212, which will be specifically described below.

[0112] In step 102211, obtain the similarities between multiple biometric images and the target biometric image respectively.

[0113] In some embodiments, a similarity metric method can be selected to obtain the similarity between each biometric image and the target biometric image. The similarity metric methods adopted include Euclidean distance, Manhattan distance, cosine similarity, etc. Machine learning techniques can also be used to calculate the similarity between the biometric image and the target biometric image. For example, obtain multiple groups of image pair data and label the similarity levels of the image pair data. Obtain the feature vectors of the images through a pre-trained convolutional neural network (such as ResNet, VGGNet, etc.). Construct a machine learning model according to the feature vectors and the similarity metric methods described above, such as a support vector machine, random forest or neural network, etc. Input the feature vectors into the model, train the model to predict the similarity, and use the loss function and optimization algorithm to optimize the model parameters. Evaluate the trained model and calculate the performance metrics of the model, such as accuracy, precision, recall, etc. Finally, optimize the model according to the evaluation results, such as adjusting the model hyperparameters, increasing the training data, using regularization methods, etc.

[0114] In step 102212, use the biometric image corresponding to the maximum similarity as the reference biometric image.

[0115] In some embodiments, use the image with the maximum similarity to the target biometric image among the multiple biometric images as the reference biometric image.

[0116] In other embodiments, any one of the multiple biometric images can also be used as the reference biometric image.

[0117] In other embodiments, the image sample generated by the target biometric image through steps 1021 to 1023 can also be used as the reference biometric image (for example, by using any one of the multiple biometric images as the reference biometric image, performing steps 1021 to 1023 to generate the corresponding image sample, and then using the image sample as the reference biometric image to re-perform steps 1022 to 1023 to generate the final image sample).

[0118] Continue to refer to Figure 4C , in step 10222, determine the reference biometric vector of the reference biometric image.

[0119] In some embodiments, for example, a Deep Neural Network (DNN) can be used as an encoder to extract a reference biometric feature vector of a reference biometric image.

[0120] In step 10223, determine the distance between the elements of the corresponding same positions in the reference biometric feature vector and the target biometric feature vector, and determine the sum of each distance as the sensitivity.

[0121] In some embodiments, let the distance function d: R n →R n If there exists a privacy algorithm M: R n →R n Any output result s on the reference biometric feature vector X1 and the target biometric feature vector X2 satisfies:

[0122]

[0123] Then it is said that the privacy algorithm M satisfies ε-differential privacy, where ε represents the privacy budget. The larger the value of ε, the lower the privacy protection strength. Here, the value of the privacy budget can be specified externally (for example, the privacy budget can be set to 0.2), and adjusted according to the actual application situation. The embodiments of the present application do not limit the specific value of the privacy budget.

[0124] In some embodiments, the distance between the reference biometric feature vector X1 and the target biometric feature vector X2 is expressed as:

[0125]

[0126] Where represents transforming the distance between the elements of the reference biometric feature vector X1 and the target biometric feature vector X2 at the i-th position to the range of [0, 1] using standardization, i max and i min respectively represent the maximum and minimum values of the element values in all element pairs.

[0127] In some embodiments, the sum of each distance between the element pairs can be used as the sensitivity, or the average value of the sum of each distance between the element pairs can be used as the sensitivity.

[0128] In other embodiments, the preset multiple of the difference between i max and i min (for example, n(i max -i min )) can also be used as the sensitivity.

[0129] In step 10224, obtain the ratio of the sensitivity to the preset privacy parameter as the scale parameter.

[0130] In some embodiments, the size of the privacy budget ε can be adjusted according to actual needs. The smaller the value of ε, the higher the intensity of privacy protection.

[0131] In step 10225, random noise is obtained through a scale parameter, and the random noise is superimposed on the biometric vector to obtain a noise-added image feature vector.

[0132] In some embodiments, referring to Figure 4E , Figure 4C step 10225 shown, it can be implemented through the following steps 102251 to 102252, which are specifically described below.

[0133] In step 102251, the square of a preset multiple of the scale parameter is used as the variance, and the preset value is used as the mean to determine the inverse function of the cumulative distribution function of the Laplace distribution.

[0134] Here, the scale parameter is used to control the distribution shape of the Laplace distribution. The smaller the scale parameter, the sharper the peak and the heavier the tail of the Laplace distribution. The larger the scale parameter, the flatter the peak and the thinner the tail of the Laplace distribution.

[0135] In some embodiments, the square of a preset multiple (such as 2 times) of the scale parameter is used as the variance, and the preset value (such as 0) is used as the mean to determine the inverse function of the cumulative distribution function of the Laplace distribution.

[0136] In some embodiments, the probability density of the Laplace distribution is expressed as:

[0137]

[0138] where b represents the scale parameter and μ represents the mean (location parameter).

[0139] The cumulative distribution function of the Laplace distribution can be expressed as:

[0140]

[0141] where x represents the random variable. When x is greater than μ, sign(x - μ) is 1; when x is less than μ, sign(x - μ) is -1.

[0142] In step 102252, the random variable of the uniform distribution is used as the input of the inverse function, and the output value of the inverse function is obtained as the random noise.

[0143] In some embodiments, the random variable (such as the value range is [0A]) that follows the uniform distribution is used as the input of the inverse function, and the output value of the inverse function is obtained as the random noise.

[0144] Among them, the inverse function can be expressed as:

[0145] F -1 (u) = μ - b * sign(u - 0.5) ln(1 - 2|u - 0.5|) (5)

[0146] Sample u from the uniform distribution [0A], input it into the above formula (5), and obtain random noise that follows the Laplace distribution.

[0147] In some other embodiments, image compression technology (such as the compression method based on wavelet transform) can also be used to transform the biometric image. Subsequently, Laplace noise is added to the transformed data for perturbation, and then the perturbed data is restored to obtain an image sample with noise. Gaussian noise can also be added to the biometric image to obtain an image sample with noise (such as the method proposed in the article DP-IMAGE: DIFFERENTIAL PRIVACY FOR IMAGE DATA IN FEATURE SPACE). The above operations are all for adding noise to the biometric image to achieve the purpose of protecting privacy. The embodiments of the present application do not limit the specific implementation methods of adding noise to the biometric image.

[0148] Continue to refer to Figure 4B , in step 1023, the denoised image feature vector is subjected to image generation processing to obtain an image sample.

[0149] In some embodiments, refer to Figure 4F , Figure 4B In the step 1023 shown, the image sample is generated by an image generation model, and the image generation model is trained by constructing it as a generative adversarial network. Refer to Figure 8 , Figure 8 is a schematic diagram of the framework of the image generation model provided by the embodiments of the present application. After the biometric image is subjected to feature extraction by an encoder and then denoising perturbation processing to obtain denoised image features, the denoised image features are input into the generator to obtain an image sample, and the network parameters of the image generation model are updated through the discrimination results of the discriminator. The training of the image generation model can be achieved through the following steps 10231 to step 10236, which will be specifically described below.

[0150] In some embodiments, refer to Figure 9 , Figure 9It is a schematic diagram of the principle of the generative adversarial network provided by the embodiments of the present application. The generative adversarial network includes two parts: a generator and a discriminator. These two parts play a game and learn from each other during the training process, and finally reach a Nash equilibrium. The generator accepts random noise as input and outputs generated images to deceive the discriminator as much as possible; the role of the discriminator is to determine whether the input image is a generated image. After this mutually restrictive training is completed, the generative adversarial network can output images with a small difference from real images, improving the credibility of the generated images.

[0151] In step 10231, an image sample is generated by the generator.

[0152] In some embodiments, through neural network layers such as convolutional layers, transposed convolutional layers, and fully connected layers in the generator, as well as operations such as activation functions and normalization, the noise-added image feature vector is converted into an image sample with the same size as the biometric image.

[0153] In step 10232, based on the image sample and the biometric image, the loss value of the generator loss function is obtained.

[0154] In some embodiments, the generator loss function can be expressed as, for example:

[0155] L G =L GAN +L INFO (6)

[0156] Wherein, L GAN represents the adversarial loss function, and L INFO represents the information loss function.

[0157]

[0158] Wherein, x i represents the i-th biometric image in the current training batch, represents the image sample corresponding to the i-th biometric image generated by the generator in the current training batch.

[0159]

[0160] Wherein, is the expected value of the biometric image x i and is the expected value of the image sample generated by the generator. is the output of the discriminator network, representing the similarity between x i and . The adversarial loss function is divided into two parts: the first part is to output 0 as much as possible during the training process, so that the discriminator cannot distinguish xi and The second part is During the training process, output 1 as much as possible so that the discriminator can distinguish the generated image samples from the real biometric images. Therefore, the generator needs to generate image samples that are as close as possible to the biometric images to deceive the discriminator and minimize the value of L GAN to a minimum

[0161] In step 10233, update the parameters of the generator through the loss value of the generator loss function

[0162] In some embodiments, calculate the gradient of the generator loss function through backpropagation and optimization algorithms (such as Adam, SGD, AdamW, etc.), and update the parameters of the generator according to the gradient direction to minimize the loss value of the generator loss function and update the parameters of the generator

[0163] In step 10234, obtain the loss value of the discriminator loss function through the image samples and the biometric images

[0164] In some embodiments, the discriminator loss function can be expressed, for example, as

[0165]

[0166] wherein, the meanings of the respective parameters in formula (9) refer to the specific description of formula (8) above and will not be elaborated here

[0167] In step 10235, update the parameters of the discriminator through the loss value of the discriminator loss function

[0168] In some embodiments, calculate the gradient of the discriminator loss function through backpropagation and optimization algorithms (such as Adam), and update the parameters of the discriminator according to the gradient direction to minimize the loss value of the discriminator loss function and update the parameters of the discriminator

[0169] In step 10236, obtain the trained image generation model through the updated generator and the updated discriminator

[0170] In some embodiments, during the training process, cross-update the parameters of the generator and the discriminator, and finally reach a Nash equilibrium, so that the generator can generate realistic image samples and the discriminator can more accurately distinguish image samples from biometric images

[0171] Continue to refer to Figure 4A , in step 103, the training device 100 trains the biometric recognition model to be trained based on multiple image samples for at least one round to obtain a local biometric recognition model

[0172] In some embodiments, multiple image samples are diverse. For example, they include image samples under different angles and lighting conditions. Each image sample has a corresponding identity label. After preprocessing the image samples (such as image normalization, cropping, resizing, rotation, flipping, etc.), the features of the processed image samples are extracted through the neural network in the biometric recognition model to be trained, and then training is carried out. In this process, it involves optimizing the parameters of the biometric recognition model to be trained, so that the biometric recognition model to be trained can accurately classify the image samples or generate corresponding feature vectors. After training is completed, model evaluation and optimization can also be carried out. For example, the performance of the model is evaluated through an independent test data set, and performance metrics of the model are calculated, such as accuracy, precision, recall, etc. Finally, the model is tuned according to the evaluation results. For example, model hyperparameters are adjusted, training data is increased, regularization methods are used, etc., to improve the accuracy and robustness of the model, and the final local biometric recognition model is obtained.

[0173] Through steps 1021 to 1023 and step 103, the local biometric recognition model is trained without directly accessing the biometric image data. By adding appropriate noise to the biometric samples, the beneficial effect of protecting user privacy is achieved, and the problem in the above related technologies that the training of the biometric recognition model needs to be carried out on a centralized server and the palmprint image data set of users needs to be aggregated, which is difficult to protect the biometric information of users to a certain extent, is solved.

[0174] Through step 103, local training of the model is realized on each training party device, achieving the beneficial effect of reducing the computing and storage burden of the data center and improving the data processing efficiency. In addition, the image samples used by the training party device for training are collected from multiple terminal devices. The image samples collected by multiple terminal devices are centrally sent to the same training party device, making the image sample data diverse while solving the problem that it is difficult to obtain a large amount of biometric data in the above related technologies.

[0175] In step 104, the training party device 100 determines the model parameter offset of the local biometric recognition model relative to the biometric recognition model to be trained.

[0176] In some embodiments, the training party device 100 loads the model parameter files of the local biometric recognition model and the biometric recognition model to be trained, and compares the parameters of the two models layer by layer. For example, statistics such as the shape and numerical range of the parameter tensors are compared layer by layer to obtain the model parameter offset.

[0177] In step 105, the training party device 100 encrypts the model parameter offset to obtain encrypted data, and sends the encrypted data to the service party device 400.

[0178] In some embodiments, referring to Figure 4G , Figure 4A in step 105 shown, the training party device 100 encrypts the model parameter offset of the local biometric recognition model relative to the biometric recognition model to be trained, which can be implemented through the following steps 1051 to 1052, and the following is a specific description.

[0179] In step 1051, obtain the public key in the key pair of each training party device, where different training party devices have different key pairs, and the key pair includes a private key and a public key.

[0180] In some embodiments, a dedicated key management system can be used to generate (for example, generate the public key and private key through the RSA (Rivest-Shamir-Adleman) public key encryption algorithm) and store the private key.

[0181] In step 1052, encrypt the model parameter offset with the public key to obtain the encrypted data of each training party device.

[0182] In some embodiments, encrypt the plaintext information of the model parameter offset with the public key to obtain the encrypted data of each training party device.

[0183] Through steps 1051 to 1052, by performing training locally on the training party device and only transmitting the model offset parameters back to the service party device, the beneficial effect of reducing communication costs is achieved, and the problem of high training costs of the model and the need for a large amount of computing resources for iterative upgrading in the related art is solved.

[0184] In step 106, the service party device 400 decrypts the encrypted data to obtain multiple model parameter offsets.

[0185] In some embodiments, referring to Figure 4H , Figure 4A step 106 shown, it can be implemented through the following steps 1061 to 1062, and the following is a specific description.

[0186] In step 1061, obtain the private key corresponding to each training party device.

[0187] In some embodiments, a key exchange protocol (such as Diffie-Hellman, RSA and other encryption methods) can be used to negotiate the private key. During the key exchange process, each party can generate the private key according to some public parameters and random numbers. For example, two relatively prime numbers p and q can be randomly generated. First, calculate the product of these two numbers n = p * q. Next, calculate the Euler's totient function Select an integer e such that e satisfies and e and are relatively prime, e is the public key. Finally, calculate the multiplicative inverse d of e modulo , and d is the private key.

[0188] In step 1062, decrypt the encrypted data with the private key to obtain the model parameter offset of each training party device.

[0189] In some embodiments, after verifying the validity of the private key, decrypt the encrypted data with the private key according to the specific encryption algorithm and protocol to obtain the model parameter offset.

[0190] In step 107, the service party device 400 generates a global biometric recognition model based on multiple model parameter offsets.

[0191] In some embodiments, referring to Figure 4I , Figure 4A shown in step 107, it can be implemented through the following steps 1071 to 1072, which are specifically described below.

[0192] In step 1071, perform weighted sum processing on the model parameter offsets of multiple training party devices to obtain the global parameter offset.

[0193] In some embodiments, perform weighted sum processing on the model parameter offsets of multiple training party devices to aggregate all the offsets to obtain the global parameter offset.

[0194] In step 1072, generate a global biometric recognition model through the global parameter offset.

[0195] In some embodiments, add the global parameter offset to the biometric recognition model to be trained to complete the update of the model and generate a global biometric recognition model.

[0196] In some embodiments, the image samples are obtained by the terminal device 200 after receiving the instruction from the service party device 100 to continue training the biometric recognition model. Before performing Figure 4A shown in step 103, the following can also be executed: in response to the number of multiple image samples sent by at least one terminal device reaching the preset training threshold, transfer to the step of training the biometric recognition model to be trained based on multiple image samples for at least one round. That is, a training party device 100 can be trained through the image samples sent by multiple terminal devices 200, realizing the diversity of training data, and thus achieving the beneficial effect of improving the generalization ability of the biometric recognition model.

[0197] In some embodiments, the global biometric recognition model is used for the service provider device 400 to perform the following processes: sending the global biometric recognition model to a third-party trusted device so that the third-party trusted device can perform a security assessment on the global biometric recognition model; in response to the security assessment of the global biometric recognition model by the third-party trusted device passing, obtaining the biometrics of the registered object through the global biometric recognition model to form a registered biometric database, and synchronizing it to the service provider device.

[0198] In some embodiments, the methods for the third-party trusted device to perform the security assessment of the global biometric recognition model include at least one of the following:

[0199] Checking the source and quality of the image samples used for training the global biometric recognition model, evaluating the architecture and parameter settings of the model, and checking the deployment situation and access control measures of the model to determine security vulnerabilities and risks;

[0200] Checking the usage and processing of the image samples during the training process of the global biometric recognition model;

[0201] Quantifying the degree of privacy protection of the global biometric recognition model when processing input data, for example, using differential privacy metrics to quantify the degree of privacy protection;

[0202] Performing an adversarial attack on the global biometric recognition model, that is, deliberately creating small perturbations of the input samples to obtain the reaction and output results of the global biometric recognition model, which can help evaluate the robustness of the model against targeted attacks (such as adding noise, modifying the input, etc.).

[0203] Through the security assessment of the global biometric recognition model by the third-party trusted device, it is ensured that the source of the image samples is legal and trustworthy, the abuse or leakage of the image sample data is avoided, the trust in the model security is increased, and through the performance of the model in the face of various abnormal situations and attacks, the robustness of the model is further improved, achieving the beneficial effects of improving the reliability and stability of the model and facilitating the model to better cope with risks and challenges in real scenarios.

[0204] In some embodiments, the global biometric recognition model is also used for the service provider device 400 to send it to multiple terminal devices 200 so that the terminal devices 200 can identify the biometrics of the object to be detected based on the global biometric recognition model, send the biometrics of the object to be detected to the service provider device 400, so that the service provider device 400 can compare the biometrics of the object to be detected with the biometrics in the registered biometric database, and send the comparison result to the terminal devices 200.

[0205] Next, an exemplary application of the embodiments of the present application in the palmprint recognition scenario will be described.

[0206] Palm brushing recognition can be used in many application scenarios, such as access control scenarios, electronic payment scenarios, identity verification scenarios, etc. Specifically, the application of palm brushing recognition in access control scenarios refers to using palmprint recognition technology for identity verification and access control authorization. The hardware facilities involved in the palmprint recognition scenario include: palmprint acquisition devices (corresponding to the terminal devices above), servers or cloud computing resources and devices (corresponding to the service provider devices above), and palmprint data storage devices (which can be stored locally on the terminal device or in a specific database or file system), etc.

[0207] The application process of palm brushing recognition is as follows:

[0208] First, the terminal device recognizes the user's palm and obtains the biometric (palmprint feature) of the user through the biometric recognition model provided by the embodiment of the present application to obtain the palmprint feature of the user (corresponding to the biometric above); next, the terminal device transmits the user's biometric to the service provider device, and the service provider device compares the user's biometric with the stored features in the registered biometric library to confirm whether the user's identity is legal and sends the verification result to the terminal device; finally, in response to the user's successful identity verification, the terminal device executes the access control opening instruction, and the user can enter the specified area; otherwise, the access control will remain closed and the user cannot enter.

[0209] Compared with the access control verification methods of related technologies, palm brushing recognition technology has higher security and accuracy, and can effectively prevent security problems such as counterfeiting and misappropriation. At the same time, palm brushing recognition technology is also more convenient and fast, and users only need to easily brush their palms to complete identity verification and access control authorization. Therefore, in modern access control scenarios, palm brushing recognition technology has been widely used.

[0210] However, in the palmprint recognition scenario of related technologies, it is necessary to upload the user's palmprint image to the central server (corresponding to the service provider device above). The central server needs to store all user images and image features for user feature recognition and model upgrade and update. The user's biometric information data is extremely sensitive. In this mode of storing user palm images in the central server, users are very likely to distrust the organization providing the palm brushing service and thus refuse to activate the palm brushing service; if the central server lacks security protection and accidentally leaks the user's palmprint image, it will cause great damage to the user's privacy; if relevant regulatory authorities strictly control the palm brushing service providers and do not allow service providers to store and use palmprint images, the traditional palmprint recognition scenario will not be able to be implemented.

[0211] See Figure 5 , Figure 5It is a schematic diagram of the application scenario of the biometric recognition model provided by the embodiments of the present application. Through the biometric recognition model training method based on federated learning, an edge server (corresponding to the training party device above) is introduced, and the palmprint data collected by some terminal devices is uploaded to the edge server. The edge server performs local training of federated learning, thereby reducing the training cost. See Figure 6 , Figure 6 It is a schematic diagram of the principle of federated learning provided by the embodiments of the present application. In federated learning, the central server (corresponding to Figure 6 the service party device in Figure 6 ) can use an open-source dataset to train a network model or directly use a pre-trained network model, and distribute the network model to each training party device (corresponding to

[0212] Training Party A, Training Party B, and Training Party C in

[0213] ). Each training party device collects user data respectively as the dataset for local network model training. Each training party device performs a local training using the local user data after a period of time. The local training uses backpropagation to update the local model parameters. This process will obtain the slope of each parameter with respect to the loss function, so as to use the gradient descent method to make the new parameters obtain a smaller loss function. After local training, the offset of the model parameters will be obtained, and the offset of the model parameters will be encrypted and transmitted to the service party device for global network model update and iteration. The service party device will perform global iteration on the results of local training of all training party devices, perform a weighted sum on the model offsets obtained from local training of all devices, and then add it to the global network model to complete an update of the global model. After the global model update is completed, the new network model will be distributed to each training party device, and all training party devices will update the model to extract new data features, thus completing an update and iteration of the model.Since there may be a risk of privacy leakage during the process of the terminal device uploading the image to the training party's server, during this process, differential privacy technology is used to add an appropriate amount of noise to the collected palm image to achieve a better privacy protection effect. Here, it is necessary to balance the size of the noise and the accuracy of the model to ensure that the error introduced by the noise is within an acceptable range. At the same time, more advanced differential privacy algorithms should be used to reduce the impact of the noise on training. Among them, the third-party trusted institution can be relevant government departments or social institutions with strong credibility. The user's palm image is stored here, which guarantees the user's privacy and security to a certain extent. When some service providing institutions need to use the user's palm image to extract features, they need to apply to these trusted institutions for use. The trusted institution conducts multi-faceted reviews on the service providing institutions to determine whether they have the risk of violation, and then allows the feature extraction model of the service providing institution to extract features from the user's palm image library of the trusted institution. The service providing institution needs to store the extracted features in its own database for feature comparison when providing palmprint recognition services.

[0214] Through the above method, in the case where the palm brushing service providing institution does not store the user's palm image, business functions such as access control, electronic payment, and identity verification can be provided for the user. When the user activates the palm brushing related service, it is necessary to collect the palm image information through the terminal device, upload the original image information to the third-party trusted institution, and upload the noisy palm image to the edge server. The edge server will perform local training, and the central server will perform global training. Thereby protecting the user's privacy data to a certain extent and improving the reputation of the service providing institution among the user group.

[0215] The above palmprint recognition technology is implemented by a biometric recognition model trained by the biometric recognition model training method provided in the embodiments of this application. See Figure 7 , Figure 7 which is a schematic diagram of the training and deployment of the biometric recognition model provided in the embodiments of this application. The following will be described in detail.

[0216] In step 201, the service device initializes the network model.

[0217] In some embodiments, the service device (corresponding to Figure 7 the central server in the above) initializes the network model and assigns initial values to each network parameter (such as initializing network parameters such as weights and biases with random numbers).

[0218] In step 202, the service device distributes the global network model.

[0219] In some embodiments, the service device distributes the global network model (corresponding to the biometric recognition model to be trained above) to each terminal device.

[0220] In step 203, the terminal device locally deploys the network model.

[0221] In some embodiments, the terminal device installs libraries and tools (such as PyTorch, etc.) for loading and running the network model, loads the model file through the Application Program Interface (API) provided by the library, and performs local deployment of the network model.

[0222] In step 204, the terminal device controls whether to continue training the model.

[0223] In some embodiments, in response to the terminal device continuing to train the model, it proceeds to execute step 205. In response to the terminal device not continuing to train the model, the terminal device uses the current model for palmprint recognition as described above.

[0224] In step 205, the terminal device collects biometric images.

[0225] In some embodiments, the terminal device collects biometric images and stores the biometric images locally.

[0226] In step 206, the terminal device performs noise addition processing on the biometric images.

[0227] In some embodiments, the terminal device obtains the biometric vectors of the biometric images, performs noise addition perturbation processing on the biometric vectors to obtain noise-added image feature vectors, and performs image generation processing on the noise-added image feature vectors to obtain image samples. Here, refer to the descriptions in steps 1021 to 1023 above.

[0228] In step 207, the training party device collects the noise-added biometric images.

[0229] In some embodiments, the terminal device sends the noise-added biometric images (i.e., image samples) to the training party device (corresponding to Figure 7 the edge server in) and the training party device collects the noise-added biometric images to prepare for subsequent training.

[0230] In step 208, when the training party device has collected a certain amount of images, it starts local training.

[0231] In some embodiments, multiple image samples are diverse. For example, they include image samples under different angles and lighting conditions. Each image sample has a corresponding identity tag. After preprocessing the image samples (such as image normalization, cropping, resizing, rotation, flipping, etc.), the features of the processed image samples are extracted through the neural network in the biometric recognition model to be trained, and then training is carried out. In this process, it involves optimizing the parameters of the biometric recognition model to be trained, so that the biometric recognition model to be trained can accurately classify the image samples or generate corresponding feature vectors. After training is completed, model evaluation and optimization are also carried out. For example, the performance of the model is evaluated through an independent test data set, and adjustments and optimizations are made according to the evaluation results to improve the accuracy and robustness of the model, and finally a local biometric recognition model is obtained.

[0232] In some embodiments, the terminal device transmits the pre-trained biometric recognition model to the training party device while transmitting the image samples.

[0233] In other embodiments, in response to the triggering operation of the instruction to continue training the biometric recognition model by the service party device, the service party device sends the biometric recognition model to be trained to multiple training party devices.

[0234] In step 209, the training party device encrypts the offset of the local training model parameters and sends it to the service party device.

[0235] In some embodiments, the training party device determines the model parameter offset of the local biometric recognition model relative to the biometric recognition model to be trained, encrypts the model parameter offset to obtain encrypted data, and sends the encrypted data to the service party device. Here, refer to the description in steps 104 to 105 above.

[0236] In step 210, the service party device decrypts the parameter offset of the training party device and performs a weighted sum.

[0237] In some embodiments, the service party device obtains the private key corresponding to each training party device, decrypts the encrypted data through the private key to obtain the model parameter offset of each training party device, performs a weighted sum processing on the model parameter offsets of multiple training party devices, aggregates all the offsets to obtain the global parameter offset. Here, refer to the description in steps 1053 to 1055 above.

[0238] In step 211, the service party device updates the global model.

[0239] In some embodiments, the service party device adds the global parameter offset to the biometric recognition model to be trained to complete the update of the model, and generates a global biometric recognition model (i.e., the updated global model).

[0240] In step 212, a third-party trusted institution detects the security of the updated model.

[0241] In some embodiments, the ways for the third-party trusted institution to evaluate the security of the global biometric recognition model through its device (corresponding to the third-party trusted device above) include at least one of the following:

[0242] Check the source and quality of the image samples used to train the global biometric recognition model, evaluate the architecture and parameter settings of the model, and check the deployment situation and access control measures of the model to discover potential security vulnerabilities and risks;

[0243] Check the usage and processing of image samples during the training process of the global biometric recognition model, ensure that the source of the image samples is legal and trustworthy, and avoid the abuse or leakage of image sample data, thereby increasing the trust in the model security;

[0244] Quantify the degree of privacy protection of the global biometric recognition model when processing input data, for example, use differential privacy metrics to quantify the degree of privacy protection;

[0245] Conduct adversarial attacks on the global biometric recognition model, that is, deliberately create small perturbations of the input samples, and observe the reaction and output results of the global biometric recognition model, which can help evaluate the robustness of the model against targeted attacks (such as adding noise, modifying the input, etc.).

[0246] In step 213, the third-party trusted institution extracts biometric features using the updated model.

[0247] In some embodiments, in response to the passing of the security evaluation of the global biometric recognition model, the third-party trusted institution obtains the biometric features of the registered objects through the global biometric recognition model to form a registered biometric database, and synchronizes it to the service-side device.

[0248] In step 214, the service-side device updates the global biometric database.

[0249] In some embodiments, the service-side device receives the registered biometric database sent by the third-party trusted institution and updates the global biometric database.

[0250] In step 215, the service-side device distributes the global network model.

[0251] In some embodiments, the service-side device distributes the global biometric recognition model to the terminal device.

[0252] In step 216, the terminal device updates the local model.

[0253] In some embodiments, the terminal device receives the global biometric recognition model sent by the service provider device, updates the local model, and proceeds to step 204 to determine whether to perform the next training.

[0254] Through steps 201 to 216, the model is trained without directly accessing the biometric image data. By adding appropriate noise to the biometric samples, the beneficial effect of protecting user privacy is achieved; by locally training the model on each training party device, the beneficial effect of reducing the computing and storage burden of the data center and improving the data processing efficiency is achieved; by using the image samples collected by multiple terminal devices on different training party devices for training, the beneficial effect of enhancing the generalization ability of the model is achieved; by performing training locally on the training party device and only transmitting the model offset parameters back to the service provider device, the beneficial effect of reducing communication costs is achieved.

[0255] Next, the implementation of the training device 133 of the biometric recognition model provided by the embodiments of the present application as a software module will be further described. In some embodiments, as Figure 2 shown, the software module in the training device 133 of the biometric recognition model stored in the memory 130 may include:

[0256] An acquisition module 1331, configured to acquire the biometric recognition model to be trained sent by the service provider device.

[0257] A training module 1332, configured to perform at least one round of training on the biometric recognition model to be trained based on the multiple image samples to obtain a local biometric recognition model.

[0258] An encryption module 1333, configured to encrypt the model parameter offset amount to obtain encrypted data, and send the encrypted data to the service provider device, so that the service provider device decrypts the encrypted data sent by multiple training party devices to obtain multiple copies of the model parameter offset amount, and generates a global biometric recognition model based on the multiple copies of the model parameter offset amount.

[0259] In some embodiments, the acquisition module 1331 is further configured to acquire multiple image samples sent by at least one terminal device, where the multiple image samples are obtained by adding noise to the multiple biometric images collected by the terminal device respectively.

[0260] In some embodiments, the training module 1332 is further configured to determine the model parameter offset amount of the local biometric recognition model relative to the biometric recognition model to be trained.

[0261] In some embodiments, the training module 1332 is further configured to, in response to the number of multiple image samples sent by the at least one terminal device reaching a preset training threshold, proceed to the step of training the biometric recognition model to be trained for at least one round based on the multiple image samples.

[0262] In some embodiments, the encryption module 1333 is further configured to obtain the public key in the key pair of each training party device, where different training party devices have different key pairs, and the key pair includes a private key and the public key; encrypt the model parameter offset through the public key to obtain the encrypted data of each training party device.

[0263] The following continues to describe the exemplary structure of the software module of the training device 433 for the biometric recognition model provided in the embodiments of the present application. In some embodiments, as Figure 3 shown, the software module in the training device 433 for the biometric recognition model stored in the memory 430 may include: a sending module 4341, configured to send the biometric recognition model to be trained to multiple training party devices, so that each training party device performs the following processing: obtaining multiple image samples sent by at least one terminal device, where the multiple image samples are obtained by adding noise to multiple biometric images collected by the terminal device respectively; training the biometric recognition model to be trained for at least one round based on the multiple image samples to obtain a local biometric recognition model; determining the model parameter offset of the local biometric recognition model relative to the biometric recognition model to be trained; encrypting the parameter offset to obtain encrypted data, and sending the encrypted data to the service party device.

[0264] A receiving module 4342, configured to receive the encrypted data and decrypt it to obtain multiple copies of the model parameter offset.

[0265] A generating module 4343, configured to generate a global biometric recognition model based on multiple copies of the model parameter offset.

[0266] In some embodiments, the sending module 4341 is further configured to send the global biometric recognition model to multiple terminal devices, so that the terminal devices identify the biometric features of the object to be detected based on the global biometric recognition model, and send the biometric features of the object to be detected to the service party device, so that the service party device compares the biometric features of the object to be detected with the biometric features in the registered biometric feature library and sends the comparison result to the terminal device.

[0267] In some embodiments, the receiving module 4342 is further configured to obtain the private key corresponding to each training party device; decrypt the encrypted data with the private key to obtain the model parameter offset of each training party device.

[0268] In some embodiments, the generating module 4343 is further configured to perform a weighted sum process on the model parameter offsets of multiple training party devices to obtain a global parameter offset; generate the global biometric recognition model with the global parameter offset.

[0269] An embodiment of the present application provides a computer program product, which includes a computer program or computer executable instructions, and the computer program or computer executable instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer-readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the above-mentioned biometric recognition model training method of the embodiment of the present application.

[0270] An embodiment of the present application provides a computer-readable storage medium storing computer executable instructions, in which computer executable instructions or a computer program are stored. When the computer executable instructions or the computer program are executed by a processor, the processor will be caused to execute the biometric recognition model training method provided by the embodiment of the present application. For example, as Figure 4A shown in the biometric recognition model training method.

[0271] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0272] In some embodiments, the computer executable instructions may be in the form of a program, software, software module, script or code, and may be written in any form of programming language (including compiled or interpreted language, or declarative or procedural language), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine or other unit suitable for use in a computing environment.

[0273] As an example, the computer executable instructions may or may not correspond to a file in the file system, and may be stored as part of a file storing other programs or data. For example, they may be stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperating files (for example, files storing one or more modules, subroutines, or code portions).

[0274] As an example, the computer-executable instructions can be deployed to execute on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed at multiple locations and interconnected by a communication network.

[0275] In summary, through the embodiments of the present application, at least one terminal device is used to perform noise addition processing on a biometric image to obtain an image sample. On the basis of protecting privacy, the randomness and diversity of biometrics for training are realized, thereby improving the robustness of the local biometric recognition model. A distributed training system is formed by multiple training party devices and one service party device. The calculation task of the model parameter offset is realized on multiple training party devices, and the service party device combines the model parameter offsets of multiple local biometric recognition models to generate a global biometric recognition model. It no longer depends on a centralized server, is easy to deploy, can be flexibly applied to different training scales, improves the efficiency of data processing during training, and improves the model generalization ability.

[0276] The above is only the embodiments of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A training method for a biometric recognition model, characterized in that, Applied to training party devices, multiple said training party devices and one service party device form a distributed training system; the method includes: Obtain the biometric recognition model to be trained sent by the service party device; Obtain multiple image samples sent by at least one terminal device, wherein the multiple image samples are obtained by the terminal device respectively adding noise to multiple collected biometric images; Based on the multiple image samples, perform at least one round of training on the biometric recognition model to be trained to obtain a local biometric recognition model; Determine the model parameter offset of the local biometric recognition model relative to the biometric recognition model to be trained; Encrypt the model parameter offset to obtain encrypted data, and send the encrypted data to the service party device so that The service party device decrypts the encrypted data sent by multiple said training party devices to obtain multiple copies of the model parameter offset, and generates a global biometric recognition model based on the multiple copies of the model parameter offset.

2. The method according to claim 1, wherein The multiple image samples are generated by the terminal device in the following manner: Perform the following processing for each said biometric image: Obtain the biometric vector of the biometric image; Perform a noise perturbation process on the biometric vector to obtain a noise-added image feature vector; Perform an image generation process on the noise-added image feature vector to obtain an image sample.

3. The method according to claim 2, wherein The performing a noise perturbation process on the biometric vector to obtain a noise-added image feature vector includes: Taking the biometric vector as the target biometric vector and the image sample corresponding to the biometric vector as the target biometric image, and perform the following processing: Determine a reference biometric image from the multiple biometric images; Determine the reference biometric vector of the reference biometric image; Determine the distance between the element pairs at the same corresponding positions in the reference biometric vector and the target biometric vector, and determine the sum of each said distance as the sensitivity; Obtain the ratio of the sensitivity to a preset privacy parameter as the scale parameter; Obtain random noise through the scale parameter, and superimpose the random noise on the biometric vector to obtain the noise-added image feature vector.

4. The method according to claim 3, wherein The determining a reference biometric image from the multiple biometric images includes: Obtain the similarity between each of the multiple biometric images and the target biometric image; Taking the biometric image corresponding to the maximum said similarity as the reference biometric image.

5. The method according to claim 3, characterized in that The obtaining random noise through the scale parameter includes: Taking the square of a preset multiple of the scale parameter as the variance and a preset value as the mean to determine the inverse function of the cumulative distribution function of the Laplace distribution; Taking a random variable of the uniform distribution as the input of the inverse function, and obtaining the output value of the inverse function as the random noise.

6. The method according to claim 1, characterized in that, The image sample is obtained by the terminal device after receiving the training biometric recognition model instruction from the service party device; Before performing at least one round of training on the to-be-trained biometric recognition model based on the multiple image samples, the method further includes: In response to the number of multiple image samples sent by the at least one terminal device reaching a preset training threshold, transfer to the step of performing at least one round of training on the to-be-trained biometric recognition model based on the multiple image samples.

7. The method according to claim 1, wherein The encrypting of the model parameter offset of the local biometric recognition model relative to the to-be-trained biometric recognition model includes: Obtain the public key in the key pair of each training party device, where different training party devices have different key pairs, and the key pair includes a private key and the public key; Encrypt the model parameter offset through the public key to obtain encrypted data of each training party device.

8. The method according to claim 1, wherein The global biometric recognition model is generated by the service party device in the following manner: Obtain the private key corresponding to each training party device; Decrypt the encrypted data through the private key to obtain the model parameter offset of each training party device; Perform weighted sum processing on the model parameter offsets of multiple training party devices to obtain a global parameter offset; Generate the global biometric recognition model through the global parameter offset.

9. The method according to any one of claims 1 to 7, wherein The global biometric recognition model is used for the service party device to perform the following processing: Send the global biometric recognition model to a third-party trusted device, so that The third-party trusted device performs a security evaluation on the global biometric recognition model. In response to the security evaluation of the global biometric recognition model by the third-party trusted device passing, obtain the biometric features of the registered object through the global biometric recognition model to form a registered biometric feature library, and synchronize it to the service party device.

10. The method according to claim 9, characterized in that, The manner in which the third-party trusted device performs a security evaluation on the global biometric recognition model includes at least one of the following: Check the source and quality of the image samples used to train the global biometric recognition model, evaluate the architecture and parameter settings of the model, check the deployment situation and access control measures of the model to determine security vulnerabilities and risks; Check the usage and processing of the image samples during the training process of the global biometric recognition model; Quantify the privacy protection level of the global biometric recognition model; Perform an adversarial attack on the global biometric recognition model to obtain the reaction and output results of the global biometric recognition model.

11. The method according to any one of claims 1 to 7, wherein The global biometric recognition model is further used for the service party device to send to multiple terminal devices, so that The terminal device recognizes the biometric features of the to-be-detected object based on the global biometric recognition model, and sends the biometric features of the to-be-detected object to the service party device, so that The service party device compares the biometric features of the to-be-detected object with the biometric features in the registered biometric feature library, and sends the comparison result to the terminal device.

12. The method according to any one of claims 1 to 5, characterized in that, The image sample is generated by an image generation model, which is trained in the form of a generative adversarial network. The generative adversarial network includes a generator and a discriminator. The training process of the image generation model includes: Generating the image sample through the generator; Obtaining the loss value of the generator loss function through the image sample and the biometric image; Updating the parameters of the generator through the loss value of the generator loss function; Obtaining the loss value of the discriminator loss function through the image sample and the biometric image; Updating the parameters of the discriminator through the loss value of the discriminator loss function; Obtaining the trained image generation model through the updated generator and the updated discriminator.

13. A training method for a biometric recognition model, characterized in that, Applied to the service-side device, the service-side device and multiple training-side devices form a distributed training system; the method includes: Sending the biometric recognition model to be trained to the multiple training-side devices, so that each training-side device performs the following processing: Obtaining a plurality of image samples sent by at least one terminal device, where the plurality of image samples are obtained by adding noise to a plurality of biometric images collected by the terminal device respectively; Training the biometric recognition model to be trained based on the plurality of image samples for at least one round to obtain a local biometric recognition model; Determining the model parameter offset of the local biometric recognition model relative to the biometric recognition model to be trained, Encrypting the model parameter offset to obtain encrypted data, and sending the encrypted data to the service-side device; Receiving the encrypted data, decrypting the encrypted data to obtain multiple copies of the model parameter offset; Generating a global biometric recognition model based on multiple copies of the model parameter offset.

14. The method according to claim 13, wherein The service-side device decrypts the encrypted data sent by the multiple training-side devices to obtain multiple copies of the model parameter offset, including; Obtaining the private key corresponding to each training-side device; Decrypting the encrypted data through the private key to obtain the model parameter offset of each training-side device; The generating a global biometric recognition model based on multiple copies of the model parameter offset includes: Performing weighted sum processing on the model parameter offsets of the multiple training-side devices to obtain a global parameter offset; Generating the global biometric recognition model through the global parameter offset.

15. A training device for a biometric recognition model, characterized in that, The device includes: An acquisition module, configured to acquire the biometric recognition model to be trained sent by the service-side device; The acquisition module is further configured to acquire a plurality of image samples sent by at least one terminal device, where the plurality of image samples are obtained by adding noise to a plurality of biometric images collected by the terminal device respectively; A training module, configured to train the biometric recognition model to be trained based on the plurality of image samples for at least one round to obtain a local biometric recognition model; The training module is further configured to determine the model parameter offset of the local biometric recognition model relative to the biometric recognition model to be trained; The encryption module is configured to encrypt the model parameter offset to obtain encrypted data, and send the encrypted data to the service-side device, so that the service-side device decrypts the encrypted data sent by multiple training-side devices to obtain multiple copies of the model parameter offset, and generates a global biometric recognition model based on the multiple copies of the model parameter offset.

16. A training device for a biometric recognition model, characterized in that, The device includes: The sending module is configured to send a biometric recognition model to be trained to multiple training-side devices, so that each training-side device performs the following processing: obtaining a plurality of image samples sent by at least one terminal device, where the plurality of image samples are obtained by adding noise to a plurality of biometric images collected by the terminal device respectively; training the biometric recognition model to be trained based on the plurality of image samples for at least one round to obtain a local biometric recognition model; determining the model parameter offset of the local biometric recognition model relative to the biometric recognition model to be trained; encrypting the parameter offset to obtain encrypted data, and sending the encrypted data to the service-side device; The receiving module is configured to receive the encrypted data and decrypt it to obtain multiple copies of the model parameter offset; The generating module is configured to generate a global biometric recognition model based on multiple copies of the model parameter offset.

17. An electronic device, characterized in that, The electronic device includes: A memory for storing computer-executable instructions; A processor, when executing the computer-executable instructions stored in the memory, implements the training method of the biometric recognition model according to any one of claims 1 to 12, or implements the training method of the biometric recognition model according to claim 13 or 14.

18. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by the processor, the training method of the biometric recognition model according to any one of claims 1 to 12 is implemented, or the training method of the biometric recognition model according to claim 13 or 14 is implemented.

19. A computer program product, comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by the processor, the training method of the biometric recognition model according to any one of claims 1 to 12 is implemented, or the training method of the biometric recognition model according to claim 13 or 14 is implemented.

Images